From brideout at haystack.mit.edu Fri Aug 1 11:37:37 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:59 2004 Subject: [gps-developers] Some small changes to GpsUI.py Message-ID: <3F2A8941.2090501@haystack.mit.edu> Anthea, I made some small changes to my program to let you check the graphs quickly. Those changes are: 1. When you reject a site, its files are moved to the subdirectory "rejected" instead of being deleted. This allows you to change your mind later and just copy them back to the DATA directory. 2. A log file is written of all sites rejected, and all sites accepted after being rerun with new TEC limits and/or time ranges dropped. 3. I modified the awkit scripts to make them faster by combining things, etc. It now takes about 15 seconds per day of data, so about 30 seconds for the two day runs we now have, to rerun a site. Just to review, all you need to do to check these graphs from anywhere in the world is: * ssh -l ajc hyperion.haystack.mit.edu * ssh nuwa * cd /usr/oasis/experiments/gps/algo * ./GpsUI.py DATA_05_2003 (for example) Directions for using GpsUI.py: Two windows will pop up, a control window and a graph window. Move them around so they don't overlap. For each graph you see, you hit either the Accept button, the Reject button, or the Rerun button. If you hit the rerun button, the data will be redone with the TEC limits and time ranges that you entered in the window, and the new graph will be displayed; at which time you can again Accept, Reject, or Rerun. If you want to go back to the full data set, just blank out all the filter fields and hit Rerun again. If you hit Reject, the site files are moved to the "rejected" directory, and you can add an optional explanation to be written in the log file. Let me know if you have any questions. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Fri Aug 1 15:18:37 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:59 2004 Subject: [gps-developers] Comparing el and az calculations Message-ID: <3F2ABD0D.1060705@haystack.mit.edu> Nils, I've stepped completely through your test of el and az calculations using the old Mike Pratt code versus your code. I've used data from 2002 and 2003, different gps satellites, etc. So far I've gotten perfect agreement in every case, so my guess is that something was wrong with that *.iono file. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Fri Aug 8 17:06:05 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] Script for creating sites.lst Message-ID: <3F3410BD.8040207@haystack.mit.edu> Anthea, I've written a script called createSitelist.py that will create a file called sites.lst with a list of site names. It's on nuwa in /usr/oasis/experiments/gps/algo. Usage is: createSitelist.py [--minLatitude=degrees --maxLatitude=degrees --minLongitude=degrees --maxLongitude=degrees] DATA_MM_YYYY You can filter the sites geographically. Also, only sites with plot_*_z files are included, since other sites had errors. I don't know if this is exactly what you need, so its not in CVS yet. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From ajc at haystack.mit.edu Fri Aug 8 20:13:24 2003 From: ajc at haystack.mit.edu (Anthea Coster) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] Script for creating sites.lst In-Reply-To: <3F3410BD.8040207@haystack.mit.edu> Message-ID: Bill: great - I will use it this weekend. I have stolen the pc from my kids now. Anthea On Fri, 8 Aug 2003, William Rideout wrote: > Anthea, > > I've written a script called createSitelist.py that will create a file > called sites.lst with a list of site names. It's on nuwa in > /usr/oasis/experiments/gps/algo. Usage is: > > createSitelist.py > [--minLatitude=degrees --maxLatitude=degrees > --minLongitude=degrees --maxLongitude=degrees] > DATA_MM_YYYY > > You can filter the sites geographically. Also, only sites with plot_*_z > files are included, since other sites had errors. > > I don't know if this is exactly what you need, so its not in CVS yet. > > Bill > > -- > Bill Rideout > MIT Haystack Observatory > Email: brideout@haystack.mit.edu > Phone: 781 981-5624 > > _______________________________________________ > gps-developers mailing list > gps-developers@openmadrigal.org > http://www.openmadrigal.org/mailman/listinfo/gps-developers > From brideout at haystack.mit.edu Mon Aug 11 15:49:23 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] Matlab files Message-ID: <3F37F343.4090305@haystack.mit.edu> Anthea, Could you send me the matlab file(s) you use to process the gps data? It doesn't have to be perfect code, but I'd like to get it into CVS. You can always make revisions later. I'm also interested in trying to see how it works. One thing I'd like to try is to see how much different the final data is if you skip the manual data filtering steps you're doing now. This should give me an idea as to how much work it will be to try to automate this step. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Tue Aug 12 16:05:21 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] New version of GpsUI Message-ID: <3F394881.5070404@haystack.mit.edu> Anthea, I put the new version of the program to display the gps graphs on nuwa. Because its slightly slower in redrawing the graph, I called it GpsUI_2.py. The old version, GpsUI.py, is still there, so you can compare. It uses a graphing program called quickplot (http://quickplot.sourceforge.net/) instead of gnuplot. It does display the x-y position of your mouse, so it should be easier to filter data without guessing the edge values. It also skips empty plot__z files. Let me know which you like better. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Aug 14 16:59:08 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] Manually filtered gps data versus unfiltered Message-ID: <3F3BF81C.7020106@haystack.mit.edu> Anthea and John, I'd like to test the degree to which the manual examination of gps data makes a difference in the final gps output. Of the total of 960 sites included in the 5/2003 data, Anthea's manual inspected rejected 36 sites, and modified 53 sites. The vast majority of the rejected files were due to missing data, and I can easily produce a filter to remove those automatically. This leaves only the few rejected files and the modified files left as a manual process. It seems very possible to me that given our large data set, skipping this manual step may well not make too much of a difference. This would allow fully automated GPS data processing, with all its many benefits. So I'd like to produce a set of tec files using the automated process that someone can compare to the manually editted data. Any suggestions for a time range? Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From jcf at haystack.mit.edu Fri Aug 15 12:07:12 2003 From: jcf at haystack.mit.edu (John Foster) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] Re: TIMES TO PROCESS AUTOMATICALLY Message-ID: <200308151607.h7FG7Cv24499@hyperion.haystack.edu> Bill, Anthea, since I'm looking at the 29 May 2003 data at present, I'd suggest that day. It should be Day 149, although the GPS data files in the oasis directory for this day are for Day 148 (different numbering system, I suppose) JF From brideout at haystack.mit.edu Tue Aug 19 11:05:53 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] Data with 20 degree filter Message-ID: <3F423CD1.4080106@haystack.mit.edu> John, A new version of the automated data processing is available under /usr/oasis/experiments/gps/algo/DATA_05_2003_el20. This data differs from the first run in that a minimum elevation angle of 20 degrees was applied. The only other filter is the removal of sites with too little data. Please let me know how this new data looks compared to Anthea's manually checked version, and the previous unchecked data. I'll also be rerunning the same data set with a 7 degree elevation filter today. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Fri Aug 22 11:30:13 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] Mapping function question Message-ID: <3F463705.7080508@haystack.mit.edu> Anthea, I noticed that the mapping function to get vertical TEC from line-of-sight TEC is different in the algo program than in Figure 1 of K. Duh's paper you sent me. In particular, the algo program at 10 degrees gives a value around 2.4, whereas in K. Duh's paper its around 2.9. See http://www.haystack.mit.edu/~brideout/Screenshot-3.png for algo's mapping function, which is calculated as given below. Do you trust either of these two functions more? The data if anything seems to decrease in TEC at lower elevations. Bill ******************************************** double IonoTrack::IonoMap(double e){ double hmin=450.00, hmax=650.0, ae=WGS84::a/1000.; double dh = hmax - hmin; // slant TEC arguments double R1 = pow((ae + hmin),2); double R2 = pow((ae + hmax),2); double R3 = pow((ae*cos(e)),2); // return mapping function return (sqrt(R2 - R3) - sqrt(R1 - R3))/dh; } -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Wed Aug 27 10:57:02 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] Revision to the original algo gps code Message-ID: <3F4CC6BE.4050803@haystack.mit.edu> Some thoughts on future code development for the GPS project: We now have two possible sources of code on which to expand our GPS processing capability, the Lincoln Lab code and GAMIT from MIT. Each has its pluses and minuses. The LL code is presently in use, but is undocumented. The GAMIT code is much better documented, but may not be easy to build upon. Anthea's summer student successfully used GAMIT code to parse Rinex-format files, but was not able to easily incorporate GAMIT's code for correcting for phase slips. Part of the reason for this is that GAMIT requires many more inputs than the LL code since its trying to solve a harder problem; its unclear how easy it will be to get the GAMIT code to work without those inputs. Given this, I decided to see if its possible to clean up and document the LL code. So far I've taken two steps: 1. Removed every file not needed to compile algo. 2. Changed some I/O code to remove non-standard C++ code that's not supported by gcc 3.X. It now compiles on all gcc versions. Just to make it easy to go back to the original code, I tagged it all "original" before making these changes. The original version only compiles on gcc 2.X compilers. My next step is to remove any unneeded methods or classes within these files. After that, I'll step through with a debugger to try to understand the program, and hopefully add documentation. I don't know which code base will be best for the future, but if nothing else this work may help make short term improvements in the present code. Let me know what you think. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Fri Aug 29 17:13:01 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:00 2004 Subject: [gps-developers] New version of gps_bin.m Message-ID: <3F4FC1DD.3070005@haystack.mit.edu> Attached is new version of gps_bin.m designed to calculate median values rather than averages; and to be faster. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 -------------- next part -------------- function [bin_dat, bin_num] = ... gps_bin(lat_grid, lon_grid, lat_width, lon_width, gdata) % [bin_dat, bin_num] = ... % GPS_BIN(lat_grid, lon_grid, lat_width, lon_width, gdata) % % Bin-sort GPS TEC files (from Anthea Coster) and return a bin-sorted array. % % Modified by B. Rideout 8/29/2003 to improve speed, return median rather % than mean of binned data. % % INPUTS: % lat_grid Array of geodetic latitude bins, in degrees. % lon_grid Array of geodetic longitude bins, in degrees. % lat_width Width of each latitude bin, in degrees. (Allows bins % which are wider or narrower than the actual grid % spacing.) % lon_width Width of each longitude bin, in degrees. (Allows bins % which are wider or narrower than the actual grid % spacing.) % gdata N x 4 array containing TEC values, in the format: % time gdlat gdlon tec % ... % time = decimal day of the year (with fraction) % gdlat = geodetic latitude, deg % gdlon = geodetic longitude, deg % tec = TEC, std units % % RETURNS: % bin_dat K x L array [K = length(lat_grid) L = length(lon_grid)] % with binned TEC data. Bins with no data are set to NaN. % bin_num K x L array with the number of samples in each TEC bin. % bin_dat = zeros(length(lat_grid), length(lon_grid)); bin_num = zeros(length(lat_grid), length(lon_grid)); % loop through each bin separately to make find faster for i=1:length(lat_grid) for j=1:length(lon_grid) ya = find(abs(lat_grid(i) - gdata(:,2)) < lat_width./2 & ... abs(lon_grid(j) - gdata(:,3)) < lon_width./2); if ~isempty(ya) bin_dat(i,j) = median(gdata(ya,4)); bin_num(i,j) = length(ya); end end end % set empty cells to NaN ya = find(bin_num == 0); bin_dat(ya) = NaN;