From brideout at haystack.mit.edu Tue Sep 2 15:10:23 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:01 2004 Subject: [gps-developers] Matlab plotting of gps data Message-ID: <3F54EB1F.7050104@haystack.mit.edu> I'm modifying the Matlab code written by Phil and Anthea to write a general purpose Matlab method to plot our GPS data stored in the tec* files in cartesian coordinates. That method will be: plot_gps_cart(directory, start_time, end_time, step_lat, start_lat, end_lat, step_lon, start_lon, end_lon, min_tec, max_tec, filename, whiteBackground) where: directory - path to directory with tec* files start_time - Matlab date string or date number to set starting time end_time - Matlab date string or date number to set ending time step_lat - degrees latitude per bin start_lat - lower limit of latitude to show in map (-90 to 90) stop_lat - upper limit of latitude to show in map (-90 to 90) step_lon - degrees longitude per bin start_lon - lower limit of longitude to show in map (-180 to 180) stop_lon - upper limit of longitude to show in map (-180 to 180) min_tec - lower limit of TEC color scale max_tec - upper limit of TEC color scale filename - filename to save image. If blank, just display, and let user save whiteBackground - 1 for white background, 0 for black Some differences from the present version: 1. I've increased the speed so it can display 20 minutes of data in about 4 seconds. 2. It's showing the median of the binned data, not the mean. 3. The bins are not overlapping (which Phil made optional), so that each data point shows up in one and only one bin. I also plan to write more methods in the future that will create a series of maps and/or movies based on this method, and also a similar method to give a polar map. Let me know if you have any comments. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From jcf at haystack.mit.edu Wed Sep 3 12:26:25 2003 From: jcf at haystack.mit.edu (John Foster) Date: Wed Jul 21 10:29:01 2004 Subject: [gps-developers] Re: Matlab plotting of gps data Message-ID: <200309031626.h83GQPC11156@hyperion.haystack.edu> Neat I want a copy of this script! My version ('gpsort') is probably the grandparent of what progressed to Phil/Anthea/Bill and still takes ~120 sec (on my laptop) to process 20 min of data. Not great for making my own movies!!! JF Comments: having lots of user-accessible switches to set the plot characteristics is VG. Having an polar-projection version would be VG, as well my options include overlaying a magnetic coord contour grid (developed from your 2-D model output - usualy run for 350 km altitude) - useful at times I also like adding F-region the shadow line (SZA=100) which can be obtained from an SZA grid for the date/time of the frame (more complexities) Final data-manipulation suggestion: being able to fill in some of the missing bins by interpolation is very helpful when preparing final/publication versions of the maps - alternately, we could set up a few hundred more GPS sites or launch a bunch more GPS satellites. From brideout at haystack.mit.edu Wed Sep 3 14:11:38 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:02 2004 Subject: [gps-developers] Re: Matlab plotting of gps data References: <200309031626.h83GQPC11156@hyperion.haystack.edu> Message-ID: <3F562EDA.4050202@haystack.mit.edu> John and Anthea, I've installed the script plot_gps_cart on hyperion. Just type help plot_gps_cart from Matlab on hyperion to get a description of how to use it. Also attached are the two *.m files, which you can use on your local version of Matlab. These files are in CVS under Millstone/gps/source/scripts. I'll let you know when I incorporate some of your suggestions. Bill John Foster wrote: > Neat > > I want a copy of this script! > > My version ('gpsort') is probably the grandparent of what progressed > to Phil/Anthea/Bill and still takes ~120 sec (on my laptop) to process > 20 min of data. > > Not great for making my own movies!!! > > JF > > Comments: > > having lots of user-accessible switches to set the plot characteristics > is VG. > > Having an polar-projection version would be VG, as well > > my options include overlaying a magnetic coord contour grid (developed from > your 2-D model output - usualy run for 350 km altitude) - useful at times > > I also like adding F-region the shadow line (SZA=100) which can be > obtained from an SZA grid for the date/time of the frame (more complexities) > > Final data-manipulation suggestion: > > being able to fill in some of the missing bins by interpolation is very > helpful when preparing final/publication versions of the maps - alternately, > we could set up a few hundred more GPS sites or launch a bunch more GPS > satellites. > > _______________________________________________ > gps-developers mailing list > gps-developers@openmadrigal.org > http://www.openmadrigal.org/mailman/listinfo/gps-developers > > -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 -------------- next part -------------- function [] = plot_gps_cart(directory, ... start_time, end_time, ... step_lat, start_lat, end_lat, ... step_lon, start_lon, end_lon, ... min_tec, max_tec, ... filename, whiteBackground) % plot_gps_cart plots gps data in cartesian coordinates % % directory - path to directory with tec* files % start_time - Matlab date string or date number to set starting time % end_time - Matlab date string or date number to set ending time % step_lat - degrees latitude per bin % start_lat - lower limit of latitude to show in map (-90 to 90) % end_lat - upper limit of latitude to show in map (-90 to 90) % step_lon - degrees longitude per bin % start_lon - lower limit of longitude to show in map (-180 to 180) % end_lon - upper limit of longitude to show in map (-180 to 180) % min_tec - lower limit of TEC color scale % max_tec - upper limit of TEC color scale % filename - filename to save image. If blank, just display, and let user save % whiteBackground - 1 for white background, 0 for black % % Notes: % Uses the median of the binned data, not the mean. % The bins are not overlapping, so that each data point shows up in one and only one bin. % % Example: % plot_gps_cart('/usr/algo/DATA_04_2002',731321.0,731321.0139,3,0,90,3,-90,0,0,100,'p1.jpeg',1) % % Written by B. Rideout 9/2/2003 % Check for input mistakes if (nargin ~= 13) error('Not enough arguments') end % cd to directory presentDir = cd; cd(directory); filelist = dir('tec-*-*-*.dat'); fileSize = size(filelist); numFiles = fileSize(1,1); % throw an error if not files found if (numFiles == 0) cd(presentDir) error('No tec-???-??-??.dat files found in directory') end % convert start_time and end time into Matlab Datenum's startTime = datenum(start_time); endTime = datenum(end_time); % make sure both times have the same year, since files don't identify year startvec = datevec(startTime); startyear = startvec(1,1); endvec = datevec(endTime); endyear = endvec(1,1); if (startyear ~= endyear); cd(presentDir) error('Plot cannot cross year boundry, since files do not include year'); end % load all files within those times count = 0; for i=1:numFiles thisfile = filelist(i).name; % skip files of wrong length if (length(thisfile) == 17) % compute its time day = double(str2num(thisfile(5:7))); hour = double(str2num(thisfile(9:10))); minute = double(str2num(thisfile(12:13))); thistime = datenum(startyear,0,0,0,0,0) + day + hour/24.0 + minute/1440.0; % skip files outside the time range if (thistime >= startTime & thistime <= endTime) fid = fopen(thisfile); if (count == 0) this_data = fscanf(fid, '%g', [4 inf]).'; count = 1; else A = fscanf(fid, '%g', [4 inf]).'; this_data = [this_data; A]; end fclose(fid); end end end % if no files in given time range, throw error if (count == 0) error('No files found in given time range'); end % convert long to -180 to 180 for row=1:length(this_data) if (this_data(row,3) > 180.0) this_data(row,3) = this_data(row,3) - 360.0; end end % bin the data [bin_dat] = gps_bin(step_lat, step_lon, this_data); % create lon_grid and lat_grid lat_grid = -90:step_lat:90; lon_grid = -180:step_lon:180; % find start and end indexes of grids startLatIndex = 1; endLatIndex = 1; for i=1:length(lat_grid) if (start_lat >= lat_grid(i)) startLatIndex = i; end if (end_lat >= lat_grid(i)) endLatIndex = i; end end startLonIndex = 1; endLonIndex = 1; for i=1:length(lon_grid) if (start_lon >= lon_grid(i)) startLonIndex = i; end if (end_lon >= lon_grid(i)) endLonIndex = i; end end % shrink the grids and data to desired range bin_dat = bin_dat(startLatIndex:endLatIndex , startLonIndex:endLonIndex); lat_grid = lat_grid(startLatIndex:endLatIndex); lon_grid = lon_grid(startLonIndex:endLonIndex); % plot data clf; axis([min(lon_grid) max(lon_grid) min(lat_grid) max(lat_grid)]); set(gcf,'position',[232 258 560 420]); pcolor(lon_grid, lat_grid, bin_dat); caxis([min_tec, max_tec]); cb = colorbar; set(get(cb,'Title'), 'String', 'TEC'); title(sprintf('GPS TEC Map from %s to %s', ... datestr(startTime), datestr(endTime))); xlabel('Geodetic Longitude, Deg'); ylabel('Geodetic Latitude, Deg'); shading flat; if (whiteBackground == 0) % turn background black set(gca,'Color', [0 0 0]); end % add coastline load coast.mat; hold on; if (whiteBackground == 0) plot(long, lat, 'w-'); else plot(long, lat, 'k-'); end % save if filename given if (length(filename) > 1) eval(['print -djpeg -zbuffer ' filename]); end % return to original dir cd(presentDir) -------------- next part -------------- function [bin_dat] = ... gps_bin(lat_step, lon_step, gdata) % [bin_dat] = ... % GPS_BIN(lat_step, lon_step, gdata) % % Bin-sort GPS TEC files (from Anthea Coster) and return a bin-sorted array. % % Modified by B. Rideout 8/29/2003 to improve speed, return median rather % than mean of binned data. % % INPUTS: % lat_step step size of latitude bins, in degrees. % lon_step step size longitude bins, in degrees. % gdata N x 4 array containing TEC values, in the format: % time gdlat gdlon tec % ... % time = decimal day of the year (with fraction) % gdlat = geodetic latitude, deg % gdlon = geodetic longitude, deg % tec = TEC, std units % % RETURNS: % bin_dat K x L array [K = 180/lat_step = 1 L = 360/lon_step + 1] % with binned TEC data. Bins with no data are set to NaN. % % To make things faster, set maximum size of bin maxBinSize = 50; % sp_bin_dat is 3D array holding binned data sp_bin_dat = zeros(ceil(180.0/lat_step)+1, ceil(360.0/lon_step)+1, maxBinSize); % bin_dat is 2D array holding mean of binned data - to be returned bin_dat = zeros(ceil(180.0/lat_step)+1, ceil(360.0/lon_step)+1); % bin_index is 2D array holding index of each binned data cell bin_index = zeros(ceil(180.0/lat_step)+1, ceil(360.0/lon_step)+1); % loop through each data point for i=1:length(gdata) % get index into sp_bin_dat indexLat = ceil(((gdata(i,2) + 90.0)+(lat_step/2.0))/lat_step); indexLon = ceil(((gdata(i,3) + 180.0)+(lon_step/2.0))/lon_step); % check whether that bin is full newIndex = bin_index(indexLat, indexLon); if (newIndex < maxBinSize) % append this point sp_bin_dat(indexLat, indexLon, newIndex + 1) = gdata(i,4); bin_index(indexLat, indexLon) = newIndex + 1; end end % now that data is loaded in sp_bin_dat, populate bin_dat for i=1:ceil(180.0/lat_step)+1 for j=1:ceil(360.0/lon_step)+1 if (bin_index(i,j) > 0) bin_dat(i,j) = median(sp_bin_dat(i,j,:)); else bin_dat(i,j) = NaN; end end end From brideout at haystack.mit.edu Thu Sep 4 08:30:43 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:03 2004 Subject: [gps-developers] Bug in gps_bin.m Message-ID: <3F573073.1040605@haystack.mit.edu> John, You were correct, and I found and fixed a bug in the latest version of gps_bin.m that added extra zeros into some median calculations. (Anthea, this didn't effect the earlier version I gave you - but this new one is much faster than even that one). I fixed the version on hyperion, and attached the corrected version for your local Matlab installations. I also attached the latest jpeg of day 150, the first 20 minutes. I'll plot out some histograms today also, so you can see the median versus mean effect. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 -------------- next part -------------- function [bin_dat] = ... gps_bin(lat_step, lon_step, gdata) % [bin_dat] = ... % GPS_BIN(lat_step, lon_step, gdata) % % Bin-sort GPS TEC files (from Anthea Coster) and return a bin-sorted array. % % Modified by B. Rideout 8/29/2003 to improve speed, return median rather % than mean of binned data. % % INPUTS: % lat_step step size of latitude bins, in degrees. % lon_step step size longitude bins, in degrees. % gdata N x 4 array containing TEC values, in the format: % time gdlat gdlon tec % ... % time = decimal day of the year (with fraction) % gdlat = geodetic latitude, deg % gdlon = geodetic longitude, deg % tec = TEC, std units % % RETURNS: % bin_dat K x L array [K = 180/lat_step = 1 L = 360/lon_step + 1] % with binned TEC data. Bins with no data are set to NaN. % % To make things faster, set maximum size of bin maxBinSize = 50; % sp_bin_dat is 3D array holding binned data sp_bin_dat = zeros(ceil(180.0/lat_step)+1, ceil(360.0/lon_step)+1, maxBinSize); % bin_dat is 2D array holding mean of binned data - to be returned bin_dat = zeros(ceil(180.0/lat_step)+1, ceil(360.0/lon_step)+1); % bin_index is 2D array holding index of each binned data cell bin_index = zeros(ceil(180.0/lat_step)+1, ceil(360.0/lon_step)+1); % loop through each data point for i=1:length(gdata) % get index into sp_bin_dat indexLat = ceil(((gdata(i,2) + 90.0)+(lat_step/2.0))/lat_step); indexLon = ceil(((gdata(i,3) + 180.0)+(lon_step/2.0))/lon_step); % check whether that bin is full newIndex = bin_index(indexLat, indexLon); if (newIndex < maxBinSize) % append this point sp_bin_dat(indexLat, indexLon, newIndex + 1) = gdata(i,4); bin_index(indexLat, indexLon) = newIndex + 1; end end % now that data is loaded in sp_bin_dat, populate bin_dat for i=1:ceil(180.0/lat_step)+1 for j=1:ceil(360.0/lon_step)+1 if (bin_index(i,j) > 0) bin_dat(i,j) = median(sp_bin_dat(i,j,1:bin_index(i,j))); else bin_dat(i,j) = NaN; end end end -------------- next part -------------- A non-text attachment was scrubbed... Name: bill_150_00_00.jpeg Type: image/jpeg Size: 111378 bytes Desc: not available Url : http://openradar.org/pipermail/gps-developers/attachments/20030904/32598cde/bill_150_00_00.jpeg From brideout at haystack.mit.edu Wed Sep 10 08:58:51 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:29:03 2004 Subject: [gps-developers] Proposal for putting GPS data on Madrigal - 365 days a year Message-ID: <3F5F200B.8090203@haystack.mit.edu> It's time to consider putting all GPS data on Madrigal, in order to make use of the archiving and data access Madrigal provides. My goal in putting all GPS data on Madrigal would not be to provide as detailed data as we can get now by running the process manually, but to give summary data in movie form that a user can use to look for interesting events that warrant further investagation. For this reason, I propose storing the following data on Madrigal: 1. Data binned in 3 degree X 3 degree X 20 minutes bins as a Madrigal file. Each gps "experiment" would cover one month. 2. Movies in mpeg format with a whole-world cartesian view, one frame per 20 minutes, and 3 degree X 3 degree bins (accessible as links). Estimated disk space needs: 1. Madrigal files: 250 MB/year of data 2. Movie files: 1500 MB/year of data (mpeg files, not avi) So for 10 years of data we would need about 18 GB of disk space, or just about 110% of what's now available on /opt/madrigal/experiments. So clearly if we decide to do this we will need more space on /opt/madrigal/experiments. This data would be inferior to the manual data in two ways: 1. There would be about 15% less data, since the automated process tends to throw out data that could partially be recovered manually. 2. The data would be stored only in binned form, not as raw data, so to display greater detail (smaller space or time bins), the manually process would be needed. Howver, since the binned data is stored in Madrigal, it would be possible to create new movies or images. Data loading: My plan for loading data would be to start with the present and work back to around 1994. After all data is loaded, I'll write a cron job to load new data each month. If someone needs more real-time data (less than one month), let me know and I'll set up the loading to contain partial months, up to 3 or 4 days from the present. Let me know if there is enough interest in this data to warrant the effort needed. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From shunrong at haystack.mit.edu Wed Sep 10 09:51:23 2003 From: shunrong at haystack.mit.edu (Shunrong Zhang) Date: Wed Jul 21 10:29:03 2004 Subject: [gps-developers] Proposal for putting GPS data on Madrigal - 365 days a year In-Reply-To: <3F5F200B.8090203@haystack.mit.edu> Message-ID: > 1. Data binned in 3 degree X 3 degree X 20 minutes bins as a Madrigal > file. Each gps "experiment" would cover one month. > I don't know but maybe 30 min is still good enough for most studies not related to traveling ionospheric disturbances. (In fact, JPL's movie progresses every 60 min; most ionosondes are at 30-60 min interval.) This will reduce madrigal file size and the movie file size too. > 2. Movies in mpeg format with a whole-world cartesian view, one frame > per 20 minutes, and 3 degree X 3 degree bins (accessible as links). > > Estimated disk space needs: > > 1. Madrigal files: 250 MB/year of data > 2. Movie files: 1500 MB/year of data (mpeg files, not avi) The official MATLAB 6.X can produce avi movie only but there are non-official functions that generates mpeg. An avi seems smaller in size than an mpeg. > > Data loading: > > My plan for loading data would be to start with the present and work > back to around 1994. After all data is loaded, I'll write a cron job to > load new data each month. If someone needs more real-time data (less > than one month), let me know and I'll set up the loading to contain > partial months, up to 3 or 4 days from the present. > Could you set Oct-Nov 2002 data at higher priority ? I'm very interested in this time period when our long-duration experiemnts took place. Along with our data, ESR data, and TIMED/GUVI data, I've collected 12 ionosonde data in the northern American and European sectors. It'll be great to have TEC data. Shunrong From pje at haystack.mit.edu Wed Sep 10 10:19:50 2003 From: pje at haystack.mit.edu (Phil Erickson) Date: Wed Jul 21 10:29:04 2004 Subject: [Fwd: Re: [gps-developers] Proposal for putting GPS data on Madrigal - 365 days a year] Message-ID: <3F5F3306.1070408@haystack.mit.edu> Oops - sent to the wrong address. (I note by the way that gps-developers messages don't seem to have a correct Reply-To: address set.) ---- phil -------------- next part -------------- An embedded message was scrubbed... From: Phil Erickson Subject: Re: [gps-developers] Proposal for putting GPS data on Madrigal - 365 days a year Date: Wed, 10 Sep 2003 10:18:40 -0400 Size: 2236 Url: http://openradar.org/pipermail/gps-developers/attachments/20030910/5fe6ce58/gps-developersProposalforputtingGPSdataonMadrigal-365daysayear.mht From pje at haystack.mit.edu Wed Sep 10 10:22:28 2003 From: pje at haystack.mit.edu (Phil Erickson) Date: Wed Jul 21 10:29:04 2004 Subject: [Fwd: Re: [gps-developers] Proposal for putting GPS data on Madrigal - 365 days a year] Message-ID: <3F5F33A4.4030202@haystack.mit.edu> Boy, you'd think I could get this right eventually. This should be the right list. (As I mentioned in the other fatal attempts to send this, we should definitely set the Reply-To: field correctly on gps-developers.) ---- phil -------------- next part -------------- An embedded message was scrubbed... From: Phil Erickson Subject: Re: [gps-developers] Proposal for putting GPS data on Madrigal - 365 days a year Date: Wed, 10 Sep 2003 10:18:40 -0400 Size: 2236 Url: http://openradar.org/pipermail/gps-developers/attachments/20030910/48c5abd1/gps-developersProposalforputtingGPSdataonMadrigal-365daysayear.mht From jmh at haystack.mit.edu Wed Sep 10 10:25:18 2003 From: jmh at haystack.mit.edu (John Holt) Date: Wed Jul 21 10:29:04 2004 Subject: [Fwd: Re: [gps-developers] Proposal for putting GPS data on Madrigal - 365 days a year] Message-ID: <200309101425.h8AEPIx02480@chaos.haystack.mit.edu> Phil is right. mpeg it is. John From ajc at haystack.mit.edu Mon Sep 15 14:29:15 2003 From: ajc at haystack.mit.edu (Anthea Coster) Date: Wed Jul 21 10:29:04 2004 Subject: [gps-developers] GPS Workshop 10-11 November 2003 Logistics Message-ID: <3F6604FB.9080900@haystack.mit.edu> *__**__* The Haystack Observatory is sponsoring a GPS workshop this 10-11 November 2003 (a Monday and Tuesday, Veteran's Day). The purpose of this workshop is to bring together a subset of ionospheric scientists within the GPS community who would like to collaborate on software development, scientific studies, and long-range planning issues. We will be sending out another email in a few weeks concerning the exact agenda of our meeting (although we are requesting that everyone prepare ~10 min summaries on their various projects). This email concerns logistics. We will reserve a group of rooms at the Westford Regency which is near Haystack and local eating establishments. We should be able to arrange a reduced (MIT) rate for workshop attendees. There is a shuttle service that can be used to take you to and from Boston 's Logan airport. If you fly into the Manchester NH Airport, you will probably have to rent a car. We will provide snacks and lunch at the workshop on both Monday and Tuesday. If enough people are interested, we will arrange an optional dinner at a nearby restaurant on Monday night. Breakfast is on your own. No workshop registration fee will be charged. To coordinate food, transportation, and lodging, we need to know: Are you planning on attending our workshop? Will you need a hotel room, and if so what days? Will you need transportation to and from the hotel to Haystack? Are you interested in attending a group dinner on Monday evening? We need these answers from everyone planning on attending the workshop by September 19^th (this Friday). If you do have suggestions as to how to organize our workshop (which we stress is a workshop not a conference), please email them to: Anthea Coster ajc@haystack.mit.edu Thank you, Anthea The original announcement is included as a txt file with this mailing. ****************************************** Anthea J. Coster MIT Haystack Observatory Atmospheric Sciences Off Route 40 Westford, MA 01886 phone: 781-981-5753 email: ajc@haystack.mit.edu ****************************************** -------------- next part -------------- We are organizing an informal (1-2 day) GPS Ionospheric Working group meeting at the Haystack Observatory in Westford, MA. The goal is to start to coordinate the research activities of our various groups, to devise strategies to share information, and to work at defining larger collaborative projects that our individual groups can work on together. The number one issue here is how best to address the science. Possible topics for discussion: - Technical issues (best receivers, issues with data processing) - Discussion of projects our different groups are working on - Major science objectives - Plan for special sessions at AGU, URSI, ION, etc upcoming meetings which highlight the GPS-related science activities. - How/where new receivers needed for atmospheric applications Time: Our preference is during the week of 10-11 November. Please let us know if you would attend and give us you input as to appropriate dates. Your suggestions as to other attendees and topics to be discussed would be appreciated. --