From brideout at haystack.mit.edu Wed Jul 16 16:14:10 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:57 2004 Subject: [gps-developers] Elevation question Message-ID: <3F15B212.6010003@haystack.mit.edu> Anthea, I've got the code working in algo to filter by elevation range. However, I see that the elevations I'm getting go below 0 degrees, rather than 0 to 90 as I expect. I found elevations going all the way down to -47 degrees, with the number of elevations below 0 degrees representing a significant portion of the data. Does this make sense to you? The azimuth values go from -180 to 180, which is reasonable. A negitive elevation seems to represent a satellite below the horizon, which doesn't seem to make sense. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Jul 17 09:01:37 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:58 2004 Subject: [gps-developers] New feature in process_gps.py Message-ID: <3F169E31.4080607@haystack.mit.edu> Anthea, Never mind the previous email about negitive elevations - I mistyped the algo command line and used the wrong antenna. So now elevation filtering is all set. I added two new arguments in the automated processing script, process_gps.py. They are: --lowerEl --upperEl lowerEl defaults to 0 degrees, upperEl to 90 degrees. To install, do just what you did Tuesday, which is to go to http://www.haystack.mit.edu/address/brideout.html and follow the directions under Auto GPS processing toolkit (that is, copy into your directory, untar, and recompile fixheader). Some examples of using the new script: process_gps.py -d smallDir --lowerEl=30 --upperEl=60 DATA_01_2001 ** only accepts elevations between 30 and 60 ** process_gps.py -d smallDir --lowerEl=15 DATA_01_2001 ** only accepts elevations above 15 ** process_gps.py -d smallDir DATA_01_2001 ** accepts all elevations ** Important note: To the best of my knowledge, there is no elevation filtering going on in algo prior to this change. This is not to say that there were no lines of code that are designed to do elevation filtering in the code - in fact, there are many lines that seemed to be designed to do elevation filtering. I spend about four hours putting print statements around all these "elevation filtering" lines, to see which ones were being called. In the end, I found that none were being called with the arguments we pass into algo. Ultimately, I added my own code in the file iono.cc where there was already code that rejected records for other reasons. Please let me know how things are going. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Jul 17 15:42:18 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:58 2004 Subject: [gps-developers] Need for disk space to process gps data Message-ID: <3F16FC1A.5050900@haystack.mit.edu> Ching, Anthea is running into a disk space problem trying to process GPS data on her laptop. Now that we have automated processing, we can use all available data, which can require a few GB's of space for a few days. Is there any way she can get 10 GB's (or more) on a network drive? Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Wed Jul 23 09:47:55 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:58 2004 Subject: [gps-developers] Short-term need for har-drive for IBM laptop Message-ID: <3F1E920B.5010107@haystack.mit.edu> Anne, Sorry to send one more email for you to read just when you get back from vacation, but... Is it possible to get a hard drive with at least 40 GBytes for an IBM laptop quickly? Anthea is using her laptop to process GPS data for the moment, and is limited by the amount of disk space her laptop has. In the slightly longer run, having a few hundred GB's of network disk space so I could preprocess GPS data would also be very helpful and save time. This would only be intermediate data to save Anthea time, and would not need to be backed up, since it could always be regenerated. In the long run, how much disk space we need for GPS data depends on how the automation of the processing goes. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Jul 24 15:43:00 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:58 2004 Subject: [gps-developers] Setting up for processing gps data Message-ID: <3F2036C4.3090300@haystack.mit.edu> Anthea, I've been talking to Ching and Anne about how to set you up to easily process gps data without worrying about disk space. The fastest way to do it is to use existing network drives, and have you ssh into hyperion. You should be able to do this from your office, from Lincoln Labs, or from home. I'll set it up with you the next time you're at Haystack. Given the larger amount of disk space available, you can then process data more efficiently by processing multiple days at once. This would mean you could look at multiple days of data in a given plot__z file. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Mon Jul 28 10:19:37 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:58 2004 Subject: [gps-developers] Processing gps data in the background Message-ID: <3F2530F9.6040101@haystack.mit.edu> Anthea, I looked at using "at" to run gps processing jobs in the background. I think we were using "at" incorrectly Friday afternoon, because the following approach seems to work: 1. Create a shell script to run. This you'll have to do by hand. Here's an example: #!/bin/sh cd /usr/oasis/experiments/gps/algo ./process_gps.py -d tmp_gps_2003_100 DATA_04_2003 2> log_04_2003.txt (Note that "#!/bin/sh" must be on the first line, and a blank line must follow). 2. Save that shell script as something like "processit.sh" in algo, and be sure it has execute permission by running "chmod a+x processit.sh". 3. Run the following command: at -f /usr/oasis/experiments/gps/algo/processit.sh now + 2 minutes You should then be able to log off. You might want to wait a few minutes and look at the log file to make sure you didn't get an error right away - for example, if the directory DATA_*_* has files in it already. Just edit "processit.sh" to run a different job, and repeat step 3. Let me know if there's any problems. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Tue Jul 29 14:12:25 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:58 2004 Subject: [gps-developers] A new gps processing method Message-ID: <3F26B909.7000009@haystack.mit.edu> Anthea, I ran into one more problem trying to do the processing using /usr/oasis/experiments - its darn slow. The slow speed is due to the fact that the file system is on the network. The program as written now writes to files constantly, especially since we made those changes to uncompress and recompress data files as we go. So I came up with two solutions, one easy (hopefully) and one hard. The hard one is to modify the program to use less file access and do more data manipulation in memory. The easy solution is for me to just run the job on grail using the local disk drive, and just copy the final data files to /usr/oasis/experiments/gps. Then you can plot the data and run the rerun_gps.py script from there, where speed shouldn't be an issue. So, if that's alright with you, just send me the list of dates you want me to process. I have 20 GB of disk space, so I can do 4 days in a row, and possibly five. If you want any geographical filters or elevation filters applied, just let me know. The data for days 149-150, 2003 will be ready Wednesday morning. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Wed Jul 30 10:15:28 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:58 2004 Subject: [gps-developers] Speeding up processing of gps data Message-ID: <3F27D300.9020204@haystack.mit.edu> Anthea, I'm trying to find a way to make it faster for you to loop through all the ~900 sites of data in a given run. I have two suggestions. Let me know if you think either or both are good ideas. Suggestion 1: Right now I have two scripts, plot_gps.py and rerun_gps.py that allow you to look at the data and either accept it, reject it, or reanalyze it. I propose to write a very small program that combines these two into one step by doing the following: 1. It loops through every plot in a directory, plotting it. 2. For each plot, you have three choices: A) Push the accept button, and the next site graph appears. B) Push the reject button, and those files are deleted, and the next site graph appears. C) Fill in text area to set either new TEC bounds, and/or time ranges to remove, and hit the Rerun button. The modified version of the graph appears. You can then choose accept, reject, or rerun again. As before, you can enter the name of the site to start at if you already processed some sites already. This should make it so that you don't need to write anything down while looking at the data, and so you only loop through the graphs once. Suggestion 2: I think you use a site.lst file in your Matlab code. If you want I can write you a script that will create such a file, including only sites within a given geographic range. That is, it will list only sites within the min and max latitudes and longitudes, just like the process_gps.py does. Let me know what you think. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Wed Jul 30 13:16:23 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:58 2004 Subject: [gps-developers] Data for May 29-30, 2003 on /usr/oasis/experiments/gps Message-ID: <3F27FD67.5030807@haystack.mit.edu> Anthea, The data for May 29-30, 2003 is now in /usr/oasis/experiments/gps/algo/DATA_05_2003. It is ready for you to look at the plots. Just to review, the procedure to analyze the data is now: ssh -l ajc hyperion.haystack.mit.edu (After login:) ssh nuwa cd /usr/oasis/experiments/gps/algo ./plot_gps.py DATA_05_2003 (Hit Cntr-D to go to next plot. If you want to start in the middle, type "./plot_gps.py -s DATA_05_2003", where is four letter site to start at.) To rerun with new TEC bounds or with time ranges removed, run: rerun_gps.py -s site [-u upperTEC -l lowerTEC -r lower,upper] DATA_MM_YYYY Note that because you are on the network file system, rerun_gps.py takes about 50 seconds to run, as opposed to about 15 locally. So you always have the option to copy the DATA directory to your laptop is you want better performance, but I don't think you actually use rerun_gps.py that much. This procedure will change if I write the program to integrate those two scripts. Let me know if you have any questions. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Wed Jul 30 16:40:22 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:59 2004 Subject: [gps-developers] Data for April 15-16, 2002 now on /usr/oasis/experiments/gps Message-ID: <3F282D36.9020809@haystack.mit.edu> Anthea, The second set of gps data, this time for April 15-16, 2002, is now in /usr/oasis/experiments/gps/algo/DATA_04_2002. I expect to have the processing GUI that I described earlier ready to run by Friday. I'll let you know when its ready. In any case, you can still use the two scripts to go through the data as before. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Jul 31 15:45:01 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:59 2004 Subject: [gps-developers] Python extentions Message-ID: <3F2971BD.8030003@haystack.mit.edu> Ching, I need two python extentions installed on nuwa to let Anthea run my GPS GUI. They are Numeric and Gnuplot. The files are available at: http://sourceforge.net/projects/numpy and http://gnuplot-py.sourceforge.net/ Could you let me know when they are installed, so I can make sure its working? Thanks, Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Jul 31 17:19:55 2003 From: brideout at haystack.mit.edu (William Rideout) Date: Wed Jul 21 10:28:59 2004 Subject: [gps-developers] GUI for looking at GPS data Message-ID: <3F2987FB.6010407@haystack.mit.edu> Anthea, The GUI application that lets you step through the GPS data is ready and installed on nuwa. I can give you a demo when you get back, or you can try it yourself. In algo, just type: ./GpsUI.py DATA_05_2003 Two windows will pop up, a control window and a graph window. Move them around so they don't overlap. For each graph you see, you hit either the Accept button, the Reject button, or the Rerun button. If you hit the rerun button, the data will be redone with the TEC limits and time ranges dropped that you entered in the window, and the new graph will be displayed; at which time you can again Accept, Reject, or Rerun. By the way, I'm rerunning the April 2002 data since the nav file was corrupt for one of the two days and we only got one out of two days. It should be ready Friday AM. Let me know if you need this program to do anything else (such as logging). Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624