A brief overview of plans for the HOPS system - C.J. Lonsdale, July 9 1996 -------------------------------------------------------------------------- As will be apparent from an inspection of HOPS in its current state, it is not yet a complete, end-to-end VLBI postprocessing system. This document describes the functionality we expect to add over the next year or two, and the long-term direction of the package. The principal driving force behind HOPS development has been the needs of the sponsoring agencies involved in geodetic VLBI. Consequently, the first order of business has been the transfer of production geodetic data processing to the UNIX platform, in support of the still active MkIII correlator at Haystack, with applicability to other MkIII correlators. This transfer is essentially complete, where completion is defined as the obsolesence of all software on the old HP-1000 computers with the exception of the online correlator control functions (up to and including program COREL and associated utilities). A second major goal of the HOPS system is to provide a highly flexible postprocessing environment for the MkIV/EVN correlator under development at Haystack and JIVE. To achieve a smooth transition from MkIII to MkIV, we have adopted an evolutionary approach to our postprocessing software development. Most of the current software is designed to be directly reusable for MkIV, but many changes/enhancements are anticipated. Among these are: 1. Redefinition of several file formats, most particularly the root and corel file binary formats. The correlator control file format has been completely redesigned (this is the VEX format), and the MkIV binary file formats have been modified for compatibility with VEX. Note however that the higher-level organization of the data (root, corel and fringe files, directory structure, naming conventions and so on) will not change, at least from the point of view of the user. The implementation of an IO library for the new file formats is in progress. 2. The addition of new file types beyond 0, 1 and 2. One of these will be an ascii log file for tracking of data processing. We hope to define and implement these file types by the end of 1996. 3. Integration with the MkIV unix-based online correlator control software. This is primarily a requirement placed on the online software, rather than a plan to adjust the HOPS system, though some changes may occur. 4. Development of enhanced data visualization tools (e.g. a graphics-based fringe plot to replace the current pure ascii plot), and GUI tools. The latter will probably be implemented via commercially available X-window encapsulation tools with little change in current application code. Finally, and of greatest interest to the astronomy community, HOPS will provide a range of tools for dealing with millimeter-VLBI data, which of course will also be largely applicable to centimeter-wavelength data. The support of mm-VLBI in HOPS is provided as part of the recently instituted Coordinated Millimeter VLBI Array (CMVA). These tools in some cases incorporate new, optimized algorithms for weak signal detection and verification. A sampling of current plans: 1. A FITS spigot will be inserted into "fourfit". The format to be supported will be the NRAO-defined radio interferometry format, as used for the VLBA. The intention is to provide the data with or without the results of the fringe-fitting performed by "fourfit". This will be the easiest path from a MkIV-style correlator into AIPS or AIPS++ for strong-source data. The FITS spigot has been deferred until fourfit undergoes major surgery for Mk4-style binary data file formats in the second half of 1996. 2. An optimized fringe detector, involving incoherent averaging and bispectral averaging, with a new fringe fitting program. This fringe detection system takes the form of a handful of separate programs, some of which are primarily of use only to the short coherence-time situation prevalent in millimeter-VLBI. The output format of this sub-package will initially be an ascii format as currently used by the Caltech package program "mk3merge". Either a FITS output format or an AIPS/AIPS++ program capable of reading the ascii format will be devised. Much of the mm-VLBI data path is already complete in v2.2, though certain refinements such as fit globalization and automation of certain steps will have to wait for v2.3 (which may be renamed v3.0 if it includes Mk4 file formats). The emphasis of the HOPS package will continue to be on the capability of examining the data in great detail, and with maximum ease. A primary objective is to provide a powerful diagnostic capability, wherein problems manifested at high levels (e.g. as in patterns of poor or discrepant results in large datasets) can be flexibly and efficiently investigated in fine detail. This capability is considered mandatory for a software package charged with handling data from a MkIII/MkIV VLBI correlator. In addition, attention will be paid to providing very rapid feedback on the results of ongoing correlations, to maximize operational efficiency. We regard most of the capabilities of HOPS as complementary to those of AIPS, AIPS++ and the Caltech package. There are no current plans to add significant calibration, model-fitting or mapping capabilities to HOPS.