--------------------------------------------------------------------------------- IMPORTING SINGLE DISH DATA INTO THIS PACKAGE --------------------------------------------------------------------------------- Here I describe how you go about importing ANY single dish data in some sort of .fits format into this package. I have successfully ported FCRAO data, ancient (and modern) Arecibo data, and CLASS .fits data. But all this code is hacked and I am not volunteering to do this for you. Below I describe how you can do this for yourself... Although written for GBT data processed by SDFITS v1.1, it is very straightforward to import any .fits single dish data. (Or, perhaps with less utility vis a vis the package's analysis capabilities, ANY .fits format data.) ------------------------------------------------------------------------------------ OVERVIEW OF THE DEFAULT BEHAVIOR ------------------------------------------------------------------------------------ At startup, the package assumes that the input file of telescope data is an SDFITS v1.1 .fits spectral line data file where the file was created using the parameters -backend=acs -mode=avg That is, it assumes the backend is the autocorrelation spectrometer (acs) and that all the individual correlator dumps associated with a given scan have been averaged. The IDL command: --------------------------------------- start_gbt,'../data/o04_d2.avg.acs.fits' --------------------------------------- Syntax: start_gbt,"fully_qualified_SDFITS_file_name" is normally the first command executed and if no SDFITS file name is given it prompts for one. START_GBT initializes the entire package and invokes two other procedures that together define the package's internal IDL data structure called {gbt_data} and then translate the SDFITS data into this format, creating the ONLINE data file which is what the package uses for all subsequent data i/o. These procedures are: INIT_DATA and MAKE_ONLINE --------- init_data Reads input SDFITS filled data file --------- Sets parameters of the IDL GBT data structure ==> Invoked by START_GBT Syntax: INIT_DATA,SDFITS_datafile ---------------------------------- INIT_DATA's sole function is to establish the size (number of identical records) of the data file and the number of data points (number of spectral channels). To do this it uses the ASTROLIB routine MRDFITS to read the file attached to a logical unit (lun) and create and IDL structure called 'sdd' which is an array of identical structures with dimension set by the size of the data file: sdd = mrdfits(lun, 1, hdr, status=status) ; makes data structure array !kount=n_elements(sdd) <= global variable !kount is the data file size nchan=n_elements(sdd[i].data) <= code examines the # data points in each element of the sdd[] data structure array !data_points=nchan <= global variable setting number of spectral channels per spectrum ------------------------------------------------------------------------------ ==> N.B. THIS PACKAGE REQUIRES THAT THE NUMBER OF CHANNELS BE CONSTANT FOR ANY GIVEN IDL SESSION. IF YOU WANT TO CHANGE THIS VALUE YOU NEED TO EXIT AND RESTART ----------------------------------------------------------------------------- Next an IDL macro, @gbt_data is invoked which uses the value of !data_points to define the packages internal data structure called {gbt_data} Then make_ONLINE is called: make_ONLINE.pro create ONLINE data file using the GBT data structure format --------------- recs_per_scan is the number of subcorrelators per scan# This is the number of independent spectra the backend produces per scan number. It is used, for example, to find the correct OFF record for total power position switched data. INPUT data is from SDFITS filled data file !datafile defined by init_data.pro which MUST be run before make_ONLINE ------------- Syntax: make_ONLINE,recs_per_scan make_ONLINE gives asks you whether or not you wish to make an ONLINE file (in case you have screwed up and are about to overwrite one!) If you do create the ONLINE file, you do this on an sdd[] record by record basis via the procedure: SD_TO_GBT sd_to_gbt.pro Convert the SDFITS structure into the {gbt_data} GBT ------------- data structure. Does this on a record by record basis for SPECTRAL LINE data. 'in' is SDFITS data structure input (sdd[]) 'out' is {gbt_data} data structure output => Invoked by MAKE_DATA and MAKE_ONLINE Syntax: SD_TO_GBT,in,out Here is what the critical part of the MAKE_ONLINE code does: ; convert SDFITS format to GBT format sdd = mrdfits(lun, 1, hdr, status=status) ; makes SDFITS data structure array out={gbt_data} ; defines an output structure ; for i=0L,(!kount-1) do begin ; loop through all sdd[] elements in=sdd[i] ******************** sd_to_gbt,in,out <===== LOOK HERE IT IS ! ******************** writeu,!onunit,out <===== write each record to the ONLINE data file !kon=!kon+1 ; increment number of ONLINE data records endfor ; NOTE: 1. each spectrum is now a distinct data record, just like CLASS. Suppose your spectrometer configuration has 4 subcorrelators (2 polarizations at two different frequencies, e.g.). This means that each data 'scan' will produce 4 {gbt_data} IDL data records in the ONLINE data file ====> 2. THIS IS HOW YOU IMPORT YOUR DATA! YOU CREATE YOUR OWN VERSION OF SD_TO_GBT THAT MAPS YOUR INPUT .FITS DATA INTO {gbt_data} ------------------------------------------------------------------------------------ HOW TO IMPORT YOUR DATA ------------------------------------------------------------------------------------ 1. You will need to modify: start_gbt init_data make_ONLINE sd_to_gbt I strongly suggest that you copy start_gbt and then edit yourself a new version. Unless otherwise instructed, do NOT change the order of the procedures and @macro files invoked here... Doing so voids your non-existant guarantee. 2. To import data you need to look at your input .fits data file closely. The best approach is to run IDLDE which allows you to see the actual data values in the sdd[] array. Here's how to do this in IDLDE: ; Read the .fits data file fname='fully qualified input .fits data file name' openr,lun,fname,/get_lun sdd = mrdfits(lun, 1, hdr, status=status) ; makes the sdd[] data structure array ; with dimension the total number of ; data records free_lun,lun a. Now examine the local variables listed in the IDLDE interface: You should see SDD defined as a structure array with the name {anonymous} and dimension Array[#] This # is the value you should set !kount=# Use !kount=n_elements(sdd) if you want to put this inside a procedure. b. Now click on the box with the + sign next to SDD in the local variable interface. This now points to another {anonymous} structure with the name [0] which is in fact the first (0th record in IDL) data record in SDD. Click on the box for [0] and, viola, now you can see the name, type, and *value* of each element of your input .fits data stream. If your input .fits data are anywhere near the nominal SDFITS standard this record should contain an array called DATA defined as single precision floating point numbers and IDLDE shows you the number of data points, i.e. spectral channels in your data. This # is the value you should set !data_points=# Use !data_points=n_elements(sdd[i].data) if you want to put this in a procedure. Here i is the record location in the sdd[] structure. 3. Now you are ready to modify INIT_DATA You have two options. a. If your .fits file had the element sdd[i].data data array variable, then you do absolutely nothing. Things will work. Your input file will be scanned and the procedure will warn you if different values for !data_points are found. If they are, you need to do some editing that is beyond the scope of this tutorial. It is straightforward, however, to write an IDL procedure to filter the sdd[] file and create other structures that are homogenous w.r.t. the number of spectral channels. These then can be processed by the package individually. The place for this filter is inside MAKE_ONLINE inside the SD_TO_GBT loop. You put the write statement inside a filter which selects records having only a single, fixed data array size. b. If your .fits file has another name for the data array, then you need to modify this line: nchan=n_elements(sdd[i].data) to correspond to whatever non-standard name your file uses. => If there is no data array then you are looking at continuum data perhaps? 4. NOW, the critical issue is to modify SD_TO_GBT.pro to accomodate your input .fits file which is now contained in the sdd[] structure array with the IDL {gbt_data} structure: a. First, realize that, over broad parameters, it does not much matter what you translate: the package is very robust. Yes, you need to translate certain critical information such as: the DATA array (!), center channel, delta_frequency_per_channel, source_name, scan_number, etc. But, this is a *very* minimalist subset of the {gbt_data} structure's current definition within the package. Said definition was set by both what GB SDFITS v1.1 was passing into the .fits file, and what I, Tom Bania, thought was needed in any single-dish database: a full header and array of spectral intensity values. b. Therefore, the strategy here, to import your particular single-dish data, is to re-write the SD_TO_GBT.pro to translate your data into the {gbt_data} structure. TAKE HEART! YOU WILL BE SURPIRSED. Point being: look at the contents of {gbt_data} in the GBT_DATA file: rewrite the procedure sd_to_gbt.pro and map as many of the variables as you can. Then try to use the package. You will be happily surprised to discover that most of the functionality works. It just works. c. Should you wish to augment the definition of the IDL internal data structure {gbt_data}, you most certainly can. Just be sure to ADD new variables, do NOT expunge any of those already there. You will find that the package works, no matter what the definition of {gbt_data} is, precisely, so far as a certain subset of internal variables are preserved. What is that subset, exactly? Got me at the moment: if you are reading this and, in fact, port other telescope .fits files into this package, then I would be VERY grateful should you share with me your experience..... 5. To be specific, here is the current code for SD_TO_GBT.PRO procedure: Note: 1. This is specific to the GBT ACS backend and SDFITS v1.1 input data. 2. The IDL package MUST have the {gbt_data} structure be of a fixed length. All of the i/o depends on this. In turn this requires that ALL strings be packed into {gbt_data} converted into ASCII coded byte data. Hence the code below for strings e.g. out.source=byte(in.object) <= converts source character string from SDFITS input into byte array for {gbt_data} ;+ ; ; sd_to_gbt.pro Convert the SDFITS structure into the internal ; ------------- GBT data structure {gbt_data} ; Does this on a record by record basis at present ; 'in' is SDFITS data structure input sdd[] ; 'out' is {gbt_data} data structure output ; => Invoked by MAKE_DATA and MAKE_ONLINE ; ; For SDFITS v1.1 ;- pro sd_to_gbt,in,out ; out.source=byte(in.object) out.observer=byte(in.observer) out.obsid=byte(in.obsid) out.scan_num=in.scan out.last_on =in.laston out.last_off=in.lastoff out.procseqn=in.procseqn out.procsize=in.procsize out.line_id=byte(in.sampler) ; get the polarization ID -- the Stokes parameter of the data. case (in.crval4) of 1: out.pol_id=byte('I') 2: out.pol_id=byte('Q') 3: out.pol_id=byte('U') 4: out.pol_id=byte('V') -1: out.pol_id=byte('RR') -2: out.pol_id=byte('LL') -3: out.pol_id=byte('RL') -4: out.pol_id=byte('LR') -5: out.pol_id=byte('XX') -6: out.pol_id=byte('YY') -7: out.pol_id=byte('XY') -8: out.pol_id=byte('YX') else: begin out.pol_id=byte('I') print,'Polarization ID failure -- Stokes I assumed' end endcase ; out.scan_type=byte(in.obsmode) out.lst=in.lst out.date=byte(in.date_obs) out.frontend=byte(in.frontend) out.ra=in.crval2 out.dec=in.crval3 out.az=in.azimuth out.el=in.elevatio out.vel=in.velocity out.sky_freq=in.obsfreq out.rest_freq=in.restfreq out.freqoff=in.foffref1 out.ref_ch=in.crpix1 out.delta_x=in.cdelt1 out.vel_def=byte(in.veldef) out.vframe=in.vframe out.rvsys=in.rvsys out.bw=in.bandwid out.tsys=in.tsys if ( strmid(in.obsmode,13,2) eq 'ON' ) then out.tsys_on = in.tsys $ else out.tsys_off = in.tsys out.tcal=in.tcal out.tintg=in.exposure out.feed=in.feed out.srfeed=in.srfeed out.beamxoff=in.beamxoff out.beameoff=in.beameoff out.sideband=byte(in.sideband) out.data_points=n_elements(in.data) out.data=in.data ; return end ==================================================================================== Finally, the 'minimal subset' of data that needs to be translated in order for the code to work is SMALLER than the number of assignments made here. There is a bit of ideosyncratic code here that translates GBT SDFITS v1.1 output into the IDL {gbt_idl} structure. In fact, some of the variables above are zero. Frankly, I have yet to ascertain just how 'minimal' minimal is.