**************************************** Working with IRIS Level 2 data in Python **************************************** This section is aimed to familiarize the reader with the Python jargon and the main module used in this tutorial: ``extract_irisL2data``. This module will allow us to: * read IRIS Level 2 files * select spectral regions of interest * load the data of these spectral windows * visualize, inspect, and interact with them in an easy and comfortable way * select points of our interest or discard the data of a selected spectral window * save the selected data and their associated points of interest * load, display or share with other colleagues IRIS data in a simple, robust, and efficient way. Once the decompressed IRIS Level 2 data are hosted locally, the next step is to read, extract, and inspect them. We have created a module with different methods (or routines) to this aim. This module is called ``extract_irisL2data``, and it is used to load and inspect either raster or SJI files. The ``extract_irisL2data`` module has several methods. The most relevant are: * ``info_fits``: shows specific information about the FITS IRIS Level 2 data file. Such as extension numbers, type and size of the data associated to each extension, the spectral window label of each extension, i.e., the spectral data contained in that number extension, and the observed spectral range. See :numref:`Low-level access to the headers, the data, and other information`. * ``only_header``: returns the main header or the extension header (through the keyword ``extension``) of an IRIS Level 2 data file, as a "header" object. The fields of the header can be accessed as the keys of a dictionary object, e.g., ``hdr['NAXIS1']``. See :numref:`Low-level access to the headers, the data, and other information`. * ``only_data``: returns the data from an extension of the IRIS Level 2 data file through the keyword ``extension`` (default is 1). The extension number and the associated spectral data can be obtained with the method ``info_fits``. See :numref:`Low-level access to the headers, the data, and other information`. * ``show_lines``: shows the spectral windows available in an IRIS Level 2 data file, their dimensions, and their observed spectral range. It returns an array, being each element of this array a string with the information of the spectral window (``window_label``). * ``raster``: a class that builds a "Raster of IRIS" object or ``RoI``. This object contains the data selected by the user from an IRIS Level 2 raster file, some basic information from the headers, and other values that makes its visualization easier. To this aim, the class ``raster`` has a method especially designed to visualize a ``RoI`` called ``quick_look``. This method allows to add "Points of IRIS" or "PoI", i.e., points of the user's interest from the data into the ``RoI`` object, which can be easily saved and analyzed. * ``SJI``: a class that makes a "SJI of IRIS" object or ``SoI``. It is similar to a ``RoI`` but contains the data from an IRIS SJI Level 2 data file. It also has its own ``quick_look`` method, which also allows to add one or more "PoI" can be added to the ``SoI``. * ``load``: loads the data for a given spectral window or, in the case of a raster file, for a list of spectral windows. * ``save``: a method that allows to write to a file a ``RoI``, a ``SoI``, or only their associated ``PoI``. * ``show_poi``: this method helps us to display a ``PoI`` associated to a ``RoI`` or to a ``SoI``. This code uses two classes of object, ``raster`` and ``SJI``, which returns two objects, ``RoI`` and ``SoI`` respectively, with 3 attributes and 1 method. For the "Raster of IRIS" object or ``RoI``, the attributes and method are: * ``filename``: contains the name of the parent raster file. * ``windows``: contains the requested observed spectral windows. This means that it is a list with the ``window_label`` of the data loaded by the user's request. * ``raster``: a dictionary, which has as many keys as spectral windows have been asked to be load. Thus, each spectral window has its unique descriptive key ``window_label``. Each of these keys are themselves a dictionary object with different keys, such as ``data``, ``wl`` with some information from the header and other useful values like the ``PoI``. * ``flush``: a method, puts the ``data`` of all the spectral windows ``window_label`` of a ``RoI`` stored in ``numpy.memmap`` array into the memory of the system. See :numref:`Reading the IRIS Level 2 data`. * ``tomemmap``: a method, puts the ``data`` of all the spectral windows ``window_label`` of a ``RoI`` stored in the memory of the system in a `numpy.memmap` array. See :numref:`Reading the IRIS Level 2 data`. * ``quick_look``: a method, to inspect interactively the spectral data of the ``RoI``. See :numref:`Inspection of a raster IRIS Level 2 data file`. For the "SJI of IRIS" object or ``SoI``, the attributes are: * ``filename``: contains the name of the parent SJI file. * ``window``: contains the observed spectral window ``window_label``. * ``SJI``: a dictionary, which unique key is named with the ``window_label``. This key is itself a dictionary containing several keys, including the ``data``, ``wl``, some information from the header and other useful values, and the ``PoI`` list. * ``flush``: a method, puts the ``data`` of the observed spectral window ``window_label`` of a ``SoI`` stored in `numpy.memmap` array to the memory of the system. See :numref:`Reading the IRIS Level 2 data`. * ``tomemmap``: a method, puts the ``data`` of the observed spectral window ``window_label`` of a ``SoI`` stored in the memory of the system in a `numpy.memmap` array. See :numref:`Reading the IRIS Level 2 data`. * ``quick_look``: a method to inspect interactively the slit-jaw images contained in the ``SoI``. See :numref:`Inspection of a SJI IRIS Level 2 data file`. * ``load``: a method allows to load the data previously saved by the author. See :numref:`Saving your data and selected points`. The following diagram describes some of the methods available in ``extract_irisL2data`` module: .. code-block:: text extract_irisL2data. |_info_fits Method: prompts information about the extension | and the data in a of a raster or SJI IRIS | Level 2 data file | |_only_header Method: returns only the main header (default) | or extension header of a raster or SJI | IRIS Level 2 data file | |_only_data Method: returns only data for and extension number | (no. 1 by default) of a raster or SJI | IRIS Level 2 data file | |_show_lines Method: returns a list of strings with the | label of the spectral windows stored | in an IRIS Level 2 data file | | |_load Method: loads a raster or SJI IRIS Level 2 file. | It returns either a 'RoI' or a 'SoI'. | It can also load a previously saved 'RoI' | or a 'SoI' | |_raster. Class: builds a 'Raster of IRIS' object or | | 'Roi' object. | |_quick_look Method: displays the data of a 'RoI'. | | Allows to add 'Point(s) of IRIS' or | | 'PoI' to the 'RoI' | |_other methods | |_SJI. Class: builds a 'SJI of IRIS' object or | | 'SoI' object. | |_quick_look Method: displays the data of a 'SoI'. | | Allows to add 'PoI' to the 'SoI' | |_other methods | |_save Method: saves a 'RoI', a 'SoI' or only their | associated 'PoI' | |_show_poi Method: displays a 'PoI' The ``RoI``, ``SoI``, and ``PoI`` objects have attributes of different kind: arrays, tuples, list, or dictionaries, and some methods. .. note:: From now on we will be referring to ``extract_irisL2data`` as ``ei`` to shorten the text. In the examples we will also be doing the same, i.e., .. code-block:: python from iris_lmsalpy import extract_irisL2data as ei ``ei.save`` and ``ei.load`` methods allows us to save and load these objects, despite having associated methods and a complex structure. For instance a ``RoI`` having data from different spectral range, i.e., storing 3D data cubes with different sizes, and different number of ``PoI`` for each spectral window. Your comments and suggestions about the modules outlined here can be sent to Alberto Sainz Dalda (asainz.solarphysics@gmail.com). A quick look at IRIS Level 2 data ================================= We are going to work with the data corresponding to the NOAA AR 12480. Thus, we are going to download a subset of data observed by IRIS. With the following commands we will download the data, which consist of one raster file and five SJI files corresponding to 5 spectral windows observed by IRIS. Once the files are hosted locally, we will take a look into the header of the files. The following command lines are the same that the ones in the :math:`1^{st}` code block of :numref:`Searching and acquiring IRIS Level 2 data`. There is no need to execute them again if they have been already run. .. code-block:: python >>> from iris_lmsalpy import hcr2fits # Let's download the data from NOAA AR 12480. >>> query_text = 'https://www.lmsal.com/hek/hcr?cmd=search-events3&outputformat=json&startTime=2016-01-14T00:00&stopTime=2016-01-15T00:00&minnumRasterSteps=320&hasData=true&minxCen=550&limit=200' >>> list_urls = hcr2fits.get_fits(query_text) Requesting the query... Downloading the file http://www.lmsal.com/solarsoft/irisa/data/level2_compressed/2016/01/14/20160114_230409_3630008076/iris_l2_20160114_230409_3630008076_SJI_1330_t000.fits.gz into /home/user/IRIS_data/ (#1 of 5) ... ... Decompressing the file iris_l2_20160114_230409_3630008076_SJI_1330_t000.fits.gz into /home/user/IRIS_data/ (#1 of 5) ... ... Let us recover the header of the raster file and show the description of the observation: .. code-block:: python >>> from iris_lmsalpy import extract_irisL2data as ei >>> raster_filename = 'iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits' >>> hdr = ei.only_header(raster_filename) >>> hdr['OBS_DESC'] 'Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co' While the SJI files contain just one spectral window per file, the raster files have several spectral windows per file. We can know what spectral windows are observed in an IRIS Level 2 data file by using the method ``ei.show_lines``. .. code-block:: python >>> SJI_filename = 'iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits' >>> lines = ei.show_lines(SJI_filename) Extracting information from file iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits... Available data with size Y x X x Image are stored in a window labeled as: ------------------------------------------------------------- Index --* Window label --* Y x X x Im --* Spectral range [AA] ------------------------------------------------------------- 0 SJI_2796 779x1400x80 2794.00 * 2798.00 ------------------------------------------------------------- Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co >>> raster_filename = 'iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits' >>> lines = ei.show_lines(raster_filename) Extracting information from file iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits... Available data with size Y x X x Wavelength are stored in windows labeled as: -------------------------------------------------------------------- Index --* Window label --* Y x X x WL --* Spectral range [AA] (band) -------------------------------------------------------------------- 0 C II 1336 779x320x374 1332.66 * 1337.50 (FUV) 1 Fe XII 1349 779x320x244 1347.64 * 1350.79 (FUV) 2 O I 1356 779x320x340 1352.19 * 1356.59 (FUV) 3 Si IV 1394 779x320x411 1390.87 * 1396.08 (FUV) 4 Si IV 1403 779x320x601 1398.60 * 1406.23 (FUV) 5 2832 779x320x113 2831.38 * 2834.23 (NUV) 6 2814 779x320x146 2812.69 * 2816.38 (NUV) 7 Mg II k 2796 779x320x530 2793.14 * 2806.61 (NUV) -------------------------------------------------------------------- Observation description: Very large dense 320-step raster 105.3x175 320s Deep x 8 Spatial x >>> # The window labels are stored in the output variable ('lines' in this example) >>> # as an array of strings. >>> print(lines) ['C II 1336' 'Fe XII 1349' 'O I 1356' 'Si IV 1394' 'Si IV 1403' '2832' '2814' 'Mg II k 2796'] Note that the name of the dictionaries returned by ``ei.load`` have the same name as the class they belong to (``raster`` and ``SJI``). These classes of objects have some methods (or functionalities) that we discuss in the following section. Reading the IRIS Level 2 data ============================= Thus, the object returned by ``ei.load`` is either a ``RoI`` or a ``SoI``. As the SJI IRIS Level 2 data are simpler than the raster files, since they have only one spectral window per file, we will start here. The philosophy behind the ``SoI`` and ``RoI`` is the same, although the latter has a more complex structure. In this tutorial, for the sake of clarity, we have named ``iris_sji`` and ``iris_raster`` to the ``SoI`` and ``RoI`` obtained by loading a SJI and a raster IRIS Level 2 data file respectively with ``ei.load``. Of course, the user can name the ``SoI`` and ``RoI`` as they wish. We use the following command to read and load the data from a SJI IRIS Level 2 file: .. code-block:: python >> # To read and load the data from IRIS Level 2 file is pretty simple! >>> iris_sji = ei.load(SJI_filename) Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpjm7s8l4k The previous command creates a ``SoI`` object, which contains the ``data`` and other values in the following structure: .. code-block:: text iris_sji. |_filename |_window = [window_label] |_SJI[window_label]. | |_data [ = SJI data cube ] | |_wl [ = wavelengths in Angstrom] | |_clip_ima [ = lower, upper thresholds ] | |_... more keys | |_flush() Method: flushes the data from the | temporary file to the memory | of the system | |_tomemmap() Method: moves the data from the memory | of the system to a temporary file | |_quick_look() Method: visualizes the data | |_other methods .. note:: ``ei.load`` method loads the data from a raster or SJI IRIS Level 2 in a numpy memory-map array, i.e., a `numpy.memmap` class. That means the data are written to a temporary file (in the raster case, as many temporary files as spectral windows are requested). Thus, the data are not loaded in the memory of the system, but written in a temporary file. In this way, the memory pressure in the system is much lower, which makes the user experience smoother and more efficient. For instance, in the following example, the memory required to load all the spectral windows of the raster file is :math:`~2.66 Gbytes`, while the memory used by the system when the data are loaded through a memory-map array is just :math:`96 Mbytes`. On the other hand, this methods requires to have enough space (:math:`~2.66 Gbytes` in this example) in a physical storage device to write the temporary files. These temporary files are removed when the ``RoI`` or ``SoI`` are "flushed" or deleted. We use the same command to read the data from a raster IRIS Level 2 data, but it returns a ``RoI`` object with the data, wavelengths sampled, and other information stored as: .. code-block:: python >>> # To read and load the data from IRIS Level 2 file is pretty simple! >>> iris_raster = ei.load(raster_filename) Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpvra72uuz The previous command only loads the data corresponding to the spectral window "Mg II k 2796", which is the default option. Using the keyword ``window_info`` of ``ei.load`` we can set what spectral window(s) will be loaded. This keyword is a list with the label of the requested spectral window(s). The label of the spectral windows observed by IRIS in a given observation are stored in the keyword ``TDESCn`` of the main header of the raster file, where ``n`` corresponds to the extension in the file where the data of that spectral window are stored. In addition, it accepts the value ``window_info=['all']`` to load all the available spectral windows in the raster file. In :numref:`Inspection of a raster IRIS Level 2 data file` we explain how to load multiple spectral windows. The ``RoI`` have the same basic attributes of a ``SoI``, but it includes more attributes devoted to the visualization tool. .. code-block:: text iris_raster. |_filename |_windows = [sel_window_label_1, ..., sel_window_label_m], m <=n |_all_windows_in_file = [window_label_1, ..., window_label_n] |_raster[window_label_1]. | |_data [ = raster data cube ] | |_wl [ = wavelengths in Angstrom] | |_clip_ima [ = lower, upper thresholds ] | |_lim_yplot [ = lower, upper thresholds ] | |_... more attributes (or keys) | |_ ... as many as m-2 selected spectral windows | |_raster[window_label_m]. | |_data | |_wl | |_clip_ima | |_lim_yplot | |_... more attributes (or keys) | |_flush() Method: flushes the data from the | temporary file to the memory | of the system | |_tomemmap() Method: moves the data from the memory | of the system to a temporary file | |_quick_look() Method: visualizes the data | |_other methods Note that ``windows`` in a ``RoI`` object is a list that may have one or several spectral window labels, while the ``SoI`` is a list with a unique spectral window. If we take a look at the data in the ``SoI`` and the ``RoI``, we can see they are `numpy.memmap` arrays: .. code-block:: python >>> print(type(iris_sji.SJI['SJI_2796'].data)) >>> print(type(iris_raster.raster['Mg II k 2796'].data)) If, for any reason, the user wants to put the data from the temporary file, i.e., from the `numpy.memmap` array, to the memory of the system, they can use the method ``flush`` of the ``Roi`` and ``SoI``: .. code-block:: python >>> iris_sji.flush() Removing temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpjm7s8l4k >>> print(type(iris_sji.SJI['SJI_2796'].data)) >>> iris_raster.flush() Removing temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpvra72uuz >>> print(type(iris_sji.SJI['SJI_2796'].data)) As you can see, the ``flush`` method closes and removes the temporary files. This action is automatically done if the variables are deleted, e.g., ``del iris_raster``. If the user wants to recover the data from the raster or SJI IRIS Level 2 files directly into the memory of the system instead that in a temporary file, the user can just simply set the keyword ``memmap = False``: .. code-block:: python >>> iris_sji = ei.load(SJI_filename, memmap = False) >>> print(type(iris_sji.SJI['SJI_2796'].data)) >>> iris_raster = ei.load(raster_filename, memmap = False) >>> print(type(iris_sji.SJI['SJI_2796'].data)) If the user has already loaded the data into the memory of the system, like in the previous example, and they want to put the data into a `numpy.memmap` array: .. code-block:: python >>> iris_sji.tomemmap() Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpwc7pqs10 >>> print(type(iris_sji.SJI['SJI_2796'].data)) >>> iris_raster.tomemmap() Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpyx5pqy3f >>> print(type(iris_raster.raster['Mg II k 2796'].data)) Low-level access to the headers, the data, and other information ================================================================ The ``RoI`` and ``SoI`` objects obtained with ``ei.load`` allow us to inspect the data from the IRIS Level 2 data files in a comfortable, easy way. These objects have the data for the observed requested data, its wavelengths, and other information as attributes, but not all the information available in the IRIS Level 2 FITS files. Some users may prefer to recover these information and other one not present in the ``RoI`` and ``SoI`` objects directly from the FITS file. ``ei`` has some methods to this aim. First, we need to understand the structure of the IRIS Level 2 FITS data file. The IRIS Level 2 FITS are multi-extension FITS files. An "extension" refers to a part of the file containing self-consistent information. This information will be in the general case, a header and its corresponding data. The first extension is named "primary" and its "extension number" is 0. The extensions in an IRIS Level 2 SJI FITS file has the following numbers: * ``0``: header and data corresponding to the spectral images observed by the SJI. * ``1``: header and auxiliary 31 values from each exposure taken by the SJI in the spectral band of the file. It is an array of float values with dimensions :math:`no. images \times 31`. * ``2``: header and extra data from each exposure taken by the SJI in the spectral band of the file. It is a record array containing 5 string fields for each exposure. The values of each field can be access as the key in a dictionary or as an attribute. See example in the last code block of this section. An IRIS Level 2 raster FITS file has the following extensions: * ``0``: header with the main information of the observation. This header has information about all the spectral windows contained in the file and other relevant information. This extension **DOES NOT** have spectral data associated. * ``1`` to ``N``: header and data for the N spectral windows contained in the file. * ``N+1``: header and auxiliary 47 values from each exposure considered in the file. It is an array of float values with dimensions :math:`no. acquisitions \times 47`. * ``N+2``: header and extra information data from each exposure considered in the file. It is a record array containing 9 string fields for each exposure. The values of each field can be access as the key in a dictionary or as an attribute. See example in the last code block of this section. The method ``ei.info_fits`` shows the information of the extensions contained in the IRIS Level 2 file. For the SJI file: .. code-block:: python >>> ei.info_fits(SJI_filename) Filename: iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 PrimaryHDU 161 (1400, 779, 80) int16 (rescales to float32) 1 1 ImageHDU 38 (31, 80) float64 2 1 TableHDU 33 80R x 5C [A10, A10, A4, A66, A55] Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co Extension No. 1 stores data and header of SJI_2796: 2794.00 * 2798.00 AA To get the main header use : hdr = ei.only_header(filename) To get header corresponding to data of SJI_2796 use : hdr = ei.only_header(filename, extension = 0) To get the data of SJI_2796 use : data = ei.only_data(filename, extension = 0, memmap = True) and for the raster file: .. code-block:: python >>> ei.info_fits(raster_filename) Filename: iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 PrimaryHDU 379 () 1 1 ImageHDU 33 (374, 779, 320) int16 (rescales to float32) 2 1 ImageHDU 33 (244, 779, 320) int16 (rescales to float32) 3 1 ImageHDU 33 (340, 779, 320) int16 (rescales to float32) 4 1 ImageHDU 33 (411, 779, 320) int16 (rescales to float32) 5 1 ImageHDU 33 (601, 779, 320) int16 (rescales to float32) 6 1 ImageHDU 33 (113, 779, 320) int16 (rescales to float32) 7 1 ImageHDU 33 (146, 779, 320) int16 (rescales to float32) 8 1 ImageHDU 33 (530, 779, 320) int16 (rescales to float32) 9 1 ImageHDU 54 (47, 320) float64 10 1 TableHDU 53 320R x 7C [A10, A10, A3, A10, A4, A66, A66] Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co Extension No. 1 stores data and header of C II 1336: 1332.66 * 1337.50 AA (FUV) Extension No. 2 stores data and header of Fe XII 1349: 1347.64 * 1350.79 AA (FUV) Extension No. 3 stores data and header of O I 1356: 1352.19 * 1356.59 AA (FUV) Extension No. 4 stores data and header of Si IV 1394: 1390.87 * 1396.08 AA (FUV) Extension No. 5 stores data and header of Si IV 1403: 1398.60 * 1406.23 AA (FUV) Extension No. 6 stores data and header of 2832: 2831.38 * 2834.23 AA (NUV) Extension No. 7 stores data and header of 2814: 2812.69 * 2816.38 AA (NUV) Extension No. 8 stores data and header of Mg II k 2796: 2793.14 * 2806.61 AA (NUV) To get the main header use : hdr = ei.only_header(filename) To get header corresponding to data of C II 1336 use : hdr = ei.only_header(filename, extension = 1) To get the data of C II 1336 use : data = ei.only_data(filename, extension = 1, memmap = True) If we now want to recover the main header of the raster file, and the header and the data corresponding the spectral window "Si IV 1403". As ``extension = 0`` is the default option, to get the main header of the file we can just type: .. code-block:: python >>> hdr = ei.only_header(raster_filename) # Let's get the individual header corresponding to Si IV 1403 >>> hdr_SiIV_1403 = ei.only_header(raster_filename, extension = 5) # Let's get the data corresponding to Si IV 1403 >>> data_SiIV_1403 = ei.only_data(raster_filename, extension = 5) Creating temporal file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpan9g9ak8 >>> print(type(data_SiIV_1403)) Note that ``ei.only_data`` can return the data as a `numpy.array`, i.e. the data are loaded into the memory of the system by using ``memmap = False`` in the ``ei.only_data`` method, or as `numpy.memmap`, i.e., the data are stored in a temporary file in the file system (the default option). In this case, the temporary file is removed from the file system by deleting the associated variable: .. code-block:: python >>> del data_SiIV_1403 Because the number of spectral windows in a raster file may vary from an observation to another, a good option to access to the last 2 extensions of the IRIS Level 2 file is to use the negative index in the ``extension`` keyword: .. code-block:: python # Let's recover the header and the data corresponding to the auxiliary # information extension >>> hdr_aux = ei.only_header(raster_filename, extension = -2) >>> data_aux = ei.only_data(raster_filename, extension = -2) # Let's recover the data for the extra information extension >>> data_extra = ei.only_data(raster_filename, extension = -1) # Let's check the names of the records >>> data_extra.dtype.names ('FRMID', 'FUVFDBID', 'FUVCRSID', 'NUVFDBID', 'NUVCRSID', 'FUVfilename', 'NUVfilename', 'FUVtemp', 'NUVtemp') We can access to the values of the variables stored in the data corresponding to the extra information extension as an attribute or as a key: .. code-block:: python # Let's access to the record 'FUVfilename' >>> data_extra.FUVfilename # or data_extra['FUVfilename'] array(['/irisa/data/level1/2016/01/14/H2300/iris20160114_23041396_fuv.fits', '/irisa/data/level1/2016/01/14/H2300/iris20160114_23042356_fuv.fits', ... more FUV filenames are displayed ... '/irisa/data/level1/2016/01/14/H2300/iris20160114_23550181_fuv.fits'], dtype='>> SJI_filename = 'iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits' >>> iris_sji = ei.load(SJI_filename, verbose = True) The provided file is a SJI IRIS Level 2 data file containing SJI_2796 data. Extracting information from file iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits... Available data with size Y x X x Image are stored in a windows labeled as: ------------------------------------------------------------- Index --* Window label --* Y x X x Im --* Spectral range [AA] ------------------------------------------------------------- 0 SJI_2796 779x1400x80 2794.00 * 2798.00 ------------------------------------------------------------- Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co The SJI data are passed to the output variable (e.g iris_sji) as a dictionary with the following keys: * iris_sji.SJI['SJI_2796'] In this case, we have named the output ``SoI`` object ``iris_sji``. This object has three attributes: * ``iris_sji.filename``, which contains the parent SJI filename * ``iris_sji.window``, which contains the spectral window in the file * ``iris_sji.SJI``, a dictionary with the key of the spectral window observed. In this example, "SJI_2796", this key is itself a Python dictionary with keys storing certain values obtained from the headers (in capital letters), others are calculated for the code (small letters), and the data collected during the observation. For making the access easier to the keys and the method, they can be called either as a key or as an attribute or method of an object. Let's see an example: .. code-block:: python >>> iris_sji.SJI['SJI_2796'].keys(): dict_keys[ "data" "wl" "date_in_filename" "iris_obs_code" "raster_info" "date_time" "DATE_OBS" "DATE_END" "TDET" "TDESCT" "TWAVE" "TWMIN" "TWMAX" "SPCSCL" "SPXSCL" "SPYSCL" "POS_X" "POS_Y" "SLTPX1IX" "SLTPX2IX" "date_time_acq" "date_time_acq_ok" "extent_arcsec_arcsec" "extent_px_px" "extent_px_arcsec" "extent_opt" "extent_opt_coords" "list_extent" "list_extent_coords" "clip_ima" "_SJI__clip_ima_ini" "poi" ]) # Accessing to a value as a key in the parent object dictionary >>> print(iris_sji.SJI['SJI_2796']['date_in_filename']) '20160114_230409' # Accessing to a value as an attribute in the parent object dictionary >>> print(iris_sji.SJI['SJI_2796'].date_in_filename) '20160114_230409' The simplest way to start working with the data stored in a ``SoI`` is through its attribute ``data``, which is a `numpy.ndarray` object, i.e., an array object, with all the normal numpy attribute and methods. Thus, we can operate with that attribute of the ``SoI`` object as any other `numpy.ndarray` object. Let's see some examples: .. code-block:: python >>> print(type(iris_sji.SJI['SJI_2796'].data)) # Let's see the size and dimensions of the data >>> print(iris_sji.SJI['SJI_2796'].data.shape) (779, 1400, 80) # Let's calculate the maximum value (avoiding NaN values) # of the 2D array corresponding to the 5th # position in the 3rd dimension. In Python the # first index of a dimension in a numpy array starts # at 0, therefore the index for the 5th position is 4. >>> import numpy as np >>> np.nanmax(iris_sji.SJI['SJI_2796'].data[:,:,4]) 3574.25 .. warning:: In Python, a 2D array starts by the upper left corner, being the :math:`1^{st}` dimension the one usually associated to the vertical axis, and the :math:`2^{nd}` dimension to the horizontal axis. In IDL, for instance, a 2D array starts by the lower left corner, being the :math:`1^{st}` dimension the one usually associated to the horizontal axis, and the :math:`2^{nd}` dimension to the vertical axis. Thus, in the example given, in Python the size of the dimensions of ``iris_sji.SJI['SJI_2796'].data`` are "[Y,X,t] = [779, 1400, 80]" Inspection of a raster IRIS Level 2 data file ============================================= By default, ``ei.load`` loads the spectra corresponding to "Mg II k 2796". The keyword ``window_info`` allow us to load other spectral windows using a list of window labels. This keyword has to be a list even if we only want to load a single spectral window, e.g., .. code-block:: python >>> iris_raster = ei.load(raster_filename, window_info = ['Si IV 1394']) >>> print(iris_raster.windows) ['Si IV 1394'] ``ei.load`` make it easier to load the data for one or several spectral windows in different files, even if they are stored in different extension numbers. Let us load some spectral windows, for instance "Si IV 1394", "2814", and "Mg II k 2796", from the raster file: .. code-block:: python >>> raster_filename = 'iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits' >>> lines = ei.show_lines(raster_filename) >>> iris_raster = ei.load(raster_filename, >>> window_info = lines[[3,6,7]], >>> verbose = True) The provided file is a raster IRIS Level 2 data file. Extracting information from file iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits... Available data are stored in windows labeled as: -------------------------------------------------------------------- Index --- Window label --- Y x X x WL --- Spectral range [AA] (band) -------------------------------------------------------------------- 0 C II 1336 779x320x374 1332.66 - 1337.50 (FUV) 1 Fe XII 1349 779x320x244 1347.64 - 1350.79 (FUV) 2 O I 1356 779x320x340 1352.19 - 1356.59 (FUV) 3 Si IV 1394 779x320x411 1390.87 - 1396.08 (FUV) 4 Si IV 1403 779x320x601 1398.60 - 1406.23 (FUV) 5 2832 779x320x113 2831.38 - 2834.23 (NUV) 6 2814 779x320x146 2812.69 - 2816.38 (NUV) 7 Mg II k 2796 779x320x530 2793.14 - 2806.61 (NUV) -------------------------------------------------------------------- Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpz43rtuc8 Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpvn8vwi7v Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpwpuby2h_ The selected data are passed to the output variable (e.g iris_raster) as a dictionary with the following keys: - ``iris_raster.raster['Si IV 1394']`` - ``iris_raster.raster['2814']`` - ``iris_raster.raster['Mg II k 2796']`` The spectral windows stored in an output object can be found in the ``windows`` attribute. In our example, the ``RoI`` object is called ``iris_raster``, let us take a look at its attributes: .. code-block:: python >>> print(type(iris_raster)) # Let's verify the spectral windows loaded >>> print(iris_raster.windows) ['Si IV 1394' '2814' 'Mg II k 2796'] # The attribute 'raster' of the 'Roi' is a dictionary with keys named # with the labels of the spectral windows loaded >>> print(iris_raster.raster.keys()) dict_keys(['Si IV 1394', '2814', 'Mg II k 2796']) #Let's see the keys in the iris_raster.raster['Mg II k 2796'] dictionary >>> print(iris_raster.raster['Mg II k 2796'].keys()) dict_keys(['data', 'wl', 'date_in_filename', 'iris_obs_code', 'raster_info', 'DATE_OBS', 'DATE_END', 'TDET', 'TDESCT', 'TWAVE', 'TWMIN', 'TWMAX', 'SPCSCL', 'SPXSCL', 'SPYSCL', 'EXPTIME', 'STEPT_AV', 'POS_X', 'POS_Y', 'date_time_acq', 'date_time_acq_ok', 'number_ext', 'binxy', 'binwl', 'extent_arcsec_arcsec', 'extent_px_px', 'extent_px_arcsec', 'extent_time_px', 'extent_time_arcsec', 'extent_opt', 'extent_opt_coords', 'list_extent', 'list_extent_coords', 'extent_display', 'extent_display_coords', 'clip_ima', 'lim_yplot', 'delay', 'poi', '_raster__count_poi', '_raster__move_count_poi', '_raster__count_coords', '_raster__z_pos', '_raster__z_pos_ori', '_raster__z_pos_ext', '_raster__dim_data', '_raster__xlim1', '_raster__ylim1', '_raster__xlim2', '_raster__ylim2', '_raster__xlim3', '_raster__ylim3', 'arr_y_ax1', 'arr_x_ax1', 'arr_x_ax3']) The capital-case keys are directly taken from the original headers. The small-case keys are derived from the headers or created for the code to be used during the visualization of the ``RoI``. The more important attributes or keys are ``data`` and ``wl``. Note that they can be accessed as an attribute or as a key: .. code-block:: python >>> print(iris_raster.raster['Mg II k 2796'].wl.shape) (530,) >>> print(iris_raster.raster['Mg II k 2796']['wl'].shape) (530,) The dimensions of the data attribute in a ``Roi`` are "[Y,X,spectral_sampling]", where the spatial dimensions Y and X are the same as in the SJI file, and the "spectral_sampling" corresponds to the sampling in the spectral positions (given in Angstroms), and contained in the variable ``iris_raster.raster[window_label].wl``. .. code-block:: python # Let's see the size and dimensions of the data stored in each key of # the output dictionary. >>> for i in iris_raster.raster.keys(): >>> print("Size and dimensions of data stored in the " >>> "iris_raster.raster[\'{}\']: {}".format(i, iris_raster.raster[i].data.shape)) Size and dimensions of data stored in the iris_raster.raster['Si IV 1394']: (779, 320, 411) Size and dimensions of data stored in the iris_raster.raster['2814']: (779, 320, 146) Size and dimensions of data stored in the iris_raster.raster['Mg II k 2796']: (779, 320, 530) Saving your data and selected points ==================================== .. note:: For the right functioning of this section and the following ones, the user must have added ``PoI`` to the spectral windows of the ``RoI`` and ``SoI`` by pressing ``a/A``. The name of the files given through the keyword ``filename`` in this section is at user's discretion, but it has to be the same during the execution of the code. Note that the extension '.jbl.gz' is automatically added to the filename if this does not have this extension. The module ``ei`` has a method or function devoted to save either a ``RoI``, or a ``SoI`` or only the ``PoI`` in a simple, easy way. This method is called ``save`` and uses ``saveall`` (see :numref:`Saving your search (and other variables)`) to write these objects into a file: .. code-block:: python # Saving the 'iris_raster' object to a file >>> ei.save(iris_raster) Saving variable(s) ... raster2save ... in roi_iris_l2_20160114_230409_3630008076_raster_t000_r00000.jbl.gz # Saving the 'iris_sji' object to a file >>> ei.save(iris_sji) Saving ... sji2save ... in soi_iris_l2_20160114_230409_3630008076_SJI_2796_t000.jbl.gz By default, ``ei.save`` sets the filename by preceding the original IRIS Level 2 data file with the tags "roi\_" or "soi\_", and changes the ".fits" extension to the ".jbl.gz" extension. If the destination file already exists, the code asks to verify whether the file should be actually overwritten or not. To avoid this safety question the user can set the keyword ``force`` equals `True`: .. code-block:: python # Saving the 'iris_sji' object to an existing file >>> ei.save(iris_sji) # Answer 'n' "File soi_iris_l2_20160114_230409_3630008076_SJI_2796_t000.jbl.gz exists. Do you want to overwrite it? Y/[n]" >>> ei.save(iris_sji, force = True) Saving ... sji2save ... in soi_iris_l2_20160114_230409_3630008076_SJI_2796_t000.jbl.gz If we want to save our data in another directory than the current one or/and with a different name, we can use the keyword ``output_dir``: .. code-block:: python # Saving the 'iris_raster' object to a file called 'roi_noaa_12480_IRIS.myext' in '/scratch/asainz/' >>> ei.save(iris_raster, filename = 'roi_noaa_12480_IRIS.myext', out_dir='/scratch/asainz/') Saving variable(s) ... raster2save ... in /scratch/asainz/roi_noaa_12480_IRIS.myext.jbl.gz Note that the codes automatically adds the extension ``.jbl.gz`` to the filename if this does not have that extension. We may be interested in saving only the saved ``PoI`` in a ``RoI`` or a ``SoI`` and discard the rest of the data, since they can be read again from the original IRIS Level 2 file. To this aim, we can use the keyword ``only_poi``. That will change the original data from the attribute ``data`` of ``RoI`` or the ``SoI`` by an array of 1x1x1 dimension with value 0. Thus, the ``RoI`` and the ``SoI`` conserve their structure but save space in disk. Let's see an example: .. code-block:: python # Saving the 'iris_raster' object to a file in '/scratch/asainz/' >>> ei.save(iris_raster, only_poi = True) Saving variable(s) ... raster2save ... in poi_roi_iris_l2_20160114_230409_3630008076_raster_t000_r00000.jbl.gz # Saving only the PoI of 'iris_raster' object to a file named 'mypoints_noaa12480_IRIS.myext' >>> my_poi_filename = 'mypoints_12480_IRIS.myext' >>> ei.save(iris_raster, filename = my_poi_filename, only_poi = True) Saving variable(s) ... raster2save ... in mypoints_12480_IRIS.myext.jbl.gz Loading your data and selected points ------------------------------------- To load a ``RoI``, a ``SoI``, or a ``only-PoI`` object we use the same method that we used for the IRIS Level 2 data. The code verifies that the input file is a gzip compressed file, and for the tags "roi\_", "soi\_", or "poi\_". Therefore, we strongly suggest to the user to name the files including these flags. If for any reason the user do not want to use these tags in the filename, these can be passed with the keywords ``roi_file`` and ``soi_file``. Let us load the files saved during the previous section: .. code-block:: python >>> myIRISdata = ei.load('roi_iris_l2_20160114_230409_3630008076_raster_t000_r00000.jbl.gz') >>> print(type(myIRISdata)) >>> print(myIRISdata.windows) ['C II 1336', 'Si IV 1394', '2832', 'Mg II k 2796'] # Let's check the size of the data of the 'C II 1336' spectral window >>> print(myIRISdata.raster['C II 1336'].data.shape) (779, 320, 374) # Loading an only-PoI RoI object named manually by the user >>> my_poi_filename = 'mypoints_12480_IRIS.myext.jbl.gz' # Note the extension .jbl.gz >>> myIRISdata = ei.load(my_poi_filename, roi_file = True) # Let's check the size of the data of the 'C II 1336' spectral window >>> print(myIRISdata.raster['C II 1336'].data.shape) (1, 1, 1) # Let's check how many PoI have the 'C II 1336' spectral window >>> print(len(myIRISdata.raster['C II 1336'].poi)) Making figures from your selected data and points ------------------------------------------------- Once the data are restored, we can easily work again with then using their corresponding ``quick_look``. For the ``only-PoI`` data, this method will do nothing, since it has no data (actually it is 1-point array). In this case, we can use the method ``ei.show_poi``. This method can be use with ``RoI`` and ``SoI`` having ``PoI`` or with ``only-PoI`` data. .. code-block:: python # Showing the 3-panel figure corresponding to the 2nd PoI of the spectral window 'Si IV 1394' # of the 'RoI' called myIRISdata restored from the file 'mypoints_noaa12480_IRIS.myext.jbl.gz' >>> ei.show_poi(myIRISdata.raster['Si IV 1394'].poi[1]) ``ei.show_poi`` accepts the following keywords: * ``show_XY_map``: `bool`, it shows the "XY_map" panel only, default is `False`. * ``show_SlitSpectra``: `bool`, it shows the "SlitSpectra" only, default is `False`. * ``show_Spectra``: `bool`, it shows the "Spectra" panel only, default is `False`. * ``show_SJI_slit``: `bool`, it shows the line overplotted over the slit position, default is `True`. * ``colorbar1``: `bool`, it shows/hides the color bar in "XY map" or in the SJI figure, default is `True`. * ``colorbar2``: `bool`, it shows/hides the color bar in "SlitSpectra", default is `False`. .. code-block:: python # Showing the XY-map panel corresponding to the 1st PoI of the spectral window '2832' # of the 'RoI' called myIRISdata restored from the file 'mypoints_noaa12480_IRIS.myext.jbl.gz' >>> ei.show_poi(myIRISdata.raster['2832'].poi[0], show_XY_map=True) # Show the 'PoI' #2 of the spectral window 'SJI_2796' of the 'SoI' called 'iris_sji' # without the line overplotted on the slit position >>> ei.show_poi(iris_sji.SJI['SJI_2796'].poi[1], show_SJI_slit=False)