Working with IRIS Level 2 data in Python
Contents
2. Working with IRIS Level 2 data in Python#
This section is aimed to familiarize the reader with the Python jargon and the main module used in this tutorial: extract_irisL2data.
This module will allow us to:
read IRIS Level 2 files
select spectral regions of interest
load the data of these spectral windows
visualize, inspect, and interact with them in an easy and comfortable way
select points of our interest or discard the data of a selected spectral window
save the selected data and their associated points of interest
load, display or share with other colleagues IRIS data in a simple, robust, and efficient way.
Once the decompressed IRIS Level 2 data are hosted locally, the next step is to read, extract, and inspect them.
We have created a module with different methods (or routines) to this aim.
This module is called extract_irisL2data, and it is used to load and inspect either raster or SJI files.
The extract_irisL2data module has several methods.
The most relevant are:
info_fits: shows specific information about the FITS IRIS Level 2 data file. Such as extension numbers, type and size of the data associated to each extension, the spectral window label of each extension, i.e., the spectral data contained in that number extension, and the observed spectral range. See Section 2.3.only_header: returns the main header or the extension header (through the keywordextension) of an IRIS Level 2 data file, as a “header” object. The fields of the header can be accessed as the keys of a dictionary object, e.g.,hdr['NAXIS1']. See Section 2.3.only_data: returns the data from an extension of the IRIS Level 2 data file through the keywordextension(default is 1). The extension number and the associated spectral data can be obtained with the methodinfo_fits. See Section 2.3.show_lines: shows the spectral windows available in an IRIS Level 2 data file, their dimensions, and their observed spectral range. It returns an array, being each element of this array a string with the information of the spectral window (window_label).raster: a class that builds a “Raster of IRIS” object orRoI. This object contains the data selected by the user from an IRIS Level 2 raster file, some basic information from the headers, and other values that makes its visualization easier. To this aim, the classrasterhas a method especially designed to visualize aRoIcalledquick_look. This method allows to add “Points of IRIS” or “PoI”, i.e., points of the user’s interest from the data into theRoIobject, which can be easily saved and analyzed.SJI: a class that makes a “SJI of IRIS” object orSoI. It is similar to aRoIbut contains the data from an IRIS SJI Level 2 data file. It also has its ownquick_lookmethod, which also allows to add one or more “PoI” can be added to theSoI.load: loads the data for a given spectral window or, in the case of a raster file, for a list of spectral windows.save: a method that allows to write to a file aRoI, aSoI, or only their associatedPoI.show_poi: this method helps us to display aPoIassociated to aRoIor to aSoI.
- This code uses two classes of object,
rasterandSJI, which returns two objects, RoIandSoIrespectively, with 3 attributes and 1 method.- For the “Raster of IRIS” object or
RoI, the attributes and method are: filename: contains the name of the parent raster file.windows: contains the requested observed spectral windows. This means that it is a list with thewindow_labelof the data loaded by the user’s request.raster: a dictionary, which has as many keys as spectral windows have been asked to be load. Thus, each spectral window has its unique descriptive keywindow_label. Each of these keys are themselves a dictionary object with different keys, such asdata,wlwith some information from the header and other useful values like thePoI.flush: a method, puts thedataof all the spectral windowswindow_labelof aRoIstored innumpy.memmaparray into the memory of the system. See Section 2.2.tomemmap: a method, puts thedataof all the spectral windowswindow_labelof aRoIstored in the memory of the system in anumpy.memmaparray. See Section 2.2.quick_look: a method, to inspect interactively the spectral data of theRoI. See Section 2.5.
- For the “SJI of IRIS” object or
SoI, the attributes are: filename: contains the name of the parent SJI file.window: contains the observed spectral windowwindow_label.SJI: a dictionary, which unique key is named with thewindow_label. This key is itself a dictionary containing several keys, including thedata,wl, some information from the header and other useful values, and thePoIlist.flush: a method, puts thedataof the observed spectral windowwindow_labelof aSoIstored innumpy.memmaparray to the memory of the system. See Section 2.2.tomemmap: a method, puts thedataof the observed spectral windowwindow_labelof aSoIstored in the memory of the system in anumpy.memmaparray. See Section 2.2.quick_look: a method to inspect interactively the slit-jaw images contained in theSoI. See Section 2.4.load: a method allows to load the data previously saved by the author. See Section 2.6.
The following diagram describes some of the methods available in extract_irisL2data module:
extract_irisL2data.
|_info_fits Method: prompts information about the extension
| and the data in a of a raster or SJI IRIS
| Level 2 data file
|
|_only_header Method: returns only the main header (default)
| or extension header of a raster or SJI
| IRIS Level 2 data file
|
|_only_data Method: returns only data for and extension number
| (no. 1 by default) of a raster or SJI
| IRIS Level 2 data file
|
|_show_lines Method: returns a list of strings with the
| label of the spectral windows stored
| in an IRIS Level 2 data file |
|
|_load Method: loads a raster or SJI IRIS Level 2 file.
| It returns either a 'RoI' or a 'SoI'.
| It can also load a previously saved 'RoI'
| or a 'SoI'
|
|_raster. Class: builds a 'Raster of IRIS' object or
| | 'Roi' object.
| |_quick_look Method: displays the data of a 'RoI'.
| | Allows to add 'Point(s) of IRIS' or
| | 'PoI' to the 'RoI'
| |_other methods
|
|_SJI. Class: builds a 'SJI of IRIS' object or
| | 'SoI' object.
| |_quick_look Method: displays the data of a 'SoI'.
| | Allows to add 'PoI' to the 'SoI'
| |_other methods
|
|_save Method: saves a 'RoI', a 'SoI' or only their
| associated 'PoI'
|
|_show_poi Method: displays a 'PoI'
The RoI, SoI, and PoI objects have attributes of different kind: arrays, tuples, list, or dictionaries, and some methods.
Note
From now on we will be referring to extract_irisL2data as ei to shorten the text.
In the examples we will also be doing the same, i.e.,
from iris_lmsalpy import extract_irisL2data as ei
ei.save and ei.load methods allows us to save and load these objects, despite having associated methods and a complex structure.
For instance a RoI having data from different spectral range, i.e., storing 3D data cubes with different sizes, and different number of PoI for each spectral window.
Your comments and suggestions about the modules outlined here can be sent to Alberto Sainz Dalda (asainz.solarphysics@gmail.com).
2.1. A quick look at IRIS Level 2 data#
We are going to work with the data corresponding to the NOAA AR 12480. Thus, we are going to download a subset of data observed by IRIS. With the following commands we will download the data, which consist of one raster file and five SJI files corresponding to 5 spectral windows observed by IRIS. Once the files are hosted locally, we will take a look into the header of the files.
The following command lines are the same that the ones in the \(1^{st}\) code block of Section 1. There is no need to execute them again if they have been already run.
>>> from iris_lmsalpy import hcr2fits
# Let's download the data from NOAA AR 12480.
>>> query_text = 'https://www.lmsal.com/hek/hcr?cmd=search-events3&outputformat=json&startTime=2016-01-14T00:00&stopTime=2016-01-15T00:00&minnumRasterSteps=320&hasData=true&minxCen=550&limit=200'
>>> list_urls = hcr2fits.get_fits(query_text)
Requesting the query...
Downloading the file http://www.lmsal.com/solarsoft/irisa/data/level2_compressed/2016/01/14/20160114_230409_3630008076/iris_l2_20160114_230409_3630008076_SJI_1330_t000.fits.gz into /home/user/IRIS_data/ (#1 of 5) ...
...
Decompressing the file iris_l2_20160114_230409_3630008076_SJI_1330_t000.fits.gz into /home/user/IRIS_data/ (#1 of 5) ...
...
Let us recover the header of the raster file and show the description of the observation:
>>> from iris_lmsalpy import extract_irisL2data as ei
>>> raster_filename = 'iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits'
>>> hdr = ei.only_header(raster_filename)
>>> hdr['OBS_DESC']
'Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co'
While the SJI files contain just one spectral window per file, the raster files have several spectral windows per file.
We can know what spectral windows are observed in an IRIS Level 2 data file by using the method ei.show_lines.
>>> SJI_filename = 'iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits'
>>> lines = ei.show_lines(SJI_filename)
Extracting information from file iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits...
Available data with size Y x X x Image are stored in a window labeled as:
-------------------------------------------------------------
Index --* Window label --* Y x X x Im --* Spectral range [AA]
-------------------------------------------------------------
0 SJI_2796 779x1400x80 2794.00 * 2798.00
-------------------------------------------------------------
Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co
>>> raster_filename = 'iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits'
>>> lines = ei.show_lines(raster_filename)
Extracting information from file iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits...
Available data with size Y x X x Wavelength are stored in windows labeled as:
--------------------------------------------------------------------
Index --* Window label --* Y x X x WL --* Spectral range [AA] (band)
--------------------------------------------------------------------
0 C II 1336 779x320x374 1332.66 * 1337.50 (FUV)
1 Fe XII 1349 779x320x244 1347.64 * 1350.79 (FUV)
2 O I 1356 779x320x340 1352.19 * 1356.59 (FUV)
3 Si IV 1394 779x320x411 1390.87 * 1396.08 (FUV)
4 Si IV 1403 779x320x601 1398.60 * 1406.23 (FUV)
5 2832 779x320x113 2831.38 * 2834.23 (NUV)
6 2814 779x320x146 2812.69 * 2816.38 (NUV)
7 Mg II k 2796 779x320x530 2793.14 * 2806.61 (NUV)
--------------------------------------------------------------------
Observation description: Very large dense 320-step raster 105.3x175 320s Deep x 8 Spatial x
>>> # The window labels are stored in the output variable ('lines' in this example)
>>> # as an array of strings.
>>> print(lines)
['C II 1336' 'Fe XII 1349' 'O I 1356' 'Si IV 1394' 'Si IV 1403' '2832'
'2814' 'Mg II k 2796']
Note that the name of the dictionaries returned by ei.load have the same name as the class they belong to (raster and SJI).
These classes of objects have some methods (or functionalities) that we discuss in the following section.
2.2. Reading the IRIS Level 2 data#
Thus, the object returned by ei.load is either a RoI or a SoI.
As the SJI IRIS Level 2 data are simpler than the raster files, since they have only one spectral window per file, we will start here.
The philosophy behind the SoI and RoI is the same, although the latter has a more complex structure.
In this tutorial, for the sake of clarity, we have named iris_sji and iris_raster to the SoI and RoI obtained by loading a SJI and a raster IRIS Level 2 data file respectively with ei.load.
Of course, the user can name the SoI and RoI as they wish.
We use the following command to read and load the data from a SJI IRIS Level 2 file:
>> # To read and load the data from IRIS Level 2 file is pretty simple!
>>> iris_sji = ei.load(SJI_filename)
Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpjm7s8l4k
The previous command creates a SoI object, which contains the data and other values in the following structure:
iris_sji.
|_filename
|_window = [window_label]
|_SJI[window_label].
| |_data [ = SJI data cube ]
| |_wl [ = wavelengths in Angstrom]
| |_clip_ima [ = lower, upper thresholds ]
| |_... more keys
|
|_flush() Method: flushes the data from the
| temporary file to the memory
| of the system
|
|_tomemmap() Method: moves the data from the memory
| of the system to a temporary file
|
|_quick_look() Method: visualizes the data
|
|_other methods
Note
ei.load method loads the data from a raster or SJI IRIS Level 2 in a numpy memory-map array, i.e., a numpy.memmap class.
That means the data are written to a temporary file (in the raster case, as many temporary files as spectral windows are requested).
Thus, the data are not loaded in the memory of the system, but written in a temporary file.
In this way, the memory pressure in the system is much lower, which makes the user experience smoother and more efficient.
For instance, in the following example, the memory required to load all the spectral windows of the raster file is \(~2.66 Gbytes\), while the memory used by the system when the data are loaded through a memory-map array is just \(96 Mbytes\).
On the other hand, this methods requires to have enough space (\(~2.66 Gbytes\) in this example) in a physical storage device to write the temporary files.
These temporary files are removed when the RoI or SoI are “flushed” or deleted.
We use the same command to read the data from a raster IRIS Level 2 data, but it returns a RoI object with the data, wavelengths sampled, and other information stored as:
>>> # To read and load the data from IRIS Level 2 file is pretty simple!
>>> iris_raster = ei.load(raster_filename)
Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpvra72uuz
The previous command only loads the data corresponding to the spectral window “Mg II k 2796”, which is the default option.
Using the keyword window_info of ei.load we can set what spectral window(s) will be loaded.
This keyword is a list with the label of the requested spectral window(s).
The label of the spectral windows observed by IRIS in a given observation are stored in the keyword TDESCn of the main header of the raster file, where n corresponds to the extension in the file where the data of that spectral window are stored.
In addition, it accepts the value window_info=['all'] to load all the available spectral windows in the raster file.
In Section 2.5 we explain how to load multiple spectral windows.
The RoI have the same basic attributes of a SoI, but it includes more attributes devoted to the visualization tool.
iris_raster.
|_filename
|_windows = [sel_window_label_1, ..., sel_window_label_m], m <=n
|_all_windows_in_file = [window_label_1, ..., window_label_n]
|_raster[window_label_1].
| |_data [ = raster data cube ]
| |_wl [ = wavelengths in Angstrom]
| |_clip_ima [ = lower, upper thresholds ]
| |_lim_yplot [ = lower, upper thresholds ]
| |_... more attributes (or keys)
|
|_ ... as many as m-2 selected spectral windows
|
|_raster[window_label_m].
| |_data
| |_wl
| |_clip_ima
| |_lim_yplot
| |_... more attributes (or keys)
|
|_flush() Method: flushes the data from the
| temporary file to the memory
| of the system
|
|_tomemmap() Method: moves the data from the memory
| of the system to a temporary file
|
|_quick_look() Method: visualizes the data
|
|_other methods
Note that windows in a RoI object is a list that may have one or several spectral window labels, while the SoI is a list with a unique spectral window.
If we take a look at the data in the SoI and the RoI, we can see they are numpy.memmap arrays:
>>> print(type(iris_sji.SJI['SJI_2796'].data))
<class 'numpy.memmap'>
>>> print(type(iris_raster.raster['Mg II k 2796'].data))
<class 'numpy.memmap'>
If, for any reason, the user wants to put the data from the temporary file, i.e., from the numpy.memmap array, to the memory of the system, they can use the method flush of the Roi and SoI:
>>> iris_sji.flush()
Removing temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpjm7s8l4k
>>> print(type(iris_sji.SJI['SJI_2796'].data))
<class 'numpy.ndarray'>
>>> iris_raster.flush()
Removing temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpvra72uuz
>>> print(type(iris_sji.SJI['SJI_2796'].data))
<class 'numpy.ndarray'>
As you can see, the flush method closes and removes the temporary files.
This action is automatically done if the variables are deleted, e.g., del iris_raster.
If the user wants to recover the data from the raster or SJI IRIS Level 2 files directly into the memory of the system instead that in a temporary file, the user can just simply set the keyword memmap = False:
>>> iris_sji = ei.load(SJI_filename, memmap = False)
>>> print(type(iris_sji.SJI['SJI_2796'].data))
<class 'numpy.ndarray'>
>>> iris_raster = ei.load(raster_filename, memmap = False)
>>> print(type(iris_sji.SJI['SJI_2796'].data))
<class 'numpy.ndarray'>
If the user has already loaded the data into the memory of the system, like in the previous example, and they want to put the data into a numpy.memmap array:
>>> iris_sji.tomemmap()
Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpwc7pqs10
>>> print(type(iris_sji.SJI['SJI_2796'].data))
<class 'numpy.memmap'>
>>> iris_raster.tomemmap()
Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpyx5pqy3f
>>> print(type(iris_raster.raster['Mg II k 2796'].data))
<class 'numpy.memmap'>
2.3. Low-level access to the headers, the data, and other information#
The RoI and SoI objects obtained with ei.load allow us to inspect the data from the IRIS Level 2 data files in a comfortable, easy way.
These objects have the data for the observed requested data, its wavelengths, and other information as attributes, but not all the information available in the IRIS Level 2 FITS files.
Some users may prefer to recover these information and other one not present in the RoI and SoI objects directly from the FITS file.
ei has some methods to this aim.
First, we need to understand the structure of the IRIS Level 2 FITS data file. The IRIS Level 2 FITS are multi-extension FITS files. An “extension” refers to a part of the file containing self-consistent information. This information will be in the general case, a header and its corresponding data. The first extension is named “primary” and its “extension number” is 0.
The extensions in an IRIS Level 2 SJI FITS file has the following numbers:
0: header and data corresponding to the spectral images observed by the SJI.
1: header and auxiliary 31 values from each exposure taken by the SJI in the spectral band of the file. It is an array of float values with dimensions \(no. images \times 31\).
2: header and extra data from each exposure taken by the SJI in the spectral band of the file. It is a record array containing 5 string fields for each exposure. The values of each field can be access as the key in a dictionary or as an attribute. See example in the last code block of this section.
An IRIS Level 2 raster FITS file has the following extensions:
0: header with the main information of the observation. This header has information about all the spectral windows contained in the file and other relevant information. This extension DOES NOT have spectral data associated.
1toN: header and data for the N spectral windows contained in the file.
N+1: header and auxiliary 47 values from each exposure considered in the file. It is an array of float values with dimensions \(no. acquisitions \times 47\).
N+2: header and extra information data from each exposure considered in the file. It is a record array containing 9 string fields for each exposure. The values of each field can be access as the key in a dictionary or as an attribute. See example in the last code block of this section.
The method ei.info_fits shows the information of the extensions contained in the IRIS Level 2 file.
For the SJI file:
>>> ei.info_fits(SJI_filename)
Filename: iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits
No. Name Ver Type Cards Dimensions Format
0 PRIMARY 1 PrimaryHDU 161 (1400, 779, 80) int16 (rescales to float32)
1 1 ImageHDU 38 (31, 80) float64
2 1 TableHDU 33 80R x 5C [A10, A10, A4, A66, A55]
Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co
Extension No. 1 stores data and header of SJI_2796: 2794.00 * 2798.00 AA
To get the main header use :
hdr = ei.only_header(filename)
To get header corresponding to data of SJI_2796 use :
hdr = ei.only_header(filename, extension = 0)
To get the data of SJI_2796 use :
data = ei.only_data(filename, extension = 0, memmap = True)
and for the raster file:
>>> ei.info_fits(raster_filename)
Filename: iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits
No. Name Ver Type Cards Dimensions Format
0 PRIMARY 1 PrimaryHDU 379 ()
1 1 ImageHDU 33 (374, 779, 320) int16 (rescales to float32)
2 1 ImageHDU 33 (244, 779, 320) int16 (rescales to float32)
3 1 ImageHDU 33 (340, 779, 320) int16 (rescales to float32)
4 1 ImageHDU 33 (411, 779, 320) int16 (rescales to float32)
5 1 ImageHDU 33 (601, 779, 320) int16 (rescales to float32)
6 1 ImageHDU 33 (113, 779, 320) int16 (rescales to float32)
7 1 ImageHDU 33 (146, 779, 320) int16 (rescales to float32)
8 1 ImageHDU 33 (530, 779, 320) int16 (rescales to float32)
9 1 ImageHDU 54 (47, 320) float64
10 1 TableHDU 53 320R x 7C [A10, A10, A3, A10, A4, A66, A66]
Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co
Extension No. 1 stores data and header of C II 1336: 1332.66 * 1337.50 AA (FUV)
Extension No. 2 stores data and header of Fe XII 1349: 1347.64 * 1350.79 AA (FUV)
Extension No. 3 stores data and header of O I 1356: 1352.19 * 1356.59 AA (FUV)
Extension No. 4 stores data and header of Si IV 1394: 1390.87 * 1396.08 AA (FUV)
Extension No. 5 stores data and header of Si IV 1403: 1398.60 * 1406.23 AA (FUV)
Extension No. 6 stores data and header of 2832: 2831.38 * 2834.23 AA (NUV)
Extension No. 7 stores data and header of 2814: 2812.69 * 2816.38 AA (NUV)
Extension No. 8 stores data and header of Mg II k 2796: 2793.14 * 2806.61 AA (NUV)
To get the main header use :
hdr = ei.only_header(filename)
To get header corresponding to data of C II 1336 use :
hdr = ei.only_header(filename, extension = 1)
To get the data of C II 1336 use :
data = ei.only_data(filename, extension = 1, memmap = True)
If we now want to recover the main header of the raster file, and the header and the data corresponding the spectral window “Si IV 1403”.
As extension = 0 is the default option, to get the main header of the file we can just type:
>>> hdr = ei.only_header(raster_filename)
# Let's get the individual header corresponding to Si IV 1403
>>> hdr_SiIV_1403 = ei.only_header(raster_filename, extension = 5)
# Let's get the data corresponding to Si IV 1403
>>> data_SiIV_1403 = ei.only_data(raster_filename, extension = 5)
Creating temporal file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpan9g9ak8
>>> print(type(data_SiIV_1403))
<class 'numpy.memmap'>
Note that ei.only_data can return the data as a numpy.array, i.e. the data are loaded into the memory of the system by using memmap = False in the ei.only_data method, or as numpy.memmap, i.e., the data are stored in a temporary file in the file system (the default option).
In this case, the temporary file is removed from the file system by deleting the associated variable:
>>> del data_SiIV_1403
Because the number of spectral windows in a raster file may vary from an observation to another, a good option to access to the last 2 extensions of the IRIS Level 2 file is to use the negative index in the extension keyword:
# Let's recover the header and the data corresponding to the auxiliary
# information extension
>>> hdr_aux = ei.only_header(raster_filename, extension = -2)
>>> data_aux = ei.only_data(raster_filename, extension = -2)
# Let's recover the data for the extra information extension
>>> data_extra = ei.only_data(raster_filename, extension = -1)
# Let's check the names of the records
>>> data_extra.dtype.names
('FRMID',
'FUVFDBID',
'FUVCRSID',
'NUVFDBID',
'NUVCRSID',
'FUVfilename',
'NUVfilename',
'FUVtemp',
'NUVtemp')
We can access to the values of the variables stored in the data corresponding to the extra information extension as an attribute or as a key:
# Let's access to the record 'FUVfilename'
>>> data_extra.FUVfilename # or data_extra['FUVfilename']
array(['/irisa/data/level1/2016/01/14/H2300/iris20160114_23041396_fuv.fits',
'/irisa/data/level1/2016/01/14/H2300/iris20160114_23042356_fuv.fits',
...
more FUV filenames are displayed
...
'/irisa/data/level1/2016/01/14/H2300/iris20160114_23550181_fuv.fits'],
dtype='<U66')
2.4. Inspection of a SJI IRIS Level 2 data file#
As we have already mentioned, the ei.load method identifies whether the file given as input is a raster or a SJI IRIS Level 2 data file or not.
Let us load the SJI data:
>>> SJI_filename = 'iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits'
>>> iris_sji = ei.load(SJI_filename, verbose = True)
The provided file is a SJI IRIS Level 2 data file containing SJI_2796 data.
Extracting information from file iris_l2_20160114_230409_3630008076_SJI_2796_t000.fits...
Available data with size Y x X x Image are stored in a windows labeled as:
-------------------------------------------------------------
Index --* Window label --* Y x X x Im --* Spectral range [AA]
-------------------------------------------------------------
0 SJI_2796 779x1400x80 2794.00 * 2798.00
-------------------------------------------------------------
Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co
The SJI data are passed to the output variable
(e.g iris_sji) as a dictionary with the following keys:
* iris_sji.SJI['SJI_2796']
In this case, we have named the output SoI object iris_sji.
This object has three attributes:
iris_sji.filename, which contains the parent SJI filenameiris_sji.window, which contains the spectral window in the fileiris_sji.SJI, a dictionary with the key of the spectral window observed.
In this example, “SJI_2796”, this key is itself a Python dictionary with keys storing certain values obtained from the headers (in capital letters), others are calculated for the code (small letters), and the data collected during the observation.
For making the access easier to the keys and the method, they can be called either as a key or as an attribute or method of an object. Let’s see an example:
>>> iris_sji.SJI['SJI_2796'].keys():
dict_keys[
"data"
"wl"
"date_in_filename"
"iris_obs_code"
"raster_info"
"date_time"
"DATE_OBS"
"DATE_END"
"TDET"
"TDESCT"
"TWAVE"
"TWMIN"
"TWMAX"
"SPCSCL"
"SPXSCL"
"SPYSCL"
"POS_X"
"POS_Y"
"SLTPX1IX"
"SLTPX2IX"
"date_time_acq"
"date_time_acq_ok"
"extent_arcsec_arcsec"
"extent_px_px"
"extent_px_arcsec"
"extent_opt"
"extent_opt_coords"
"list_extent"
"list_extent_coords"
"clip_ima"
"_SJI__clip_ima_ini"
"poi"
])
# Accessing to a value as a key in the parent object dictionary
>>> print(iris_sji.SJI['SJI_2796']['date_in_filename'])
'20160114_230409'
# Accessing to a value as an attribute in the parent object dictionary
>>> print(iris_sji.SJI['SJI_2796'].date_in_filename)
'20160114_230409'
The simplest way to start working with the data stored in a SoI is through its attribute data, which is a numpy.ndarray object, i.e., an array object, with all the normal numpy attribute and methods.
Thus, we can operate with that attribute of the SoI object as any other numpy.ndarray object.
Let’s see some examples:
>>> print(type(iris_sji.SJI['SJI_2796'].data))
<class 'numpy.ndarray'>
# Let's see the size and dimensions of the data
>>> print(iris_sji.SJI['SJI_2796'].data.shape)
(779, 1400, 80)
# Let's calculate the maximum value (avoiding NaN values)
# of the 2D array corresponding to the 5th
# position in the 3rd dimension. In Python the
# first index of a dimension in a numpy array starts
# at 0, therefore the index for the 5th position is 4.
>>> import numpy as np
>>> np.nanmax(iris_sji.SJI['SJI_2796'].data[:,:,4])
3574.25
Warning
In Python, a 2D array starts by the upper left corner, being the \(1^{st}\) dimension the one usually associated to the vertical axis, and the \(2^{nd}\) dimension to the horizontal axis.
In IDL, for instance, a 2D array starts by the lower left corner, being the \(1^{st}\) dimension the one usually associated to the horizontal axis, and the \(2^{nd}\) dimension to the vertical axis.
Thus, in the example given, in Python the size of the dimensions of iris_sji.SJI['SJI_2796'].data are “[Y,X,t] = [779, 1400, 80]”
2.5. Inspection of a raster IRIS Level 2 data file#
By default, ei.load loads the spectra corresponding to “Mg II k 2796”.
The keyword window_info allow us to load other spectral windows using a list of window labels.
This keyword has to be a list even if we only want to load a single spectral window, e.g.,
>>> iris_raster = ei.load(raster_filename, window_info = ['Si IV 1394'])
>>> print(iris_raster.windows)
['Si IV 1394']
ei.load make it easier to load the data for one or several spectral windows in different files, even if they are stored in different extension numbers.
Let us load some spectral windows, for instance “Si IV 1394”, “2814”, and “Mg II k 2796”, from the raster file:
>>> raster_filename = 'iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits'
>>> lines = ei.show_lines(raster_filename)
>>> iris_raster = ei.load(raster_filename,
>>> window_info = lines[[3,6,7]],
>>> verbose = True)
The provided file is a raster IRIS Level 2 data file.
Extracting information from file iris_l2_20160114_230409_3630008076_raster_t000_r00000.fits...
Available data are stored in windows labeled as:
--------------------------------------------------------------------
Index --- Window label --- Y x X x WL --- Spectral range [AA] (band)
--------------------------------------------------------------------
0 C II 1336 779x320x374 1332.66 - 1337.50 (FUV)
1 Fe XII 1349 779x320x244 1347.64 - 1350.79 (FUV)
2 O I 1356 779x320x340 1352.19 - 1356.59 (FUV)
3 Si IV 1394 779x320x411 1390.87 - 1396.08 (FUV)
4 Si IV 1403 779x320x601 1398.60 - 1406.23 (FUV)
5 2832 779x320x113 2831.38 - 2834.23 (NUV)
6 2814 779x320x146 2812.69 - 2816.38 (NUV)
7 Mg II k 2796 779x320x530 2793.14 - 2806.61 (NUV)
--------------------------------------------------------------------
Observation description: Large dense 320-step raster 105.3x120 320s Deep x 8 Lossless co
Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpz43rtuc8
Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpvn8vwi7v
Creating temporary file... /var/folders/jn/wjxzjh7n2nv58g9t3qjmgdr8000mky/T/tmpwpuby2h_
The selected data are passed to the output variable (e.g iris_raster) as a dictionary with the following keys:
iris_raster.raster['Si IV 1394']iris_raster.raster['2814']iris_raster.raster['Mg II k 2796']
The spectral windows stored in an output object can be found in the windows attribute.
In our example, the RoI object is called iris_raster, let us take a look at its attributes:
>>> print(type(iris_raster))
<class 'ei.raster'>
# Let's verify the spectral windows loaded
>>> print(iris_raster.windows)
['Si IV 1394' '2814' 'Mg II k 2796']
# The attribute 'raster' of the 'Roi' is a dictionary with keys named
# with the labels of the spectral windows loaded
>>> print(iris_raster.raster.keys())
dict_keys(['Si IV 1394', '2814', 'Mg II k 2796'])
#Let's see the keys in the iris_raster.raster['Mg II k 2796'] dictionary
>>> print(iris_raster.raster['Mg II k 2796'].keys())
dict_keys(['data', 'wl', 'date_in_filename', 'iris_obs_code', 'raster_info',
'DATE_OBS', 'DATE_END', 'TDET', 'TDESCT', 'TWAVE', 'TWMIN', 'TWMAX', 'SPCSCL',
'SPXSCL', 'SPYSCL', 'EXPTIME', 'STEPT_AV', 'POS_X', 'POS_Y',
'date_time_acq', 'date_time_acq_ok', 'number_ext', 'binxy', 'binwl',
'extent_arcsec_arcsec', 'extent_px_px', 'extent_px_arcsec', 'extent_time_px',
'extent_time_arcsec', 'extent_opt', 'extent_opt_coords', 'list_extent',
'list_extent_coords', 'extent_display', 'extent_display_coords',
'clip_ima', 'lim_yplot', 'delay', 'poi', '_raster__count_poi',
'_raster__move_count_poi', '_raster__count_coords', '_raster__z_pos',
'_raster__z_pos_ori', '_raster__z_pos_ext', '_raster__dim_data',
'_raster__xlim1', '_raster__ylim1', '_raster__xlim2', '_raster__ylim2',
'_raster__xlim3', '_raster__ylim3', 'arr_y_ax1', 'arr_x_ax1', 'arr_x_ax3'])
The capital-case keys are directly taken from the original headers.
The small-case keys are derived from the headers or created for the code to be used during the visualization of the RoI.
The more important attributes or keys are data and wl.
Note that they can be accessed as an attribute or as a key:
>>> print(iris_raster.raster['Mg II k 2796'].wl.shape)
(530,)
>>> print(iris_raster.raster['Mg II k 2796']['wl'].shape)
(530,)
The dimensions of the data attribute in a Roi are “[Y,X,spectral_sampling]”, where the spatial dimensions Y and X are the same as in the SJI file, and the “spectral_sampling” corresponds to the sampling in the spectral positions (given in Angstroms), and contained in the variable iris_raster.raster[window_label].wl.
# Let's see the size and dimensions of the data stored in each key of
# the output dictionary.
>>> for i in iris_raster.raster.keys():
>>> print("Size and dimensions of data stored in the "
>>> "iris_raster.raster[\'{}\']: {}".format(i, iris_raster.raster[i].data.shape))
Size and dimensions of data stored in the iris_raster.raster['Si IV 1394']: (779, 320, 411)
Size and dimensions of data stored in the iris_raster.raster['2814']: (779, 320, 146)
Size and dimensions of data stored in the iris_raster.raster['Mg II k 2796']: (779, 320, 530)
2.6. Saving your data and selected points#
Note
For the right functioning of this section and the following ones, the user must have added PoI to the spectral windows of the RoI and SoI by pressing a/A.
The name of the files given through the keyword filename in this section is at user’s discretion, but it has to be the same during the execution of the code.
Note that the extension ‘.jbl.gz’ is automatically added to the filename if this does not have this extension.
The module ei has a method or function devoted to save either a RoI, or a SoI or only the PoI in a simple, easy way.
This method is called save and uses saveall (see Section 1.1) to write these objects into a file:
# Saving the 'iris_raster' object to a file
>>> ei.save(iris_raster)
Saving variable(s) ...
raster2save
... in roi_iris_l2_20160114_230409_3630008076_raster_t000_r00000.jbl.gz
# Saving the 'iris_sji' object to a file
>>> ei.save(iris_sji)
Saving ...
sji2save
... in soi_iris_l2_20160114_230409_3630008076_SJI_2796_t000.jbl.gz
By default, ei.save sets the filename by preceding the original IRIS Level 2 data file with the tags “roi_” or “soi_”, and changes the “.fits” extension to the “.jbl.gz” extension.
If the destination file already exists, the code asks to verify whether the file should be actually overwritten or not.
To avoid this safety question the user can set the keyword force equals True:
# Saving the 'iris_sji' object to an existing file
>>> ei.save(iris_sji)
# Answer 'n'
"File soi_iris_l2_20160114_230409_3630008076_SJI_2796_t000.jbl.gz exists. Do you want to overwrite it? Y/[n]"
>>> ei.save(iris_sji, force = True)
Saving ...
sji2save
... in soi_iris_l2_20160114_230409_3630008076_SJI_2796_t000.jbl.gz
If we want to save our data in another directory than the current one or/and with a different name, we can use the keyword output_dir:
# Saving the 'iris_raster' object to a file called 'roi_noaa_12480_IRIS.myext' in '/scratch/asainz/'
>>> ei.save(iris_raster, filename = 'roi_noaa_12480_IRIS.myext', out_dir='/scratch/asainz/')
Saving variable(s) ...
raster2save
... in /scratch/asainz/roi_noaa_12480_IRIS.myext.jbl.gz
Note that the codes automatically adds the extension .jbl.gz to the filename if this does not have that extension.
We may be interested in saving only the saved PoI in a RoI or a SoI and discard the rest of the data, since they can be read again from the original IRIS Level 2 file.
To this aim, we can use the keyword only_poi.
That will change the original data from the attribute data of RoI or the SoI by an array of 1x1x1 dimension with value 0.
Thus, the RoI and the SoI conserve their structure but save space in disk.
Let’s see an example:
# Saving the 'iris_raster' object to a file in '/scratch/asainz/'
>>> ei.save(iris_raster, only_poi = True)
Saving variable(s) ...
raster2save
... in poi_roi_iris_l2_20160114_230409_3630008076_raster_t000_r00000.jbl.gz
# Saving only the PoI of 'iris_raster' object to a file named 'mypoints_noaa12480_IRIS.myext'
>>> my_poi_filename = 'mypoints_12480_IRIS.myext'
>>> ei.save(iris_raster, filename = my_poi_filename, only_poi = True)
Saving variable(s) ...
raster2save
... in mypoints_12480_IRIS.myext.jbl.gz
2.6.1. Loading your data and selected points#
To load a RoI, a SoI, or a only-PoI object we use the same method that we used for the IRIS Level 2 data.
The code verifies that the input file is a gzip compressed file, and for the tags “roi_”, “soi_”, or “poi_”.
Therefore, we strongly suggest to the user to name the files including these flags.
If for any reason the user do not want to use these tags in the filename, these can be passed with the keywords roi_file and soi_file.
Let us load the files saved during the previous section:
>>> myIRISdata = ei.load('roi_iris_l2_20160114_230409_3630008076_raster_t000_r00000.jbl.gz')
>>> print(type(myIRISdata))
<class 'extract_irisL2data.raster'>
>>> print(myIRISdata.windows)
['C II 1336', 'Si IV 1394', '2832', 'Mg II k 2796']
# Let's check the size of the data of the 'C II 1336' spectral window
>>> print(myIRISdata.raster['C II 1336'].data.shape)
(779, 320, 374)
# Loading an only-PoI RoI object named manually by the user
>>> my_poi_filename = 'mypoints_12480_IRIS.myext.jbl.gz' # Note the extension .jbl.gz
>>> myIRISdata = ei.load(my_poi_filename, roi_file = True)
# Let's check the size of the data of the 'C II 1336' spectral window
>>> print(myIRISdata.raster['C II 1336'].data.shape)
(1, 1, 1)
# Let's check how many PoI have the 'C II 1336' spectral window
>>> print(len(myIRISdata.raster['C II 1336'].poi))
2.6.2. Making figures from your selected data and points#
Once the data are restored, we can easily work again with then using their corresponding quick_look.
For the only-PoI data, this method will do nothing, since it has no data (actually it is 1-point array).
In this case, we can use the method ei.show_poi.
This method can be use with RoI and SoI having PoI or with only-PoI data.
# Showing the 3-panel figure corresponding to the 2nd PoI of the spectral window 'Si IV 1394'
# of the 'RoI' called myIRISdata restored from the file 'mypoints_noaa12480_IRIS.myext.jbl.gz'
>>> ei.show_poi(myIRISdata.raster['Si IV 1394'].poi[1])
ei.show_poi accepts the following keywords:
show_XY_map:bool, it shows the “XY_map” panel only, default isFalse.show_SlitSpectra:bool, it shows the “SlitSpectra” only, default isFalse.show_Spectra:bool, it shows the “Spectra” panel only, default isFalse.show_SJI_slit:bool, it shows the line overplotted over the slit position, default isTrue.colorbar1:bool, it shows/hides the color bar in “XY map” or in the SJI figure, default isTrue.colorbar2:bool, it shows/hides the color bar in “SlitSpectra”, default isFalse.
# Showing the XY-map panel corresponding to the 1st PoI of the spectral window '2832'
# of the 'RoI' called myIRISdata restored from the file 'mypoints_noaa12480_IRIS.myext.jbl.gz'
>>> ei.show_poi(myIRISdata.raster['2832'].poi[0], show_XY_map=True)
# Show the 'PoI' #2 of the spectral window 'SJI_2796' of the 'SoI' called 'iris_sji'
# without the line overplotted on the slit position
>>> ei.show_poi(iris_sji.SJI['SJI_2796'].poi[1], show_SJI_slit=False)