Data Processing Steps and Levels
Geolocation and Level 1A data
AIRS data processing begins with receipt of Level 0 instrument counts. Once the Level 0 data are organized, algorithms perform geolocation refinement and conversion of raw data numbers to engineering units (DN to EU). Finally, the level 1A data are collected into granules. Level 1A products are archived but not publicly available.
Calibration of Level 1B and 1C radiances
Calibration processing uses baseline measurements from an onboard calibration source and from space views to convert instrument units into physical radiance units. AIRS, AMSU and HSB calibrated radiances are also geolocated, using known spacecraft orientation and instrument pointing characteristics. Calibrated and geolocated radiances are archived as Level 1B (L1B) files (see Table 3 below). The calibration algorithms are generally simpler than the retrieval algorithm, and calibration software has been updated less frequently than retrieval software. Note: The first update to the radiances since launch will be in Version 7.
A new release of the AIRS Level 1C (L1C) product (Version 6.7) resamples the AIRS L1B to a common frequency set, fills spectral gaps, and removes obvious radiometric outliers related to co-registration errors (Manning et al., 2019). The AIRS L1C Version 6.7 operates on the Version 5 L1B radiances. The L1C algorithm uses principal component reconstruction (PCR) to adjust the radiances to be more representative of what would be expected without spatial errors. Using the L1C algorithm will result in lower errors for problematic channels in individual spectra. However, SI-traceability of the radiances for L1C is more complex due to the use of the PCR, and Level 1C uncertainties have not yet been estimated.
The AIRS Level 1B and Level 1C radiance products are generated with algorithms developed by the AIRS Project.
Retrieval of Level 2 data from Level 1B radiances: Description and comparison of AIRS-team and CLIMCAPS-Aqua data sets
Development of the AIRS-team Level 2 algorithm began prior to launch, and continued through the release of the V7 data set. The NASA-supported CLIMCAPS Level 2 algorithm was developed to process data from instruments on multiple satellites. An overview and comparison of the different instrument configurations of CLIMCAPS is presented here. The CLIMCAPS Science Application Guide applies to all the configurations and Smith and Barnet (2020) present the AIRS/AMSU version of the CLIMCAPS data set. These data are currently available at the NASA GES/DISC for the Suomi-NPP, JPSS-1, and Aqua satellites. Additional retrieval algorithms have been developed for AIRS, and associated publications are available on the AIRS website.
The AIRS-team retrieval system includes separate software to invert AIRS-Only, AIRS/AMSU and AIRS/AMSU/HSB radiances into estimates of geophysical state, referred to as Level 2 (L2) products. These separate retrieval algorithm paths accommodate the loss of HSB and the degradation of some AMSU channels; see Figure 13, Figure 14a and Figure 14b. CLIMCAPS produces AIRS-only and AIRS/AMSU records, but no AIRS/AMSU/HSB record.
The CLIMCAPS retrieval algorithm is based on AIRS-team Version 5. There are some fundamental differences, however. Examples: CLIMCAPS uses the MERRA reanalysis as the first guess, while Version 5 used a regression based on radiosonde observations; the AIRS-team versions 6 and 7 use a neural network first guess, and all cloud quantities are reported at AIRS resolution; CLIMCAPS uses fewer iterations in clould clearing than the AIRS-team algorithms, as can be seen in a comparison of processing flow in Figure 11 and Figure 12.
A compact comparison between the retrieval choices, CLIMCAPS and the AIRS-team retrievals, is shown in Table 2 below.
Table 2: Summarized comparison of CLIMCAPS and AIRS-team V7.
|
CLIMCAPS |
AIRS-team V7 |
Instruments |
AIRS/AMSU (Aqua) CrIS/ATMS (SNPP, JPSS-1) |
AIRS/AMSU/HSB (Aqua) |
Retrieval Method |
Hybrid Optimal estimation (OE) approach with Singular Value Decomposition regularization on cloud-cleared radiances on AIRS 3x3 fields of regard (FoR) |
Least squares estimation using Singular Value Decomposition regularization on cloud-cleared radiances on AIRS 3x3 FoR |
Retrieval profiles |
Temperature, water vapor, O3, CO, CH4, CO2, HNO3, SO2, and N2O (see this table) |
Temperature, water vapor, O3, CO, CH4 (see Table 3) |
Retrievals in cloudy atmospheres? |
Yes, in partly cloudy atmospheres but not homogeneous FoR |
Yes, in partly cloudy atmospheres but not homogeneous FoR |
A-priori for T, H2O and O3 |
MERRA-2 reanalysis |
Neural network (NN) first guess of clear-sky radiances, temperature, and water vapor layer amount, using overtime and latitude-dependent O3 layer amount climatology |
A-priori for other trace gas profiles |
Climatology |
Climatology |
A-priori error propagation |
Both diagonal and off-diagonal terms of error covariance matrices are propagated for temperature, water vapor and ozone; only diagonal terms of the error covariance matrix are propagated for all other trace gas species |
Only diagonal terms of the error covariance matrix are propagated for all retrieval variables |
Infrared spectral channels |
Channel subsets are selected for each retrieval variable used in OE |
Over 500 IR channels are used in non-linear regression Channel subsets are selected for each retrieval variable |
Latency |
1 month delay due to dependence on MERRA-2 |
Near real-time; No dependence on reanalysis product |
Averaging Kernels |
Averaging kernels for each FoR in state vector elements for temperature, water vapor, O3, CO, CH4, CO2 and HNO3 |
Averaging kernels for each FoR in state vector elements for temperature, water vapor, O3, CO, and CH4 |
Cloud Clearing |
Cloud clearing using clear sky radiance estimate from MERRA-2; no iteration on clear sky radiances |
Clear sky radiances estimate initially based on NN stochastic cloud clearing followed by sequential estimates based on clear sky radiances from retrieved state |
Cloud retrievals |
Two layers of clouds for each FoR. Cloud fractions for each layer are retrieved in each field-of-view (FoV). Cloud-top pressure is retrieved for each FoR (3x3 FoVs). |
Two layers of clouds for each FoR. Cloud fractions and cloud-top pressure for each layer are retrieved in each field-of-view (FoV). Ice cloud optical thickness, phase, and effective radius for each FoV |
Retrieval quantities
Retrieved quantities include surface temperature and emissivity, profiles of temperature and water vapor, trace gas amounts, and cloud top properties. They are listed in Table 3 for the AIRS-team retrieval and CLIMCAPS. Note that the table does not include a large number of quality control flags for retrieved quantities.
Table 3: AIRS Level 1B and Level 2 AIRS-team and CLIMCAPS retrieval quantities. CLIMCAPS has AMSU resolution cloud properties and no phase or effective radius information. A more complete list of retrieved and derived variables for CLIMCAPS is given here in the CLIMCAPS Science Application Guide. *Product resolution here refers to horizontal resolution, about 15 km for the AIRS instrument and about 45 km for AMSU.
Level 1B: Observed Radiances |
Physical Characteristics |
Product Resolution* |
AIRS Radiances |
Upwelling infrared radiances at spacecraft |
AIRS |
AMSU Radiances |
Upwelling microwave radiances at spacecraft |
AMSU |
HSB Radiances |
Upwelling microwave radiances at spacecraft |
HSB (AIRS) |
Visible/Near Infrared |
Images in Vis/NIR |
2.3 km grid over AIRS footprints |
Level 2: Retrieved Quantity |
|
Product Resolution* AIRS-Team/ CLIMCAPS (if different) |
Cloud Cleared IR Radiance |
Spectrally resolved radiance from cloud-free part of AMSU scene. |
AMSU |
Sea Surface Temperature |
Surface property |
AMSU |
Land Surface Temperature |
Surface property |
AMSU |
Spectrally Resolved Land Surface Emissivity |
Surface property |
AMSU |
Temperature Profile |
Height-resolved, surface to mesosphere |
AMSU |
Water Vapor Profile |
Height-resolved, surface to upper troposphere |
AMSU |
Total Precipitable Water |
Total water in retrieved water vapor profile |
AMSU |
Effective Cloud Fraction (product of areal coverage and gray cloud emissivity) |
Cloud top property |
AIRS |
Cloud Top Height |
Cloud top property |
AIRS/AMSU |
Cloud Top Temperature |
Cloud top property |
AIRS/AMSU |
Cloud Phase |
Cloud top property |
AIRS/None |
Cloud Effective Radius |
Cloud top property |
AIRS/None |
Ozone Profile |
Height-resolved, upper troposphere to mesosphere |
AMSU |
Total Ozone |
Total ozone in retrieved ozone profile |
AMSU |
Carbon Monoxide |
Amount in middle troposphere |
AMSU |
Methane |
Amount in middle to upper troposphere |
AMSU |
Outgoing Longwave Radiation |
Radiative flux derived from spectrally resolved radiances. |
AMSU |
Carbon Dioxide |
Mid-troposphere amount |
3×3 AMSU |
Algorithm flow
The retrieval algorithms include a forward radiance model and have three basic steps:
- First guess. An initial estimate of geophysical state. For AIRS-team Version 7 this is a neural network data set trained on several days of model reanalyses. The AIRS retrieval problem is ill posed so several solutions may satisfy the observed radiances. The AIRS-team Version 6 and 7 retrieval algorithms use a neural network to obtain a reasonable initial estimate of geophysical state. Version 5 and earlier versions used a regression algorithm. CLIMCAPS uses MERRA reanalysis as first guess.
- Cloud clearing. An estimate of the clear-scene AIRS radiance within an AMSU footprint containing a 3×3 set of AIRS (and possible HSB) spectra; see Figure 2 and Figure 9. This process is effective for clouds whose mean optical is roughly unity or less over the 3×3 AIRS footprint set. Cloud clearing does not require entirely clear scenes to be effective, but does require differences in clouds between AIRS fields of view within the 3×3 set.
- The physical retrieval. Refinement of the first guess to give an improved estimate of geophysical state, such that radiances modeled on that estimate are a best fit to cloud cleared radiance.
The AIRS-team algorithms iterate cloud clearing and the physical retrieval (steps 2 and 3) while CLIMCAPS performs a single cloud clearing pass. See Figure 11 and Figure 12 for the processing flow for the two different algorithms.
The AIRS-team retrieval algorithm is described in greater detail in Susskind et al. (2003; 2011), and in the Algorithm Theoretical Basis Document (AIRS L2 ATBD), with most recent updates in the latest testing report(AIRS V7 L2 Performance Test and Validation Report). Details of the CLIMCAPS retrieval algorithm are described in Smith & Barnet (2019; 2020).
Figure 11: AIRS-team V7 retrieval flow diagram. Explanations and symbols are explained in Table 4.
Figure 12: CLIMCAPS retrieval flow diagram. This diagram was originally published in the CLIMCAPS Science Application Guide. Acronyms and symbols are explained in Table 4.
Table 4: Symbols and acronyms in the CLIMCAPS and AIRS-team V7 retrieval flow diagrams in Figure 11 and Figure 12.
Table symbol |
Description |
|
α |
Cloud fraction retrieval |
|
ε; εsw; εlw |
Earth surface emissivity; shortwave surface emissivity; longwave surface emissivity |
|
ρ |
Earth surface reflectivity |
|
δT(p)*** |
Delta temperature: T(p) with subset of MW channels minus T(p)** from MW+IR channels. This step tests the quality of the T(p)** retrieval. |
|
CAMEL |
CLIMCAPS implementation of the Combined ASTER and MODIS Emissivity database over Land (Hook, 2019c, 2019a, 2019b) |
|
CC |
Cloud Clearing that includes retrieval of cloud fraction and cloud top pressure |
|
CCR |
Level 2 Cloud Cleared Radiance product |
|
CD flags |
Constituent Detection flags for isoprene, ethane, propylene and ammonia |
|
CH4 |
Methane retrieval on 100 pressure layers using a subset of IR channels |
|
Climatology |
Global representation of atmospheric variables |
|
Cld prop |
Cloud-top, thermodynamic phase, and ice cloud property retrieval |
|
CO |
Carbon monoxide retrieval on 100 pressure layers using a subset of IR channels |
|
CO2 |
Carbon dioxide retrieval on 100 pressure layers using a subset of IR channels |
|
CTP |
Cloud top pressure retrieval |
|
H2O |
Water vapor retrieval on 100 pressure layers using a subset of MW+IR channels |
|
HNO3 |
Nitric acid retrieval on 100 pressure layers using a subset of IR channels |
|
IR |
Infrared |
|
LAC |
Local angle correction of IR radiances within 3 x 3 fields of view |
|
Level 1 |
NASA geolocated, calibrated radiance products for IR and MW measurements |
|
Level 2 |
NASA geophysical products retrieved from Level 1 radiance measurements. |
|
LIQ |
Liquid water path |
|
Masuda |
Infrared sea surface emissivity model: Masuda et al. (1988) as modified by Wu & Smith (1997) |
|
MERRA2 |
Modern-Era Retrospective analysis for Research and Applications Version 2 (GMAO, 2015) collocated in time and space to the CLIMCAPS instrument footprints. |
|
MODIS |
UW‑Madison global MODIS IR baseline‑fit emissivity product |
|
MW |
Microwave |
|
MW QC closure test |
Microwave Quality Control closure test |
|
N2O |
Nitrous oxide retrieval on 100 pressure layers using a subset of IR channels |
|
O3 |
Ozone retrieval on 100 pressure layers using a subset of IR channels |
|
PS |
Surface pressure |
|
QC/err |
Quality Control and error estimates |
|
RET |
Level 2 geophysical retrieval product |
|
SCC/NN |
Stochastic Cloud Clearing / Neural Network retrieval |
|
SO2 |
Sulphur dioxide retrieval on 100 pressure layers using a subset of IR channels |
|
T(p)*; T(p)** |
Temperature retrieval on 100 pressure levels using a subset of MW+IR channels – first retrieval (*); second and final retrieval (**) |
|
TS |
Surface skin temperature |
|
X* |
X here refers to [TS, εsw, ρ, T(p), H2O, εlw, O3]. Before the successive steps solving for each of these variables, the variable being solved for is reset to the a-priori. |
Vertical Representation: Over-Resolved Vertical Grid
The fundamental output of both CLIMCAPS and the AIRS-team retrieval system is Level 2 profile quantities containing 100 vertical layers, and 101 levels between layers including a lowest pressure level of zero. This resolution is used because the AIRS radiative transfer model –the rapid transmittance algorithm, or RTA–, embedded in the retrievals requires fine vertical resolution for numerical stability. However, this vertical resolution is significantly higher than that needed to convey the information content of the input spectra.
The AIRS-team Level 2 retrieval estimates mean water vapor, ozone, carbon monoxide, and methane in 100 layers, and methane in 100 layers and temperature on the levels between layers. AIRS products prior to Version 6 reported only column totals and layer-mean amounts for these gases. The AIRS-team Version 7 Level 2 Standard and Support Products (see immediately below) for these gases are also reported as level products (values at the specific pressure level upon which they are reported).
The level gas amounts are derived from the internal 100-layer quantities by a smoothing spline, tuned to convey information content and atmospheric variability. Please see AIRS V7 L2 Levels Layers Trapezoids for further details.
Following its AIRS-team Version 5 heritage, CLIMCAPS retrieves T(p) as the temperature at each pressure level on a 100-level grid, while it retrieves H2O, O3, CO, CH4, HNO3, N2O, and SO2 (i.e., all gases except CO2) as column densities [molecules/cm2] on 100 pressure layers. Further details pertaining to layers and levels in CLIMCAPS are covered here in the CLIMCAPS Science User Guide.
Level 2 and Level 3: AIRS-team Support and Standard Products and True Vertical Resolution
The AIRS-team data products are reported on two resolutions. The first is the finer 100-layer grid just described, and second is a reduced vertical resolution grid to approximate the true vertical resolution of the observations:
- Support Products are reported at the native retrieval resolution of 100 layers and 101 levels. Their intended uses include calculation of radiances via the AIRS RTA, the calculation of level quantities from layer quantities, beta testing future products and for investigation of the operation of the retrieval algorithm, or analyses of vertical information content. The AIRS averaging kernels (measures of information content) are included in the Support products.
- Standard Products are reported at pressure levels of 1100, 1000, 925, 850, 700, 600, 500, 400, 300, 250, 200, 150, 100, 70, 50, 30, 20, 15, 10, 7, 5, 3, 2.0, 1.5, 1.0, 0.5, 0.2, and 0.1 hPa. Their reduced resolution better reflects the true information content. (The true resolution can be derived using the averaging kernels in the Support Products.) Standard products are intended for most data users as they embody the majority of the information in the retrieval.
These two AIRS Level 2 and Level 3 data products, Support and Standard, are shown in Figure 14a and Figure 15. CLIMCAPS data are not reported on the Standard grid, so only the 100-layer Support product grid is used.
Both AIRS-team and CLIMCAPS data sets have formal methods for applying averaging kernels and verticality functions to convey true vertical resolution. The CLIMCAPS method is described here and in Smith and Barnet (2019), A similar method can be applied to the AIRS-team retrievals to reduce the 100 levels to true resolution and is described in AIRS V7 L2 Levels Layers Trapezoids.
Gridding of Level 2 data to produce Level 3 mapped quantities
As can be seen in Figure 2, the Aqua orbit tracks are not aligned in a north-south direction. Also, the sun-synchronous Aqua orbit means that most locations are sampled twice per day, but with different sampling times for different longitudes.
To simplify interpretation of retrieved quantities, Level 3 (L3) products include summary statistics of geophysical parameters reported on a 1°×1° grid cells over −180.0° to +180.0° longitude and −90.0° to +90.0° latitude, reported daily and calendar monthly.
Level 3 data products are separated into ascending and descending portions of the orbit. Days in AIRS Level 3 products do not span noon to midnight. Instead, the daily time step is chosen so that the longitude range −180 to 180 is covered in exactly 24 hours. Level 3 products are derived and provided separately from both the CLIMCAPS and AIRS-team retrievals.