Data Processing Steps and Levels

Describes the steps involved in processing the AIRS data, from Level 0 to Level 3. Includes material on vertical representation (layers and levels) and vertical resolution, and gridding.

Geolocation and Level 1A data

AIRS data processing begins with receipt of Level 0 instrument counts. Once the Level 0 data are organized, algorithms perform geolocation refinement and conversion of raw data numbers to engineering units (DN to EU). Finally, the level 1A data are collected into granules. Level 1A products are archived but not publicly available.

Calibration of Level 1B and 1C radiances

Calibration processing uses baseline measurements from an onboard calibration source and from space views to convert instrument units into physical radiance units. AIRS, AMSU and HSB calibrated radiances are also geolocated, using known spacecraft orientation and instrument pointing characteristics. Calibrated and geolocated radiances are archived as Level 1B (L1B) files (see Table 3 below). The calibration algorithms are generally simpler than the retrieval algorithm, and calibration software has been updated less frequently than retrieval software. Note: The first update to the radiances since launch will be in Version 7.

A new release of the AIRS Level 1C (L1C) product (Version 6.7) resamples the AIRS L1B to a common frequency set, fills spectral gaps, and removes obvious radiometric outliers related to co-registration errors (Manning et al., 2019). The AIRS L1C Version 6.7 operates on the Version 5 L1B radiances. The L1C algorithm uses principal component reconstruction (PCR) to adjust the radiances to be more representative of what would be expected without spatial errors. Using the L1C algorithm will result in lower errors for problematic channels in individual spectra. However, SI-traceability of the radiances for L1C is more complex due to the use of the PCR, and Level 1C uncertainties have not yet been estimated.

The AIRS Level 1B and Level 1C radiance products are generated with algorithms developed by the AIRS Project.

Retrieval of Level 2 data from Level 1B radiances: Description and comparison of AIRS-team and CLIMCAPS-Aqua data sets

Development of the AIRS-team Level 2 algorithm began prior to launch, and continued through the release of the V7 data set.  The NASA-supported CLIMCAPS Level 2 algorithm was developed to process data from instruments on multiple satellites. An overview and comparison of the different instrument configurations of CLIMCAPS is presented here. The CLIMCAPS Science Application Guide applies to all the configurations and Smith and Barnet (2020) present the AIRS/AMSU version of the CLIMCAPS data set. These data are currently available at the NASA GES/DISC for the Suomi-NPP, JPSS-1, and Aqua satellites. Additional retrieval algorithms have been developed for AIRS, and associated publications are available on the AIRS website.

The AIRS-team retrieval system includes separate software to invert AIRS-Only, AIRS/AMSU and AIRS/AMSU/HSB radiances into estimates of geophysical state, referred to as Level 2 (L2) products. These separate retrieval algorithm paths accommodate the loss of HSB and the degradation of some AMSU channels; see Figure 13, Figure 14a and Figure 14b. CLIMCAPS produces AIRS-only and AIRS/AMSU records, but no AIRS/AMSU/HSB record.

The CLIMCAPS retrieval algorithm is based on AIRS-team Version 5.  There are some fundamental differences, however.  Examples: CLIMCAPS uses the MERRA reanalysis as the first guess, while Version 5 used a regression based on radiosonde observations; the AIRS-team versions 6 and 7 use a neural network first guess, and all cloud quantities are reported at AIRS resolution;  CLIMCAPS uses fewer iterations in clould clearing than the AIRS-team algorithms, as can be seen in a comparison of processing flow in Figure 11 and Figure 12.

A compact comparison between the retrieval choices, CLIMCAPS and the AIRS-team retrievals, is shown in Table 2 below.  

Table 2: Summarized comparison of CLIMCAPS and AIRS-team V7.

 

CLIMCAPS

AIRS-team V7

Instruments

AIRS/AMSU (Aqua) CrIS/ATMS (SNPP, JPSS-1)

AIRS/AMSU/HSB (Aqua)

Retrieval Method

Hybrid Optimal estimation (OE) approach with Singular Value Decomposition regularization on cloud-cleared radiances on AIRS 3x3 fields of regard (FoR)

Least squares estimation using Singular Value Decomposition regularization on cloud-cleared radiances on AIRS 3x3 FoR

Retrieval profiles

Temperature, water vapor, O3, CO, CH4, CO2, HNO3, SO2, and N2O (see this table)

Temperature, water vapor, O3, CO, CH4 (see Table 3)

Retrievals in cloudy atmospheres?

Yes, in partly cloudy atmospheres but not homogeneous FoR

Yes, in partly cloudy atmospheres but not homogeneous FoR

A-priori for T, H2O and O3

MERRA-2 reanalysis

Neural network (NN) first guess of clear-sky radiances, temperature, and water vapor layer amount, using overtime and latitude-dependent O3 layer amount climatology

A-priori for other trace gas profiles

Climatology

Climatology

A-priori error propagation

Both diagonal and off-diagonal terms of error covariance matrices are propagated for temperature, water vapor and ozone; only diagonal terms of the error covariance matrix are propagated for all other trace gas species

Only diagonal terms of the error covariance matrix are propagated for all retrieval variables

Infrared spectral channels

Channel subsets are selected for each retrieval variable used in OE

Over 500 IR channels are used in non-linear regression

Channel subsets are selected for each retrieval variable

Latency

1 month delay due to dependence on MERRA-2

Near real-time; No dependence on reanalysis product

Averaging Kernels

Averaging kernels for each FoR in state vector elements for temperature, water vapor, O3, CO, CH4, COand HNO3  

Averaging kernels for each FoR in state vector elements for temperature, water vapor, O3, CO, and CH4

Cloud Clearing

Cloud clearing using clear sky radiance estimate from MERRA-2; no iteration on clear sky radiances

Clear sky radiances estimate initially based on NN stochastic cloud clearing followed by sequential estimates based on clear sky radiances from retrieved state

Cloud retrievals

Two layers of clouds for each FoR. Cloud fractions for each layer are retrieved in each field-of-view (FoV). Cloud-top pressure is retrieved for each FoR (3x3 FoVs).

Two layers of clouds for each FoR. Cloud fractions and cloud-top pressure for each layer are retrieved in each field-of-view (FoV). Ice cloud optical thickness, phase, and effective radius for each FoV

Retrieval quantities

Retrieved quantities include surface temperature and emissivity, profiles of temperature and water vapor, trace gas amounts, and cloud top properties. They are listed in Table 3 for the AIRS-team retrieval and CLIMCAPS. Note that the table does not include a large number of quality control flags for retrieved quantities.

Table 3: AIRS Level 1B and Level 2 AIRS-team and CLIMCAPS retrieval quantities. CLIMCAPS has AMSU resolution cloud properties and no phase or effective radius information. A more complete list of retrieved and derived variables for CLIMCAPS is given here in the CLIMCAPS Science Application Guide. *Product resolution here refers to horizontal resolution, about 15 km for the AIRS instrument and about 45 km for AMSU.

Level 1B:

Observed Radiances

Physical Characteristics

Product Resolution*

AIRS Radiances

Upwelling infrared radiances at spacecraft

AIRS

AMSU Radiances

Upwelling microwave radiances at spacecraft

AMSU

HSB Radiances

Upwelling microwave radiances at spacecraft

HSB (AIRS)

Visible/Near Infrared

Images in Vis/NIR

2.3 km grid over AIRS footprints

Level 2:

Retrieved Quantity

 

Product Resolution*

AIRS-Team/ CLIMCAPS (if different)

Cloud Cleared IR Radiance

Spectrally resolved radiance from cloud-free part of AMSU scene.

AMSU

Sea Surface Temperature

Surface property

AMSU

Land Surface Temperature

Surface property

AMSU

Spectrally Resolved Land Surface Emissivity

Surface property

AMSU

Temperature Profile

Height-resolved, surface to mesosphere

AMSU

Water Vapor Profile

Height-resolved, surface to upper troposphere

AMSU

Total Precipitable Water

Total water in retrieved water vapor profile

AMSU

Effective Cloud Fraction (product of areal coverage and gray cloud emissivity)

Cloud top property

AIRS

Cloud Top Height

Cloud top property

AIRS/AMSU

Cloud Top Temperature

Cloud top property

AIRS/AMSU

Cloud Phase

Cloud top property

AIRS/None

Cloud Effective Radius

Cloud top property

AIRS/None

Ozone Profile

Height-resolved, upper troposphere to mesosphere

AMSU

Total Ozone

Total ozone in retrieved ozone profile

AMSU

Carbon Monoxide

Amount in middle troposphere

AMSU

Methane

Amount in middle to upper troposphere

AMSU

Outgoing Longwave Radiation

Radiative flux derived from spectrally resolved radiances.

AMSU

Carbon Dioxide

Mid-troposphere amount

3×3 AMSU

Algorithm flow

The retrieval algorithms include a forward radiance model and have three basic steps:

  1. First guess. An initial estimate of geophysical state. For AIRS-team Version 7 this is a neural network data set trained on several days of model reanalyses. The AIRS retrieval problem is ill posed so several solutions may satisfy the observed radiances. The AIRS-team Version 6 and 7 retrieval algorithms use a neural network to obtain a reasonable initial estimate of geophysical state. Version 5 and earlier versions used a regression algorithm. CLIMCAPS uses MERRA reanalysis as first guess.
  2. Cloud clearing. An estimate of the clear-scene AIRS radiance within an AMSU footprint containing a 3×3 set of AIRS (and possible HSB) spectra; see Figure 2 and Figure 9. This process is effective for clouds whose mean optical is roughly unity or less over the 3×3 AIRS footprint set. Cloud clearing does not require entirely clear scenes to be effective, but does require differences in clouds between AIRS fields of view within the 3×3 set.
  3. The physical retrieval. Refinement of the first guess to give an improved estimate of geophysical state, such that radiances modeled on that estimate are a best fit to cloud cleared radiance.

The AIRS-team algorithms iterate cloud clearing and the physical retrieval (steps 2 and 3) while CLIMCAPS performs a single cloud clearing pass.  See Figure 11 and Figure 12 for the processing flow for the two different algorithms.

The AIRS-team retrieval algorithm is described in greater detail in Susskind et al. (2003; 2011), and in the Algorithm Theoretical Basis Document (AIRS L2 ATBD), with most recent updates in the latest testing report(AIRS V7 L2 Performance Test and Validation Report). Details of the CLIMCAPS retrieval algorithm are described in Smith & Barnet (2019; 2020).

airsV7_alg_flow

Figure 11: AIRS-team V7 retrieval flow diagram. Explanations and symbols are explained in Table 4.

climcaps_alg_flow.png

Figure 12: CLIMCAPS retrieval flow diagram. This diagram was originally published in the CLIMCAPS Science Application Guide. Acronyms and symbols are explained in Table 4.

Table 4: Symbols and acronyms in the CLIMCAPS and AIRS-team V7 retrieval flow diagrams in Figure 11 and Figure 12.

Table symbol

Description

α

Cloud fraction retrieval

ε; εsw; εlw

Earth surface emissivity; shortwave surface emissivity; longwave surface emissivity

ρ

Earth surface reflectivity

δT(p)***

Delta temperature: T(p) with subset of MW channels minus T(p)** from MW+IR channels. This step tests the quality of the T(p)** retrieval.

CAMEL

CLIMCAPS implementation of the Combined ASTER and MODIS Emissivity database over Land (Hook, 2019c, 2019a, 2019b)

CC

Cloud Clearing that includes retrieval of cloud fraction and cloud top pressure

CCR

Level 2 Cloud Cleared Radiance product

CD flags

Constituent Detection flags for isoprene, ethane, propylene and ammonia

CH4

Methane retrieval on 100 pressure layers using a subset of IR channels

Climatology

Global representation of atmospheric variables

Cld prop

Cloud-top, thermodynamic phase, and ice cloud property retrieval

CO

Carbon monoxide retrieval on 100 pressure layers using a subset of IR channels

CO2

Carbon dioxide retrieval on 100 pressure layers using a subset of IR channels

CTP

Cloud top pressure retrieval

H2O

Water vapor retrieval on 100 pressure layers using a subset of MW+IR channels

HNO3

Nitric acid retrieval on 100 pressure layers using a subset of IR channels

IR

Infrared

LAC

Local angle correction of IR radiances within 3 x 3 fields of view

Level 1

NASA geolocated, calibrated radiance products for IR and MW measurements

Level 2

NASA geophysical products retrieved from Level 1 radiance measurements.

LIQ

Liquid water path

Masuda

Infrared sea surface emissivity model: Masuda et al. (1988) as modified by Wu & Smith (1997)

MERRA2

Modern-Era Retrospective analysis for Research and Applications Version 2 (GMAO, 2015) collocated in time and space to the CLIMCAPS instrument footprints.

MODIS

UW‑Madison global MODIS IR baseline‑fit emissivity product

MW

Microwave

MW QC closure test

Microwave Quality Control closure test

N2O

Nitrous oxide retrieval on 100 pressure layers using a subset of IR channels

O3

Ozone retrieval on 100 pressure layers using a subset of IR channels

PS

Surface pressure

QC/err

Quality Control and error estimates

RET

Level 2 geophysical retrieval product

SCC/NN

Stochastic Cloud Clearing / Neural Network retrieval

SO2

Sulphur dioxide retrieval on 100 pressure layers using a subset of IR channels

T(p)*; T(p)**

Temperature retrieval on 100 pressure levels using a subset of MW+IR channels – first retrieval (*); second and final retrieval (**)

TS

Surface skin temperature

X*

X here refers to [TS, εsw, ρ, T(p), H2O, εlw, O3]. Before the successive steps solving for each of these variables, the variable being solved for is reset to the a-priori. 

Vertical Representation: Over-Resolved Vertical Grid

The fundamental output of both CLIMCAPS and the AIRS-team retrieval system is Level 2 profile quantities containing 100 vertical layers, and 101 levels between layers including a lowest pressure level of zero. This resolution is used because the AIRS radiative transfer model  –the rapid transmittance algorithm, or RTA–, embedded in the retrievals requires fine vertical resolution for numerical stability. However, this vertical resolution is significantly higher than that needed to convey the information content of the input spectra.

The AIRS-team Level 2 retrieval estimates mean water vapor, ozone, carbon monoxide, and methane in 100 layers, and methane in 100 layers and temperature on the levels between layers. AIRS products prior to Version 6 reported only column totals and layer-mean amounts for these gases. The AIRS-team Version 7 Level 2 Standard and Support Products (see immediately below) for these gases are also reported as level products (values at the specific pressure level upon which they are reported).

The level gas amounts are derived from the internal 100-layer quantities by a smoothing spline, tuned to convey information content and atmospheric variability. Please see AIRS V7 L2 Levels Layers Trapezoids for further details.

Following its AIRS-team Version 5 heritage, CLIMCAPS retrieves T(p) as the temperature at each pressure level on a 100-level grid, while it retrieves H2O, O3, CO, CH4, HNO3, N2O, and SO2 (i.e., all gases except CO2) as column densities [molecules/cm2] on 100 pressure layers. Further details pertaining to layers and levels in CLIMCAPS are covered here in the CLIMCAPS Science User Guide.

Level 2 and Level 3: AIRS-team Support and Standard Products and True Vertical Resolution

The AIRS-team data products are reported on two resolutions.  The first is the finer 100-layer grid just described, and second is a reduced vertical resolution grid to approximate the true vertical resolution of the observations:

  • Support Products are reported at the native retrieval resolution of 100 layers and 101 levels. Their intended uses include calculation of radiances via the AIRS RTA, the calculation of level quantities from layer quantities, beta testing future products and for investigation of the operation of the retrieval algorithm, or analyses of vertical information content. The AIRS averaging kernels (measures of information content) are included in the Support products.
  • Standard Products are reported at pressure levels of 1100, 1000, 925, 850, 700, 600, 500, 400, 300, 250, 200, 150, 100, 70, 50, 30, 20, 15, 10, 7, 5, 3, 2.0, 1.5, 1.0, 0.5, 0.2, and 0.1 hPa. Their reduced resolution better reflects the true information content. (The true resolution can be derived using the averaging kernels in the Support Products.) Standard products are intended for most data users as they embody the majority of the information in the retrieval.

These two AIRS Level 2 and Level 3 data products, Support and Standard, are shown in Figure 14a and Figure 15.  CLIMCAPS data are not reported on the Standard grid, so only the 100-layer Support product grid is used.

Both AIRS-team and CLIMCAPS data sets have formal methods for applying averaging kernels and verticality functions to convey true vertical resolution. The CLIMCAPS method is described here and in Smith and Barnet (2019), A similar method can be applied to the AIRS-team retrievals to reduce the 100 levels to true resolution and is described in AIRS V7 L2 Levels Layers Trapezoids.

Gridding of Level 2 data to produce Level 3 mapped quantities

As can be seen in Figure 2, the Aqua orbit tracks are not aligned in a north-south direction. Also, the sun-synchronous Aqua orbit means that most locations are sampled twice per day, but with different sampling times for different longitudes.

To simplify interpretation of retrieved quantities, Level 3 (L3) products include summary statistics of geophysical parameters reported on a 1°×1° grid cells over −180.0° to +180.0° longitude and 90.0° to +90.0° latitude, reported daily and calendar monthly.

Level 3 data products are separated into ascending and descending portions of the orbit. Days in AIRS Level 3 products do not span noon to midnight. Instead, the daily time step is chosen so that the longitude range 180 to 180 is covered in exactly 24 hours. Level 3 products are derived and provided separately from both the CLIMCAPS and AIRS-team retrievals.