PeakLab v1 Documentation Contents AIST Software Home AIST Software Support

Modeling Spectra Tutorial II - Powder FTNIR

In this second modeling tutorial, we will generate certain of the models documented in the Modeling Spectra - Part II - FTNIR Data white paper.

You may wish to first explore the first modeling tutorial, Modeling Spectra Tutorial - Liquid UV-VIS, if you have not already done so. It covers many of the basics of direct spectral modeling.

Where as the the Modeling Spectra - Part II - FTNIR Data white paper focuses on many of the intricacies of FTNIR spectral modeling, this tutorial will focus on maximizing the accuracy and robustness when selecting a spectral model.

FTNIR Spectral Modeling

In this tutorial, we will demonstrate the much more challenging FTNIR spectral modeling of powders. We will estimate the total curcuminoids in turmeric powders.

By directly analyzing the powders, we will see a much more complex modeling problem. The reflectance of powders is impacted by both powder particle size and powder moisture levels. The particle size density will vary across the turmeric samples as a consequence of processing, and the NIR spectra for a given turmeric will vary with the moisture (there are two prominent water bands in the NIR).

FTNIR Spectroscopy of Curcumin vs Turmeric - Obfuscating Absorbances

In the Modeling Spectra - Part II - FTNIR Data white paper, we illustrate the rather staggering differences between the curcuminoid FTNIR spectra and that of the turmeric powders which are known to consist of many different molecules, some having NIR absorbances that compete with the two principal bands used for curcuminoid analysis. In this tutorial will we exclusively fit the 1650-1850 nm NIR band.

Fig. 1 Obfuscation of Curcuminoid Target Spectrum in Natural Turmeric (average spectrum from 146 turmeric samples), 1600-1850 nm, Equal Area

In Fig. 1, we repeat the plot from the white paper of the obfuscation of the curcuminoid spectra within the turmeric powders. The blue spectrum is for a curcuminoid standard. The amber curve is the average FTNIR spectrum from 146 different turmeric samples.

Powder vs. Liquid Spectra

We expect this NIR modeling to be much more challenging than the UV-VIS of Part I. We have these obfuscating absorbances in the modeling bands. Turmeric, a natural agricultural product, is known to contain a very large number of components (some authors have reported up to two hundred) and its curcuminoid content and fractions can vary immensely from one sample to the next. The average curcuminoid content is only about 4% of the overall solids in a turmeric powder. On average, 96% of the turmeric solids are non-curcuminoid. Further, all that a compound within the turmeric needs to appear within this 1650-1850 modeling band is a C-H bond. The additional absorbances of non-curcuminoids in the modeling bands are thus not surprising.

In the Modeling Spectra - Part II - FTNIR Data white paper we extensively cover the many other challenges that exist in extracting quantitative information from NIR reflectance powder spectra.

For this modeling, we know we have non-curcuminoid spectral information that will impact the absorbances. We also know that even with basic sieving to remove the larger particles, there can be significant differences in spectral magnitudes of samples that appear virtually identical in the HPLC. Two turmeric samples with with essentially identical curcuminoid chromatography can have very different spectra as a consequence of the optics of powder reflectance as well as non-curcuminoid content. We also have concerns with the nonlinearity of the powder spectra with moisture content.

Resolution and S/N of FTNIR Spectra

In the In the Modeling Spectra - Part II - FTNIR Data white paper, we estimated a Gaussian SD for the curcuminoid standard in FTNIR spectra at 3.6 nm, an exceptional resolution. In the Fourier Estimation of Spectroscopic Resolution topic, we confirm that estimated resolution in a Fourier domain procedure.

We also tested the S/N of individual FTNIR turmeric spectra using the DSP menu's Fourier S/N Estimation and we see an exceptional 0.020-0.025 % estimated white noise in these spectra.

In the Modeling Spectra - Part II - FTNIR Data white paper we explain our rationale for using a 2 nm spacing in our direct spectral fits to this FTNIR data.

Direct Spectral Fit

From the **Model** menu's **Import Data Matrix** item, select the **Import Spectroscopic Data Matrix...**
option. Select the data matrix file **ModelingFTNIR_SpectralData.csv **from the program's installed
default data directory (\PeakLab\Data). Answer **No** when asked if you wish to proceed
to fit this data.

From the **Model** menu's **Graph Data Matrix **item, select the **Graph All Data Sets in
Spectroscopic Data Matrix...** option. Select **1650** for the starting X-column and **1850 **for
the ending X-column and click **OK**.

In the identification portion of the dialog, enter **pctTotal:4** as the search string and click
**Find**. This will highlight all of the data in the file where this substring appears in the data
set identification column(s). In this example, all spectra are highlighted where the total
% curcuminoid ranges from 4.00 to 4.99 %:

Fig. 2 FTNIR Powder Spectra with total curcuminoids 4.00-4.99 highlighted

As you will note, this is not a simple modeling problem. The total percentages vary from near 0
to very 14% in the above set of spectra, and there is no wavelength in the NIR where there is not a wide
range in spectral magnitudes for this 4-5% band that is highlighted. Click the different normalizations
**None**, **Unit Area**, **Unit Magnitude**, **Zero Mean**, and **Mn=0 SD=1** to see if
any of these result in these 4-5% samples converging to a narrow band in the set if spectra. You should
be able to fairly swiftly confirm that this will be a multivariate modeling problem that will consist
of some number of wavelengths as the independent variables. Click **OK** to close.

Once the Import
Data Matrix | Import Spectroscopic Data Matrix... item in the **Model** menu is used to load a
spectroscopic modeling data matrix into the program, the Fit
GLM Models option is available for fitting full-permutation
GLM models, smart
stepwise forward models, and sparse
PLS models, the latter two each using the full permutation models as their starting point.

The PeakLab full permutation GLM models are a simpler and more affective alternative to the PLS (partial least squares) and PCR (principal component regression) procedures.

From the **Model** menu, select the the **Fit Full Permutation GLM Models...** option. This
is the entry point for all predictive
modeling.

Fig 3A. The fit specification for the initial fitting of the turmeric spectra

There are seven sections that govern the modeling.

X Predictors

We want to fit all of the X-predictors through the full eight allowed in the GLM full permutation
algorithm. Click **All** in the X Predictors section to select all of the predictor counts if needed.

X Predictor Filters

Because we will be using such a small X column spacing (2 nm), we wisely use the default filters
to thin the wavelengths fitted as the parameter count increases. Click **Default** so that the filtering
is shown as above.

Stepwise Fits

We will use the defaults. Be sure **Add stepwise fits** is **checked**, the **Maximum count
of x predictors** is set to **12**, the **Stepwise t to enter** is **2.50**, and the **Stepwise
t to remove** is **2.00**.

For Each X Predictor Count

We will also use the defaults. Be sure **Evaluate n best models** is set to **1000**, **Keep
n best predictions** is **100**, and the **sort by** is set to **predicted r2**. These numbers
specify retaining the best 1000 models (by least squares error in the model fits) at each predictor count,
evaluating them for the best r^{2} of prediction, and keeping 100 models for the GLM
Review.

Data

Select the **pctTotal** column in the data file for the **Selected Y column**, **1650**
as the **Starting X column**, and **1850** as the **Ending X column**. Enter 2 for the **X column
spacing**. You should see, in green, a fitting time less than 1 minute. We will begin the fitting with
101 parameters, the spectral values from 1650 to 1850 at an increment of 2 nm. The estimate should reflect
that 60 million actual models will be fitted.

Prediction Errors

Be sure the Leave One Out is selected.

Custom Fit Options

Leave the **Refit to X column spacing = 1 **unchecked. The 2 nm resolution will be sufficient.

Click **OK** to start the fitting.

Fig 3B. The rē progression in the fitting, each bar represents an average of the 10 best models of that predictor count

If you did the UV-VIS of Part I tutorial, you saw a very different outcome from fitting the liquid UV-VIS spectra of the samples solubilized in the chromatography's mobile phase. In the liquid UV-VIS case of total curcuminoid content modeling, there were no competing absorbances and there were no powder measurement issues.

Click **OK** when the fitting is done. Accept the default file name and **Save** the full
fit information to disk. By subsequently importing this file using the Model menu's Import
GLM Models for Review...** **item, you can recall these fits in a single step.

Reviewing the Direct Spectral Fits

You will now see the GLM Review. The GLM Review assembles the full-permutation GLM models, the smart stepwise forward models, and the sparse PLS models for an intelligent selection of a final predictive model.

The two most important windows will be the Model List window and the main GLM Review window containing the model plot and significance map.

Finding the Optimal Count of Predictors (wavelengths)

The first step in the process, however, will usually be the determination of the optimum parameter
count. Click the** Fit Performance** button.

From the window containing the graphs of the fit metrics, click the **Toggle Display of
Reference Data** in the toolbar. This will disable the reference data in the plot which contains the
model fit statistics. We will look only at the prediction statistics.

Fig. 4 The prediction statistics as a function of predictor count

Although we want to see where the SE of prediction is at its best (second panel, 8 WLs), we prefer to look at the information criteria, the Akaike AICc (fourth panel, 8 WLs), the Bayesian BIC (fifth panel, 7 WLs) and the Minimum Description Length MDL (sixth panel, 7 WLs). The average absolute prediction error is also useful (eighth panel, 8 WLs). The metrics are consistent in suggesting that a 7 predictor or 8 predictor model is going to be needed for the best prediction performance.

Click **OK** to close the fit performance window.

Limiting the Models to Only those of Interest

In the Model
List's **Filter menu**, select the **Include 7 Predictor Models** to limit the list to those
with just 7 wavelengths. Also select the **Include 8 Predictor Models**. The list now contains
only models with 7 and 8 wavelengths:

Indx nX Predr2 PredAvErr Model

1 7 0.9411508 0.3023832 Step[7,3](1654,1656,1670,1688,1750,1798,1828)

2 8 0.9409551 0.2956805 Step[8,7](1658,1672,1682,1688,1702,1748,1782,1828)

3 8 0.9408350 0.2994798 Step[8,2](1656,1674,1680,1688,1702,1750,1782,1828)

4 8 0.9408093 0.2953673 Step[8,4](1656,1672,1682,1688,1702,1750,1782,1828)

5 8 0.9405478 0.3041807 Step[8,3](1654,1656,1670,1688,1750,1782,1810,1828)

6 7 0.9402614 0.3019120 Step[7,4](1654,1656,1668,1688,1750,1800,1828)

7 8 0.9402247 0.2964691 Step[8,7](1658,1672,1682,1688,1702,1758,1782,1828)

8 8 0.9401743 0.3061181 Step[8,7](1654,1656,1670,1686,1750,1782,1810,1828)

...

For these data, the best fits, by the 'leave one out' prediction metric, are the stepwise models which begin with various predictor counts of the full permutation fits. For these data, the best overall prediction goodness of fit rē is only going to be about 0.94 as contrasted with the greater than 0.99 observed in the UV-VIS modeling of the same turmeric samples.

If this data matrix is fit to 6-factor, 7-factor, 8-factor, and 9-factor PLS models (NIPALS
algorithm), at a 2nm spacing (101 wavelengths), the leave one out Prediction r^{2} is 0.923, 0.928,
0.926, and 0.925. We have never encountered a spectral data matrix where the PLS or PCR models perform
as well for prediction as the better direct spectral fits.

Note that the intelligent stepwise fits do not use the full permutation filters, so even if a WL is omitted from the permutation matrix by a filter, it may still appear within one of PeakLab's stepwise models. Please note that these are not the simple forward stepwise models you may have fitted in statistics software. Each of these "stepwise" models contains a kind of multidimensional search that seeks to find the best models from a given full permutation model starting point. In this different form of algorithm, a WL predictor that is removed from a model can be added back further along in the process. In general, you will see some benefit in these stepwise models that arise from starting with very effective full permutation models. The stepwise algorithm allows models up to 15 predictors to be fit.

Selecting a Model by Best X-Predictor Significance Compliance

The best model in the fitting, by r^{2} of prediction error, using the leave one out
estimate of prediction error, is a 7-predictor model with a r^{2} of prediction of 0.9412. For
our purposes, we will select a 7-predictor model (as suggested by the BIC and MDL information criterion)
based upon a sort of the models by compliance with the average of the significance at the different wavelengths.

The premise is that the best retained predictive models of a given predictor count will, in the aggregate, contain a strong picture of the most significant predictors for that predictor count. Please see the X-Predictor Compliance section in the Model List topic for a more detailed explanation of this sort metric. You can also see a 3D significance plot across all predictor counts.

From the **Filter** menu in the Model
List, unselect the **Include 8 Predictor Models**. From the **Sort** menu, choose** By X-Predictor
Compliance**. Be sure the first model in the list is selected:

Fig. 5A The selected model plot for the predictive FTNIR model of total curcuminoids

Fig. 5B The selected model and 7-predictor significance plot for the predictive FTNIR model

By requiring the selected fit to consist of highly compliant wavelengths with the retained best 7-predictor count fits, we minimize the possibility of a predictor appearing in the model by random chance.

In the prediction modeling science, one seeks to mitigate the influence of random chance or indirect correlations. A hydrophobic species that is the object of a predictive model, for example, may have an inverse correlation with a hydrophilic species that is also present, but such indirect effects are dangerous in predictions. There may be samples where this additional entity isn't even present, and others where the inverse correlation doesn't apply. You want to have the wavelength band of the species being estimated well represented in the model. In a complex model such as this, where powder moisture, core particle size, and aggregation must also be managed by the model, there will be a mix of positively and negatively correlated spectral wavelengths.

In seeking a model with a high measure of wavelength compliance with the retained best 7-predictor models, we selected a 7-predictor model (Fig. 5A) with an rē of 0.9361 and an average prediction error of 0.3017 % total curcuminoid.

Removing Outliers and Refitting

By visual inspection, there is one prediction that is colored red (>3SE) and one colored yellow (>2SE). These may be outliers that contain an error in the chromatography or spectroscopy. They may as esily be outliers in the samples, quite distinct and apart from all of the other samples in the study. In general, you should manually inspect such samples for the chromatography and spectroscopy replicates to see if the deviation is process related. If not, you should study the chemistry of these samples to learn why of nearly 150 distinct samples, these two could not be effectively modeled. For the purpose of this tutorial, we will assume these two samples are a result of process error. We will treat them as outliers and refit the data with an outlier removal step. Even if we suspect the overall goodness of fit is unlikely to change, it is possible the wavelengths in the best models will follow a certain pattern to keep these two residuals as small as possible. Without that constraint, the wavelength map be more accurate.

Click the **Data** button to open the GLM
Data window. In the **Content** menu, be sure **SE Prediction Intervals** is checked. In the
**Outliers** menu, be sure only **Flag Outside Prediction Interval** is checked and that **Prediction
Interval Confidence** is set to **95 Confidence**. You should see '5 Flagged Outliers' in the Data
window's title bar.

Highlighted spectra are informational only. The metric producing the highlight state will appear in bold.

Flagged spectra are those which will be removed when the GLM
Data window's **File** menu's **Save Data with Flagged Outliers Removed** is used. When a spectrum
is flagged, all displayed metrics will be grayed and it will be shown with an asterisk.

Use the GLM
Data window's **File** menu's **Save Data with Flagged Outliers Removed** item and enter the
any name and location for the new data file. It will contain 142 spectra instead of 147. The 95% prediction
interval criterion resulted in the removal of five of the spectra.

Click **OK** in the main GLM
Review window. From the program's main **Model** menu, use the **Import Data Matrix | Import Spectroscopic
Data Matrix** and then select the file you just saved with the outliers removed.

From the **Model** menu, once again select the the **Fit Full Permutation GLM Models...**
option. The options previously set should be present. Click **OK** to begin the fitting.

Fig 6. The rē progression in the fitting with the data where the outliers were removed

As you will note, the r^{2} goodness of fit averages have significantly improved with
the removal of these five points we treated as outliers.

Click **OK** when the fitting is done and specify a file name of your choosing to
**Save** this full fit information to disk.

Selecting a Model by Multiple Criteria

From the **Filter** menu in the **Model List**, check **Single Predictor
Count** and then check the **Include 7 Predictor Models**. In the **List** menu be sure
**Avg Error of Prediction**, **Median Error of Prediction**, and **XPredictor Compliance** are
checked. From the **Sort** menu, choose **By Predicted Average Error**.

Fig 7. Model list for the data fitted with potential outliers removed

We will choose a model with a good prediction average error and prediction median
error, and also one whose WL's are strongly represented in the retained set of 7-predictor models. Select
the Model with the Compliance of .66 at index **43** as shown above.

Fig. 8A The model plot for the selected model

Fig. 8B The 7-predictor significance plot for the chosen model, the model in yellow, the map of all retained 7-predictor models in white

If you look closely at the significance plot, this specific model (in yellow) aligns with many of the WLs in the 100+ retained 7-parameter model averages.

Evaluating a Predictive Model

To this point we have used the built-in 'leave one out' prediction estimates for model selection. In this prediction error approach, all of the samples except the one being predicted are independently fitted and the one withheld sample is processed through that specific set of parameters. The whole of that fit process generates 1 error value. The process is then repeated for the second sample, the third, and so on until there is a set of prediction errors of the same size as the data matrix. This is a good way to test well-bracketed data where the probability of a sample outside the bounds represented in the model data matrix is low.

When working with smaller sample counts where the possibility of a sample being outside the existing bounds represented in the data matrix is high, you can use random sampling errors. This procedure creates random in-sample and out-of-sample data sets of a size and count of sets which you specify, and the errors can be average or worst case. This is how one addresses a predictive model where the model data is insufficient to fully map the expected samples.

In general, you will use the Predict feature to evaluate subsequent data once a model has been implemented.

If you have data that wasn't as rigorously generated (in this example, where an average moisture content was used), or where subsequent chromatography better separated the pertinent components going into a total estimate, those data may be useful for a separate out-of-sample prediction analysis. If the model data consisted of replicate averages that were carefully screened, you may wish to use this feature to see what the predictions would look like if there was no cleansing of the data, or if only a single sample was generated in a field analysis where an average of replicates would not generally be made.

Click the **Predict** button and select the spectral data matrix file **ModelingFTNIR_PredictionData1.csv
**from the program's installed default data directory (\PeakLab\Data). The is the raw data that
with laboratory errors (there were samples errantly mislabeled or swapped) and with all spectral replicates
present as individual samples (no average was made).

In the model graph's toolbar, select the **Common Scaling for Y and Y2 plots** to use a common Y,Y2
autoscaling for the model and prediction plots.

The prediction plot of known vs. predicted is in the upper portion of the graph. Many of these red points are laboratory sample or replicate errors. You should pay close attention when such data are observed. The more accurate the input data, the more accurate the predictive models will be. With these errors in the data, the average prediction error is .375% compared to .255% in the cleaned, averaged data.

Right click the model plot and from the popup menu select **Mark Inactive Outside
Prediction Interval**.

The Confidence Intervals dialog can be used to change the graph's prediction interval. Be sure to change the Y2 axis confidence level.

This does an admirable job removing the erroneous spectra from the prediction, while retaining the replicates. If you look at the last line in the title, the prediction average error is .282 with the replicates and .255 with the averages used in the modeling. Roughly comparable, this gives a sense for the degree to which the goodness of fit improves from the averaging.

Click the **Prediction** button to close the Prediction
window and to remove the prediction from the upper portion of the plot.

Note that the common scaling is retained when a prediction is closed. This assumes you may wish to enter
additional sets of prediction data, as we will do. You would again use this **Common Scaling for Y and
Y2 plots** item in the model graph's toolbar to restore a separate scaling for the residuals plot.

Click the **Predict** button once more and select the spectral
data matrix file **ModelingFTNIR_PredictionData2.csv **from the program's installed
default data directory (\PeakLab\Data). The is out-of-sample data done at an earlier time without certain
refinements in procedure.

The average prediction error from these data is .453. You will note that the scatter is appreciably higher.

Select the first equation in the list and using the down arrow key, step through the first fifty or so models looking for the lowest prediction error for this prediction data set. You should see none below 0.4 average prediction error.

Robust Predictive Models

One of the benefits of direct spectral fits is that models are far more flexible and far simpler than PLS or PCR models. You can, for example, choose a dozen different predictive models you happen to especially like, implement an average or median of the predicted values from each, and still use less computation than a single evaluation of a PLS or PCR model.

You can also use the quintile errors reported for the models to develop quintile-specific models. One or more models is set for each quintile of Y-values, and an overall model is used to determine which of the quintile-specific models will be used for the prediction. We have realized a reduction in prediction errors on the order of 20% by using this method.

If one of the objectives of the modeling is to produce truly robust predictions, you
may wish to explore the convenience of built-in sparse
PLS models. With respect to how such models are evaluated, they are similar to PLS and PCR evaluations
except that the WL map is sparse; only certain wavelengths are represented. When the **Generate Sparse
PLS Models** option is checked in the fitting dialog, the most significant direct spectral models are
integrated into a weighted average prediction that produces a higher count of predictors. In a sparse
PLS model, you will see something akin to PLS[15,35,8]. This means that the model consists of 15 predictors
(15 wavelengths, 16 estimated coefficients). These will be the 15 most significant predictors in the retained
8-predictor models. This overall model will have been built by the statistical goodness of fit weighting
of the 35 direct spectral fits of 8-predictor models which are a subset of these 15 significant wavelengths.
You can readily equate the 8 predictor count with a factor count.

In the Model
List's **Filter** menu, check **No Filtering** and then check **Only Stepwise and Sparse PLS
Fits**. Be sure the **Sort** is still **By Predicted Average Error**. Select the first sparse
PLS model in the list (you will have to scroll down):

Although the leave one out average predicted error is slightly higher, 0.293 instead of 0.277 for the single model we selected, this model combines 11 very effective 8-predictor models, all of which are subsets of the most significant 13 predictor WLs in the 8-predictor model. In a sparse PLS model, the WLs are not equally represented, but weighted by that WL's significance in the retained models. Certain of the WLs will have higher contributions than others, just as occurs in PLS models.

Implementing a sparse-PLS model, or any of PeakLab's spectral models, is quite easy.
Click the **Numeric** button in the GLM
Review window if the Numeric
Summary is not presently displayed. In its **Options** menu, select the first **C++** code option.
The code is added to the Numeric Summary:

Although the main language export options are C++ and Visual BASIC, the GLM Prediction window also offers Mathematica, Maple, and Excel model export.

Spectral Modeling

As with serious statistical multivariate linear modeling, direct spectral modeling is both art and science. PeakLab furnishes some of the best predictive tools in the industry to ensure your models perform as well as the modeling data permits, and the visualization is such that any issues should be apparent. Further, the full permutation GLM and Stepwise models are standard in that there is nothing of a black-box type of algorithm, and the selected fits can readily be exactly replicated in any professional-grade statistical software.