EXPERIMENTS

First, we briefly introduce the collected hyperspectral palmprint and dorsal hand vein dataset. Then, the optimal band and pattern selection are performed on different modalities, respectively. Afterwards, multimodal identification and verification results are presented, correspondingly. At last, the time consumption of the proposed method is analysed.

Multimodal Hyperspectral Palmprint and Dorsal Hand Vein Dataset

We constructed a hyperspectral palmprint and dorsal hand vein dataset captured from the same volunteers utilising the proposed hyperspectral imaging device (refer to Section 1.2). As mentioned in Section 1.2, the device can acquire hyperspectral images covering a spectrum range of 520-1,040 nm with 10 nm intervals, which means that the images on 53 different spectrums can be obtained. The dataset was acquired from 209 persons, and each volunteer was required to provide both left and right hands for imaging. This dataset contains two sessions which were acquired with intervals about 30days. In each session, a volunteer was requested to capture both their left and right hands a total of five times. Therefore, this dataset totally includes 443,080 (209 subjects x 5 samples x 2 objects x 53 bands x 2 sessions x 2 modalities) images. Some original and ROI samples from one object are shown in Figures 1.10 and 1.11, respectively.

Optimal Pattern and Band Selection

To obtain the best performance in recognition using multimodal features, we should select the best bands for palmprint and dorsal hand vein, respectively, in which the image contains rich and clear information and can derive the most discriminative features. For different feature patterns, including LBP, LDP, 2D-Gabor, and DCF, we aim to choose the optimal pattern and band for palmprint recognition and dorsal hand vein recognition, respectively. For every experiment, each algorithm was conducted 10 times. Finally, the mean accuracy of recognition rate was calculated as the performance evaluation:

FIGURE 1.10 Hyperspectral palmprint (a) and dorsal hand vein (b) samples.

FIGURE 1.11 The hyperspectral palmprint and dorsal hand vein ROIs coming from the same individual, (a) denotes palmprint ROI samples and (b) denotes dorsal hand vein ROI samples. From left to right, top to down, the band increases from 520 to 1040nm with 10nm intervals.

In this work, the experiments were implemented using MATLAB 2015a on a CPU 3.40 with RAM 16.0 GB running Windows 10.

When we extracted LBP and LDP features from the ROI image, each image was segmented into 16 non-overlapping sub-images with the same size of 32 x 32. Afterwards, the LBP or LDP features were extracted from each sub-image and further to be concatenated to one feature vector. As for 2D-Gabor, we defined a bank with five scales on eight directions. Otherwise, we applied VGG-F for DCF extraction with the DCF derived from the 19th layer of VGG-F. At last, the nearest neighbour (l-NN) was chosen for identification and verification. Figures 1.12 and 1.13 show the identification rates of different patterns on each band of the hyperspectral palmprint and dorsal hand vein cubes, respectively.

From Figure 1.12a, one can see that LBP achieved the highest ARR (98.09%) on the 44th band corresponding to 950 nm. LDP (Figure 1.12b) obtained the highest ARR (94.74%) on the 41th band corresponding to 930nm. 2D-Gabor (Figure 1.12c) achieved the highest ARR (76.08%) on the 37th band corresponding to 880 nm. DCF (Figure 1.12d) obtained the highest ARR (97.89%) on the 21th band corresponding to 730nm. As for the dorsal hand vein results presented in Figure 1.13, one can see that LBP (Figure 1.13a) achieved the highest ARR on the 38th band corresponding to 880nm with 92.20%. LDP (Figure 1.13b) achieved the highest ARR of 97.00% on the 52th band corresponding to 1030nm. 2D-Gabor (Figure 1.13c) achieved the highest ARR on the 40th band corresponding to 900 nm with 88.20%. DCF (Figure 1.13d) obtained the highest ARR on the 26th band corresponding to 780 nm with 92.20%.

Both Figures 1.12 and 1.13 show that different patterns have their own corresponding optimal bands. For hyperspectral palmprint identification, LBP can achieve the highest ARR on 950 nm with 98.09%. On the other hand, in hyperspectral dorsal hand vein identification, LDP can achieve the highest ARR on 1,030 nm with 97.00%.

FIGURE 1.12 ARRs of different patterns for each band of the hyperspectral palmprint cube: (a) LBP. (b) LDP. (c) 2D-Gabor. and (d) DCF. When the ARR >0.9, the bar colour is black. With 0.7 < ARR <0.9, the bar colour is dark grey. When ARR<0.7, the bar is coloured light grey.

FIGURE 1.13 ARRs of different patterns for each band of the hyperspectral dorsal hand vein cube: (a) LBP, (b) LDP, (c) 2D-Gabor, and (d) DCF. When the ARR >0.9, the bar colour is black. With 0.7 < ARR < 0.9, the bar colour is dark grey. When ARR < 0.7, the bar is coloured light grey.

Multimodal Identification

After pattern and band selection, we can obtain a multimodal pattern combination of W„_u „, where m is the selected pattern for the palmprint and n is the selected pattern for the dorsal hand vein. To select the optimal combinations of patterns for both the palmprint and dorsal hand vein, we tested every combination for identification in groups of two, including VT_Lbp. lbp. Wlbp.ldp- Wlbp. dcf. W'ldp.dcf. Wldp.ldp. Wdcf.dcf. Vfi.iiP. Gabor. W_LDp._Gabor, Wdcf. Gabor, and W_Gab0], _Gabor. The performance of multimodal identification was evaluated once again using ARR.

Table l .2 depicts the identification ARRs of different combinations for multimodal identification. From this table, it can be observed that W_LBPLBP achieved the highest ARR with 99.21% compared to the other features. As we know, LBP can achieve the highest ARR for palmprint recognition (refer to Section 1.4.2), while LDP obtained the highest ARR for the dorsal hand vein. In addition, from Table 1.2, we can see that multimodal identification produces a better identification performance than uni- modal identification for either palmprint or dorsal hand vein.

Multimodal Verification

In addition to the identification results mentioned above, we performed verification as well. Verification is a one-to-one matching scheme to verify if the given two samples are from the same object or sharing the same label. The performance of multimodal verification was evaluated using equal error rate (EER) as follows:

TABLE 1.2

Identification ARRS of Different Pattern Combinations for Multimodal Identification

TABLE 1.3

Multimodal Verification with Different Features

FIGURE 1.14 ROC curves of different features for verification.

where NGRA denotes the number of times of intra-class test, NIRA denotes the number of times of inter-class test, NFR presents the number of times of false rejections, and NFA presents the number of times of false acceptances. Therefore, we can obtain the EER, while FAR is equal to 1-GAR. Due to the fact that the same pattern has a better fusion property, as shown in the identification experiments, we conducted verification experiments using the patterns of W_{LBP LBP}, W_{LDP LDP}, W_Gabor>Gabor, and VF_dcfjx-p. Table 1.3 illustrates the verification results. From Table 1.3, we can see that W_{DCT DCF} obtained the lowest EER of 0.002%. Figure 1.14 shows the ROC curves of GAR and FAR for the four combined patterns.

Computational Complexity Analysis

For computational complexity evaluation, we compared the computation costs of Wlbp.lbp- Wldp.ldp- Wbabor.Gabor- and W_[XF ,_х.,. due to the fact that these four fusion strategies have similar feature extraction, fusion, and matching procedures. We randomly selected 100 classes from the multimodal dataset in Section 1.4.1. For each pattern, the experiments were conducted five times with one test sample and the remaining data as the training samples. At last, we calculated the mean time as the time consumption. From Table 1.4, it can be seen that W_{LBP LBP} takes an average time

TABLE 1.4

Time Cost of Different Kinds of Features

of 24.740 ms for feature extraction, which is the quickest when compared with the other methods. Furthermore, the average matching time was 0.063 ms for FLbp.lbp, which is the lowest simultaneously.