Project Topics
Get Complete Project Material File(s) Now! »
Toward industrial hyperspectral imaging
Hyperspectral imaging originated from remote sensing [Wolfe97] and has been explored for various applications ranging from earth to space observation and astronomy [Goetz09, Hege04, Berné10]. With the advantage of acquiring two-dimensional images across a wide range of the electromagnetic spectrum, hyperspectral imaging can be applied to numerous areas, including archaeology and art conservation [Liang12], agriculture, vegetation and water resource control [Govender07], biology [Zimmermann03, Matthäus08] and medicine [Li13, Lu14]. Another very promising application area is material analysis, in particular food quality and safety control and waste sorting [Gowen07, Feng12]. These types of application are termed as industrial hyperspectral imaging. From a technologic point of view, industrial hyperspectral imaging raises new challenges. Basically, the question at hand is the following: is it possible to develop advanced fast data processing methods that can be used in real-time industrial hyperspectral imaging systems?
The widely accepted imaging systems are wiskbroom and pushbroom and we propose to develop slice by slice recursive data processing. In this context, the recursion should act along the 1D scanning dimension y which can be assimilated to time.
Super-resolution in hyperspectral images
While real time causal processing of hyperspectral images is recognized as an important method-ological problem[Du09, Chang10, Chen14], most of the existing methods (such as anomaly detection, classification, endmember estimation) only consider memoryless processing. In opposition, the ad-dressed deconvolution problem introduces a memory and requires to develop a causal data processing adapted to pushbroom and wiskbroom imaging systems.
Hyperspectral microscopy
Hyperspectral imaging can be applied to microscopy permitting the capture and identification of different spectral signatures of nanoscale samples, such as cells and bacteria, present in an optical field [Schultz01]. Hyperspectral imaging microscopy is now becoming an indispensable technique for the biological sciences [Sinclair06, Vermaas08] and medical application [Lu14]. Among the dif-ferent spectroscopic techniques allowing to produce hyperspectral images, we can mention fluores-cence [Hiraoka02], infrared [Piqueras13, Offroy10] and Raman [Salzer09] microscopies. The work in [Henrot13a] was part of the HAESPRI (Hyperspectral Analysis and Enhanced Surface Probing of Representative bacteria-mineral Interaction) project whose aim was to study the interaction of bacterial-mineral systems through spectral images acquired with different optical modalities. The acquisition and processing of real data concern two aspects:
• bacterial bio-sensor imaging in confocal fluorescence microscopy,
• imaging of minerals and chemical compounds in confocal Raman microscopy.
Super-resolution in spectral microscopy
The increasing interest in nanosciences in many research fields like physics, chemistry, biology, and medecine requires instrumental improvements to address the sub-micrometric analysis challenges. The question at hand relates to the possibility of overcoming the limitation of the optical system. There are two main strategies to increase the resolution:
• The data fusion approach whose aim is to produce a high-resolution image from the fusion of the low-resolution images with the knowledge of the noise distribution, the decimation operator, the blur and the shift of the scene. This approach motivated many works in recent years [Villa10, Simões15, Wei15, Kanatsoulis18]. Restricting our attention to hyperspectral microscopy, we can mention the work of [Offroy10, Offroy12, Offroy15] which has developed an acquisition set-up to get low resolution images and experimentally assessed the resolution gain of super-resolution in Infrared and Raman microscopies. In particular they showed that super-resolution coupled with blind hyperspectral unmixing (MCR-ALS in chemometrics) make it possible to go beyond the diffraction limit with an algorithmic approach and that the spatial resolution can be improved by 65%.
• An alternative approach consists in using a finer sampling grid. The super-resolution problem can then be formulated as an hyperspectral image deconvolution problem. This is the approach devellopped in the Ph.D of Simon Henrot [Henrot13a]. In particular, he proposed advanced non-negative deconvolution methods [Henrot13c, Henrot13b] and analyzed the impact of de-convolution on blind hyperspectral unmixing [Henrot14b]. These approaches are connected to multichannel image restoration which was carried out with Wiener methods in [Hunt84, Galatsanos89]. Other strategies such as those in [Galatsanos91b, Giovannelli05, Zhao13] were also introduced, but only in an offline setting. Finally, let us mentioned [Jemec16] which shows that the deconvolution approach may results in a resolution improvement of more that 50% with pushbroom sensors.
Super-resolution in industrial hyperspectral imaging
While the super-resolution in industrial imaging can be stated similarly to the case of microscopy, the real time context results in additional difficulties.
The image fusion approach imposes to have multiple low resolution imaging systems working in parallel. In a real time context, this results in implementation difficulties for synchronizing the parallel acquisitions. In addition, such an approach generates significant additional costs since at least two optical cores need to work in parallel. From a data processing point of view, fast online least-mean-squares (LMS) algorithms for super-resolution of image sequence were proposed and analyzed in [Elad99, Costa07]. The key idea is based on the fact that the observed object is measured for each image in a different position (either because of camera motion or the motion of objects), and thus, several images can be combined to create an enhanced resolution output image. Note that the image sequence super-resolution problem is highly connected to the image fusion approach.
The deconvolution approach is very attractive in a real time setting since its allows to use the same optical core and to adopt a finer spatial sampling grid. However to maintain the same acquisition rate between successive frames, it is necessary to reduce the integration time resulting in a higher noise level. This motivates the development of fast online deconvolution algorithms for the restoration of blurred and noisy hyperspectral images. A Kalman filter (KF) based sequential multichannel image restoration was proposed in [Galatsanos91a] allowing a slice by slice restoration. However, because of the size of a hyperspectral image slice, the computational cost of such a KF remains prohibitive for real time applications leading us to consider least-mean-squares (LMS) based approaches.
Online deconvolution for pushbroom imaging system
The industrial hyperspectral imaging system considered in this work is a pushbroom imaging system. Samples to be imaged are carried by a moving conveyor. The whole spacial scene is observed line by line which means the hyperspectral data cubes are acquired slice by slice, sequentially in time (dimension « scan Y » in Figure 1.3(b)). By analyzing these slices, the system controls and sorts input materials right after each line scanning. The second objective of this thesis is to develop a fast online hyperspectral image deconvolution method adapted to pushbroom imaging system and compatible with real-time processing in industrial applications.
Sequential convolution model and causality issues
The 3 dimensions of the hyperspectral data cube of size N × K × P are referred to as spatial (across-track), time (along-track) and spectral dimensions. The current slices of observed and original data cube are denoted by Yk ∈ RN ×P and Xk ∈ RN × P , respectively, at time instant k. Considering their vectorized version yk and xk, and a finite length blurring kernel of size L along the time dimension, the following acquisition model can be obtained by extending the offline distortion model (1.2): L yk−(L−1) 2 = Hℓxk−ℓ+1 + ek−(L−1) 2 (1.13).
where ek the corresponding noise. Matrix Hℓ is a block-diagonal matrix composed of Toeplitz matrices; it corresponds to the ℓ-th column of convolution kernel for different wavelengths. In the context of industrial imaging, spatial image resolution is addressed as an online deconvolution problem aiming at sequentially restoring of xk.
The sequential convolution model (1.13) is designed to take into account causality issues of the convolution kernel and associated estimates. The convolution kernel H⋆p = hp hp with L 1 hp = hp h p ⊤ is centered around 0 and is non-causal. In order to make the blurring kernel ℓ Mℓ 1ℓ causal, it has to be shifted by (L − 1) 2. It is thus necessary to delay the observation by (L − 1) 2 samples denoted by y˜k as follows: L y˜k = yk−(L−1) 2 = Hℓxk−ℓ+1 + ek−(L−1) 2 (1.14) .
Hyperspectral image deconvolution
Hyperspectral imaging consists in observing a spatial scene at several wavelengths. Physically, such an image can be obtained as a stack of two-dimensional (2D) images equipped with optical filters or as a collection of one-dimensional (1D) spectra acquired by a spectrometer. Hyperspectral imaging is used in a wide range of applications including remote sensing [Goetz09], chemistry [Duponchel03, Piqueras14], food science [Gowen07], biology [Zimmermann03] and medical imaging [Lu14]. Among the different spectroscopic techniques allowing to produce hyperspectral images, we can mention infrared, Raman [Salzer09] and fluorescence [Hiraoka02] microscopies. The problem at hand aims at removing the blur affecting the observed images. Such a blurring arises, for example, when we want to increase the spatial resolution of the imaging spectrometer. To do that, it is necessary to choose a spatial sampling lower than the instrument resolution.
Discrete representation of the blurred images
The unknown hyperspectral image is denoted by X and the observed image by Y. Considering that the discrete image X has P wavelengths λ1 λ P , it can be seen as a stack of images {Xp p = 1 P }. Xp is a matrix of size N × K. By concatenating the columns of each image Xp, the hyperspectral image can be reorganized into P vectors {xp p = 1 P } of length N K each, or a single vector x of length N KP . We use similar notations for the observed image, substituting letter y to letter x.
The blurred image corresponds to the 2D (circular1 ) convolution of Xp with filter H⋆p known. Define the 2D spatial convolution kernel H⋆p = hp hp with hp = hp h L1ℓMℓ which is p ⊤. 1ℓ first column: [hp (M+1) 2ℓ first row: [hp (M+1) 2ℓ p 0 0 h p h p 2ℓ ] (2.1).
Table of contents :
1 Introduction
1.1 Context and industrial objective
1.2 Hyperspectral imaging
1.2.1 Spectroscopy
1.2.2 Acquisition modes of a hyperspectral image
1.2.3 Toward industrial hyperspectral imaging
1.2.4 Hyperspectral microscopy
1.3 Super-resolution in hyperspectral images
1.3.1 Super-resolution in spectral microscopy
1.3.2 Super-resolution in industrial hyperspectral imaging
1.3.3 Observation model
1.4 Scientific contribution of the thesis
1.4.1 Regularization parameters estimation in non-negative hyperspectral image deconvolution
1.4.2 Online deconvolution for pushbroom imaging system
1.4.3 Joint unmixing and deconvolution of hyperspectral images
1.5 Organization of the thesis
1.6 Publications
2 Regularization parameter estimation for non-negative hyperspectral image deconvolution
2.1 Introduction
2.2 Hyperspectral image deconvolution
2.2.1 Discrete representation of the blurred images
2.2.2 Hyperspectral image deconvolution
2.3 Hyperspectral image deconvolution as a multi-objective optimization
2.3.1 Multi-objective Optimization
2.3.2 Shape of the estimated response surface
2.4 Choosing the Regularization Parameters
2.4.1 Maximum curvature criterion
2.4.2 Minimum distance criterion
2.4.3 A grid-search strategy for MDC
2.5 Examples and Experiments
2.5.1 Performances of MCC and MDC for 2D image deconvolution
2.5.2 An illustrative example of the non-negativity constrained hyperspectral image deconvolution
2.5.3 Performances of MCC and MDC for non-negative hyperspectral image deconvolution
2.5.4 Application to hyperspectral fluorescence microscopy
2.6 Conclusion
2.7 Supplementary material: behavior of the MDC and MCC for different types of hyperspectral images
2.7.1 Simulated hyperspectral images
2.7.2 Performance evaluation and result presentation
2.7.3 Discussion
3 Online deconvolution for industrial hyperspectral imaging systems
3.1 Introduction
3.2 Blurring and causality issues
3.2.1 Scanning technologies and data structure
3.2.2 Blurring and noise
3.2.3 Causality
3.3 Online image deconvolution
3.3.1 Block Tikhonov
3.3.2 Sliding-block regularized LMS (SBR-LMS)
3.3.3 Algorithm implementation and computational cost
3.4 Transient behavior analysis
3.4.1 Mean and mean-squares transient behavior model
3.4.2 Stability condition
3.5 Experimental results
3.5.1 Validation of the transient behavior model
3.5.2 Effects of the parameters
3.5.3 Performances
3.5.4 Real hyperspectral image deblurring
3.6 Conclusions
4 Unmixing and deconvolution for hyperspectral images
4.1 Introduction
4.2 Linear unmixing
4.2.1 Observation model
4.2.2 Joint unmixing-denoising (JUDN) method
4.2.3 Separated unmixing and denoising (SUDN) method
4.2.4 Comparison of JUDN and SUDN
4.3 Unmixing and deconvolution for hyperspectral images
4.3.1 Observation model
4.3.2 Offline unmixing and deconvolution
4.3.3 Online unmixing and deconvolution
4.3.4 Non-negative JUDC
4.3.5 Efficient implementation of the NN-JUDC
4.4 Experimental results
4.4.1 Simulated hyperspectral image
4.4.2 Application to wood waste sorting
4.5 Conclusion
5 Conclusion
6 Résumé étendu
6.1 Contexte et objectif industriel
6.2 Imagerie hyperspectrale
6.2.1 Spectroscopie
6.2.2 Acquisition d’une image hyperspectrale
6.2.3 Imagerie hyperspectrale industrielle
6.2.4 Microscopie spectrale
6.3 Superresolution in hyperspectral images
6.3.1 Super-resolution en microscopie spectrale
6.3.2 Super-résolution en imagerie hyperspectrale industrielle
6.3.3 Modèle d’observation
6.4 Contributions scientifiques de la thèse
6.5 Publications associées à la thèse
Bibliographie
