Research Projects

© 2023_Poster_M_Saraoglu

--> PDF Download

Selected Publications

YouTube Video: https://youtu.be/OJRNFl-cG-o

https://youtu.be/OJRNFl-cG-o

Publication: https://link.springer.com/chapter/10.1007/978-3-030-58920-2_14

https://link.springer.com/chapter/10.1007/978-3-030-58920-2_14

Primary project goal:  In the frame of proposed project, we will implement the described method into a prototypical hardware demonstrator. An embedded board (KrakenBox) will play the role of a dedicated error detector. We plan to apply the discussed above neural networks for industrial automation systems that usually employ modern Ethernet-based buses for data exchange between distributed industrial components. E.g., PROFINET (Process Field Net) is an industry technical standard for data communication over Industrial Ethernet, designed for collecting data from, and controlling equipment in industrial systems, with particular strength in delivering data under tight time constraints. The figure below shows a typical automation system linked via the PROFIBus. Our error detector will be connected to the PROFINets (green) bus, learn the normal system behavior in order to detect anomalies in the future.

© A. Morozov

Our background methods:  In our lab, we develop a novel hybrid approach that combines DL techniques to create effective and efficient CPS error detectors. The signals of cyber-physical systems are mainly time-series data. In [1], we proposed a new on-line error detection and mitigation approach for typical sensor, computing hardware, and network errors of cyber-physical systems using deep learning-based methods, see the figure above.  More specifically, we train a Long Short-Term Memory (LSTM) network as a single step prediction model for the detection and mitigation of errors, like spikes, or offsets. In order to detect the long-duration errors that show no sharp change (a sudden drop or rise) between two successive data samples when errors occurred, e.g., network delays, we train an LSTM encoder-decoder as a multi-step prediction model. We also introduce the on-line error mitigation approach. Automatic recovery is achieved by replacing the detected errors with the predicted values.

[1] Ding, K., Ding, Sh., Morozov, A., Fabarisov, T., Janschek, K.: On-line Error Detection and Mitigation for Time-series Data of Cyber-physical Systems using Deep Learning based Methods. In Proceedings of 2019 15th IEEE European Dependable Computing Conference (EDCC), Naples, Italy, 2019, pp. 7-14, doi: 10.1109/EDCC.2019.00015.
**Distinguished Paper Award**

Abstract

Complex systems, e. g. Cyber-Physical Systems (CPS), typically comprise distributed components with embedded controllers that are running in parallel and share data over a communication network. Concurrency, synchronization, temporal sequence, and timing affect the correct system behavior. Due to effects like network induced delays and varying execution times, the timing is considered not only deterministic but also stochastic. One of the challenges in the model-based systems design is the use of appropriate design and analysis models. Models are the only available source of information for system analysis in the early stages of the system design. Hence the models must comprehensively describe the mutual interaction of heterogeneous components (software, hardware, mechanical parts) on common abstract levels. Model-based design allows the analysis, visualization, test, and validation of complex systems in early design stages without a physical prototype. The analysis results are used to predict functional as well as non-functional system characteristics. The benefits are potentially saving time and cost in the product design by detecting specification inconsistencies and design errors in early design phases, evaluating various design options, finding optimal design solutions. The concept phase is the first design step of the system development cycle. In the concept phase, high level abstract structural and behavioral models are generated. Widely used modeling languages are the Unified Modeling Language (UML) for software-based systems, the Systems Modeling Language (SysML), and the Architecture Analysis and Design Language (AADL) for heterogeneous and complex systems.

Dependability is a generic term and includes system properties like safety, reliability, and availability. A system is considered as dependable, if the probability of a system failure is negligible (below a given threshold). A challenging task of model-based system design is the specification and analysis of system dependability properties. For the dependability analysis, different types of errors and their probability of occurrence are considered. We can distinguish between data and timing errors. In this thesis, the focus is on the model-based analysis of timing properties. When a function has to be fulfilled within a finite time, it is a real-time function. If the timing does not comply with the specification, it is considered as a timing error. Timing errors may be caused by temporary internal faults like delays. If a timing error leads to an observable malfunction, it becomes a timing failure. Timing errors are critical and have to be analyzed because they may decrease the system’s dependability or even lead to failure. Thus, the analysis and prediction of timing properties in early design phases is a valuable but challenging task. Consider an autonomous moving device. Its equipped with sensors to perceive its environment. If data from a sensor that measures the distance to obstacles are correct but delayed, the appropriate action, either to correct the movement path or to stop the vehicle, will be delayed and may lead to a collision with an obstacle. The workflow of the proposed analytical approach is shown in Figure 1. The workflow includes the following automated steps.

• Annotation of the baseline model with stochastic timing properties.
• Transformation into a formal stochastic Petri net model.
• Generation of a reachability graph.
• Generation of an input model for the probabilistic model checking tool.
• Identification of the time constraints.
• Formalization of the time constraints.
• Generation of properties for the probabilistic model checking tool.
• Computation of relevant metrics.

The computed results are metrics, e. g. the probability of the occurrence of timing errors, which allows the assessment of the conformity to the requirements. A sensitivity analysis reveals the contribution of specific model parameter to the computed results and helps to identify candidates for design improvements.

© Th. Mutzke

Selected Publications

Abstract

A failure in the behavior of safety-critical systems such as Unmanned Aerial Vehicles (UAV) may lead to hazards and serious consequences. Therefore, UAVs must perform correctly throughout their intended life cycle. However, the functional system specification is not sufficient to ensure proper performance. Model-based systems engineering approaches are commonly used for the development of such systems. The assessment of non-functional reliability, safety, or timing requirements in the early development phases helps to prevent conceptually wrong decisions. UAVs usually contain complex and tightly coupled hardware and software components that influence each other. Hence, a holistic approach to analyze the non-functional properties is preferable. The concept of phased mission systems enables the specification of individual requirements for different phases. For instance, the reliability requirements or system specifications are different for UAV flights over an agricultural field and a highway. Considering these individual requirements during the assessment allows more accurate predictions of the success or failure of a mission and can further improve the holistic approach. The objective of the project is to develop a holistic model-based approach to dependability assessment that supports system software co-engineering for UAVs. The approach combines stochastic timing analysis and the data error propagation analysis and enables the consideration of phase-specific requirements as well as the identification, formalization, and analysis of timing properties. The solution proposed is centered around the Systems Modeling Language (SysML) and the systematic transformation of annotated SysML models to Markov chain models. The proposed approach exploits modern probabilistic model checking techniques for the quantification of several dependability metrics. The method supports the systematic analysis of system specifications that contain both structural and behavioral system properties.

© M. Steurer

Selected Publications

Abstract - Project

Model-based control software development is widely used in a variety of safety-critical domains including automotive, aerospace, and industrial automation. Control algorithms are typically developed using the combination of two classical types of models: time discrete block diagrams (BD) and discrete event finite state machines (FSM). The model-based approach ensures high consistency between base-line models and production code, which allows avoiding many faults that could be introduced in case of manual software development. However, even certified and well-tested model-driven control software is vulnerable to hardware faults, e.g. single event upsets. Such faults result in silent data errors that might propagate through the system to critical outputs. The likelihood of fault activation and error propagation depends on a large number of factors. Analysis of this complex process is extremely helpful in a wide range of analytical tasks associated with dependable systems development. Recently we have introduced a new probabilistic approach to error propagation analysis using a dual-graph error propagation model. The central idea is a synchronous examination of control flow, data flow and reliability properties of system components. A discrete time Markov chain (DTMC) model was applied in order to obtain probabilities of erroneous and error-free system execution scenarios. This approach has been tested and shown promising results on a mechatronic case study that was developed using a base-line UML model. The proposed project is devoted to a model-based probabilistic error propagation analysis for control algorithms built from hybrid time-discrete BD and discrete event FSM models. On the basis of our baseline dual-graph error propagation model and the Markov-based approach, we will develop extensions for abstract modeling of specific BD/FSM properties (e.g. hierarchical nesting, multi-rate, internal memory) and methods for automated mapping of hybrid BD/FSM models in dual-graph error propagation models and computationally efficient DTMC models. The newly developed methods are prototyped in a toolchain enabling validation of the new approach on the basis of a representative case study.

Abstract - PhD Thesis Kai Ding

© Kai Ding

Model-based design is a common methodology in the development of embedded complex control systems. Control system engineers typically prefer to use MATLAB Simulink and suitable automatic code generators for the development and deployment of software. Embedded systems are subject to random hardware faults. This thesis proposes an analytical method for the evaluation of the reliability properties of control systems that are designed with Simulink models. The method is based on a transformation of the assembly code, which is generated from the Simulink model, into a formal stochastic error propagation model as well as its quantification through underlying Markov chain models and state-of-the-art probabilistic model-checking techniques. In the case of model-based development, redundancy mechanisms are preferable for direct application at the model level (Simulink model level). This thesis introduces a systematic classification of fault-tolerant design patterns. Such patterns can be applied to the Simulink model to tolerate random hardware faults, and taken into account during the control system design. This thesis proposes a model-level reliability evaluation of Simulink models. The efficiency of the proposed model-level evaluation is verified by a comparison of the reliability properties that are assessed at the assembly and model levels.

Selected Publications

List of the most important publications resulting from this project, overall, 20 papers were published within the project period.

Abstract 

Embedded hardware and software systems are vulnerable to environmental impacts, such as single event upsets that may cause silent data corruption and result in data errors at the software level. The occurred data can propagate further through heterogeneous system components. Error propagation has an influence on the system behavior in critical situations. Analysis of this phenomenon gives sound support for fault-tolerant system design. The error propagation analysis provides quantitative estimation of the likelihood of error propagation to hazardous parts of a system, which is an important input for the reliability assessment methods and helps to identify components that should be equipped with error detection or error recovery mechanisms. In the frame of this project we have extended our analytical toolset and applied it for the analysis of a distributed space-to-ground haptic feedback system: ErrorPro - a tool for error propagation analysis, which consists of a user interface, and a complex back-end, which performs Markov-based analysis of data error propagation to selected system parts; Simulink Parser - a MATLAB software, which analyses a Simulink model and transforms it into XML format, supported by ErrorPro, for further analysis; ErrorSim - a MATLAB-based simulator that allows a user to mark error-prone Simulink blocks, run a number of simulations, and evaluate a number of error in selected blocks.

Abstract

Error propagation analysis is an important part of a system development process. This research addresses a model-based analysis of spreading of transient data errors through mechatronic systems. Error propagation models for such kind of systems must use a high abstraction level that allows the proper mapping of the mutual interaction of heterogeneous system elements such as software, hardware and physical parts. A number of appropriate approaches have been introduced in recent years. The majority of them are based only on a system data flow representation. This research shows that the system control flow has to be considered as well for a complete picture of error propagation. A core part of this work is a new probabilistic error propagation model based on two digraphs: a control flow graph and a data flow graph. The structures of these graphs can be derived systematically during a system development process. The knowledge about an operational profile and individual parameters of the system elements allows the definition of additional system properties. A discrete time Markov chains model is applied for simultaneous analysis of control flow and data flow of the system. This Markov chain can be generated automatically using the framework of the introduced dual-graph error propagation model. Specific computation of this Markov chain gives the distribution of the probabilities of different erroneous and error-free scenarios of system operation. It helps to perform a precise reliability evaluation, to speed up fault localization and error detection processes, and to develop an effective testing strategy. 

PhD Thesis

Andrey Morozov, Dual-graph model for error propagation analysis of mechatronic systems. TU Dresden, Vogt 2012, ISBN 978-3-938860-56-4, pp. 1-129

Abstract

© E. Koycheva

Nowadays, the performance characteristics of information and automation solutions are mostly determined during their commissioning. If performance deficiencies are then identified, changes are necessary, which necessarily entail additional time and investment costs. A process that enables the localization and elimination of performance deficiencies in the early design phases, especially before cost-intensive software implementation and hardware procurement, is of paramount importance to avoid cost and schedule overruns in the project. This work proposes an approach to early model-level performance evaluation that allows verification of system properties during the design phase.
A comprehensive methodology based on model-to-model transformations was developed for the transformation of the MARTE-annotated UML design into Generalized Nets. This means that the transformation rules are defined at the metamodel level, with a separate set of rules being specified for UML and for MARTE. This approach ensures a high degree of reuse of the concept.
The developed approach uses the modeling language UML (Unified Modeling Language) for system modeling. Performance properties are added to the system model by annotating the UML elements with MARTE (UML Profile for Modeling and Analysis of Real-time and Embedded Systems) stereotypes. Although a MARTE-annotated UML model can broadly represent the performance-relevant properties of the system under investigation, the lack of semantic formality of the model does not allow performance analysis to be directly based on it. Therefore, this model is transformed into a formal domain that allows its evaluation. In this work, the performance analysis domain represents the class of Generalized Nets (GN), a Petri-Net-based higher level approach.

© E. Koycheva

For the application of the presented approach, the famework schematically shown in the figure was developed. A UML tool with MARTE support exports the annotated UML model; transformation software imports this and transforms it into a GN model. The GN model is fed to a simulator. The results from the simulation of the GN model are used for system verification and optimization and can be partially fed back into the initial model. The interfaces between the tools are XML-based - XMI (XML Metadata Interchange) represents the annotated UML model and XGN (XML for Generalized Nets) the GN models in XML format.

© E. Koycheva

Selected Publications

Abstract: Over the last decade, the exploration of comets and asteroids by space probes has received increasing importance (ESA's Rosetta & Philae, JAXA's Hayabusa-1/2, NASA's OSIRIS-REx). From a scientific perspective, asteroids and comets, the "small bodies" in the solar system, like time capsules, accommodate detailed knowledge about the origins and the evolution of the solar system and of life. Furthermore, recent observations suggest the existence of rich mineral resources on asteroids, which could be used for the construction and refuelling of spacecraft, or in an economic way for the extraction of rare resources. The overall goal of the project is to maximize the scientific yield on a small body of our solar system. Our innovative concept is based on a hovering spacecraft which serves as a mobile surface platform with capabilities of maximum surface accessibility and flexibility. By means of energy efficient near-surface flight phases with altitude and attitude control, any point on the surface can be reached. This enables research missions with in-situ experiments at any characteristically interesting location of the surface. As a major technological building block, the project addresses the ability for autonomous operation of the probe by means of trajectory planning procedures in only partially known environment. This includes all functions of a guidance, navigation and control system. The focus is on motion planning tasks, i.e. for very limited computing power available on board of a spacecraft. The follow-on project Astrone KI is focused on increasing the resilience against unforeseen events, as well as in the increase of robustness and the performance in the presence of uncertain model knowledge by using Artificial Intelligence to solve GNC tasks.

TUD task: Astrone KI navigation development

Selected Publications

© K. Janschek

Abstract: The exploration of small bodies in the solar system, such as comets and asteroids, by space probes is receiving increasing attention (ESA's Rosetta & Philae, JAXA's Hayabusa-1/2, NASA's OSIRIS-REx). Like time capsules, these small objects house detailed knowledge about the origins and evolution of the solar system and life, respectively. Furthermore, rich mineral resources are expected there. The surfaces of the extremely small and thus gravitationally weak comets and asteroids are extremely rocky and irregular, making ground-based approaches to locomotion unsafe and ineffective. Even the smallest disturbances can lead to unintended loss of traction and uncontrolled lift-off or, in the worst case, mission loss. Consequently, suitable system concepts must ensure complete control over all motion states and mission phases. The overall goal of the project is to maximize the scientific yield on a small body of our solar system. Our innovative concept is based on a hovering spacecraft, it avoids the aforementioned mobility constraints and, as a mobile surface platform, comes up with capabilities of maximum surface accessibility and flexibility. By means of energy-efficient altitude- and attitude-controlled near-surface flight phases, any point on the surface can be reached. This enables research missions with in-situ experiments at any characteristically interesting location on the surface. As an essential technological building block, the project addresses the capability of autonomous operation of the probe by means of trajectory planning procedures in only partially known environments. This includes all functions of a Guidance, Navigation and Control (GNC) system, with the focus on motion planning tasks for space applications, i.e. for only very limited available computing power on board a spacecraft.

TUD task: LiDAR-based navigation

Selected Publications

Kurzfassung

Entwickelt wird ein sich selbst orientierender, autark arbeitender Roboter für Inspektion und Schadensdetektion sowie Überwachungsfunktionen auf industriellen Flachdächern. Er orientiert sich selbständig und erkundet Dachflächen. Dabei werden Schäden, die für das Dach, Menschen oder Prozesse unter dem Dach Risiken darstellen, detektiert und auch Überwachungsfunktionen ausgeführt.

Teilprojekt für Technische Universität Dresden - Institut für Automatisierungstechnik:
Entwicklung von Algorithmen und der Software zu den odometrischen Berechnungen und zur kamerabasierten Bestimmung der Position und Orientierung für den Roboter sowie Tests in einer Gesamtsimulationsumgebung.

© K. Janschek

​

ESA Project IRPN - Image Recognition and Processing for Navigation
This activity provides design, development and verification of the necessary capabilities in image processing for position, pose and angular motion detection on uncooperative targets in an Active Debris Removal scenario.

​

ESA Project IRPN - Image Recognition and Processing for Navigation
Relative chaser-target state estimates are provided by specific image recognition and processing (IRP) algorithms and sensor fusion algorithms in a navigation function (NAV) using sensor data from cameras in the visible spectrum (VIS) and infrared spectrum (IR)) and light detection and ranging (LIDAR) considering as reference mission the Active Debris Removal of ENVISAT (100 … 2 m).

Project Abstract

Due to the increasing number of space debris and faulty satellites in the Earth orbit, in the recent years On-Orbit Servicing was put more and more into the focus of research. The different challenges for spacecraft (S/C) operations led to several strategies and projects. However, the majority of the demonstrated experiments for On-Orbit Servicing aimed at rendezvous with cooperative targets. ESA's Clean Space initiative aims at Active Debris Removal (ADR) for de-orbiting defunct satellites or pieces of space junk. In such a rendezvous mission the target is uncooperative. In order to perform the approach sequence, several constraints have to be taken into account, especially in low Earth orbit (LEO) where the priority debris for removal is located (most of the times in sun-synchronous orbits). Vision-based relative navigation is a key technology for those missions. At the Institute of Automation at TU Dresden (TUD) the vision-based relative navigation with uncooperative targets was studied in the past, including the estimation of the target object's pose and motion and the 3d reconstruction of the shape of an unknown target. Thus, in the context of the Clean Space initiative, TUD was prime in the activity Image Recognition and Processing for Navigation (IRPN). The developed system estimates the relative pose between chaser S/C and the uncooperative target satellite ENVISAT using different complementary vision-based sensors (cameras in the visible spectrum and thermal infrared spectrum) and light detection and ranging (LIDAR) by using image processing techniques and an estimation filter. The conditions (target with reflective surface materials, ambiguities of geometric features, quick change of illumination conditions due to target rotation/tumbling) result in very demanding needs for the navigation in the ADR scenario, in particular for the image processing. Thus, the verification and validation scenarios for testing those algorithms have to consider before mentioned effects to guarantee the algorithms' performance and robustness. As a consequence in IRPN several test environments with several test scenarios have been developed and set up.

Selected Publications

PROJECT AND SYSTEM OVERVIEW

The Image Recognition and Processing for Navigation (IRPN) activity consisted of the development and testing of a distributed vision-based navigation system for relative navigation between spacecraft by means of different sensors. In the reference mission a chaser S/C approaches ENVISAT in LEO from a distance of 100 m to 2 m. ENVISAT was assumed to be out of control, performing tumbling motions and to be uncooperative in the sense of being uncontrolled and not prepared for any kind of rendezvous techniques.

© F. Schnitzer

Fig. 1: Illustration of a chaser spacecraft approaching to satellite ENVISAT

The main functional parts of the system developed in IRPN are shown in Fig. 2: The system estimates the relative pose between the S/Cs using different complementary vision-based sensors (cameras in the visible spectrum (VIS) and infrared spectrum (IR)) and light detection and ranging (LIDAR), sensor specific processing algorithms (image recognition and processing, IRP) and a navigation function (NAV). The measurements from different sensors are first processed by the IRPs and those estimations are merged and filtered afterwards in a relative navigation filter in the navigation function. The main computing rate is 10 Hz, except for the LIDAR part of the system, which has a computing interval of 0.3 s.

© F. Schnitzer

Fig. 2: In the IRPN project algorithms are developed to process camera images (visible spectrum and thermal infrared) and/or LIDAR point clouds to determine the relative pose of the uncooperative satellite ENVISAT.

The activity was performed by following contractors: The Institute of Automation at Technische Universität Dresden was prime, responsible for the IRP algorithms for VIS and IR, for the navigation function and for the test campaigns. Airbus Defence & Space (Bremen) contributed the IRP algorithms for processing the LIDAR data. For testing the algorithms with synthetic image data, the Camera Simulator from Astos Solution GmbH was used. Jena-Optronik GmbH provided the software component for simulating realistic behaviour and errors of the LIDAR sensor. The sensor data was generated according to trajectory data that had been simulated by Airbus Defence & Space (Friedrichshafen).

© F. Schnitzer

Fig. 3: System concept, consortium and allocation of responsibilities in the IRPN project.

All IRPs and the overall IRPN system have been tested in different test environments (model-in-the-loop (MIL), processor-in-the-loop (PIL)) and on one hand with synthetic (rendered) images and on the other hand with real images (generated on-ground in rendezvous simulator).

ALGORITHMS FOR IMAGE RECOGNITION AND PROCESSING FOR NAVIGATION

The IRPN algorithms can be divided into four main parts: Three IRPs (VIS, IR and LIDAR) and NAV. All IRPs directly determine the relative state between sensor and target. All IRPs consist of two stages: An initialization stage and a tracking stage.

The camera IRPs (VIS and IR) use for initialization a dense stereo reconstruction to determine a coarse initial relative pose estimate by comparing disparity-based hypotheses for their congruence with the image data. For tracking, they use this initial pose to further improve the accuracy and track the relative movement over time by utilizing the a priori known 3d model of the target which is fit to the image by matching projected lines and edges in the image and minimizing the distances between them.

© F. Schnitzer

Fig. 4: Image-based pose estimation using line tracking algorithms to determine the relative pose and the uncertainty of this estimate for the known target.

An additional measurement is derived from the image data by determining the Optical Flow fields, which characterize the motion of the target satellite within the field of view of the camera. In the ESA GNC IRPN activity a simulation model of Optical Correlator (OCOR) was designated for this task. The Optical Flow vectors are used for improving the expected relative pose before conducting the actual pose tracking. The design of the LIDAR-based pose estimation consists of an architecture that uses a smart scanning LIDAR which is able to control its field of view and its laser power level. For initialization the LIDAR IRP only assumes the target being in the field of view and computes a coarse estimate of the relative pose. Its tracking algorithm performs an iterative matching scheme between the point cloud and the known geometry of the target for very robust and lighting independent relative pose estimation.

© F. Schnitzer

Fig. 5: Determination of Optical Flow fields (see green lines in the right figure) with TUD's optical correlator hardware (OCOR).

The Navigation Function incorporates fusion and filtering of all available information (including local sensor level measurements) by usage of an Extended Kalman Filter (EKF). It provides especially estimates for all chaser-target spacecraft relative states with low latency (and the corresponding estimated uncertainty). Beside the relative navigation specific sensor measurements (camera/LIDAR pose estimates), also information from the chaser AOCS (chaser position/attitude, acceleration) and the known (relative) S/C dynamics are considered by the filter algorithm for better robustness and performance. Furthermore, NAV is designed to be able to deal with different delays of the sensor measurements.

© F. Schnitzer

Fig. 6: Navigation Function for estimating relative position, velocity, attitude and attitude rate of satellite ENVISAT

ALGORITHM VERIFICATION AND VALIDATION

The test campaigns for verification and validation of the algorithms have been performed in three test environments: Model-in-the-loop (MIL) tests with a MATLAB Simulink facility and synthetic image data, processor-in-the-loop (PIL) tests on an Extended dSPACE System with synthetic image data and tests using real on-ground image data (rendezvous spacecraft simulator). All images, simulated LIDAR measurements, AOCS data and Optical Flow data were generated offline based on representative reference trajectories. Using the offline generated input data the test campaigns could be performed in real-time (or faster).

MIL Tests

A MATLAB Simulink model has been developed in order to validate the IRPN algorithms. It contains the IRP algorithms and the navigation function as well as the different interface blocks for reading the data from hard disk and saving the test results to files. The step time of the Simulink model had been set to 0.1 s, but on a Core-i7-4790K CPU (4x4.0 GHz) the algorithms performed faster than real-time.

© F. Schnitzer

Fig. 7: The IRPN MATLAB Simulink MIL model for processing the image and LIDAR data

For Monte Carlo experiments in the MIL facility, synthetic image and LIDAR data were used based on trajectories from different approach scenarios. Each trajectory has a length of 4000 s, comprising 40,000 image sets each for VIS and IR as well as 13,334 LIDAR data sets.

© F. Schnitzer

Fig. 8: VIS (left) and IR (right) synthetic images from trajectories of different scenarios. (top/bottom).

PIL Tests

In the PIL facility the automatic code generation and compilation for an external hardware target and the possibilities for parallelization have been demonstrated. This showed the general real-time capability of the IRPN system.

© F. Schnitzer

Fig. 9: Architecture overview for the PIL facility. The IRPN algorithms are implemented on different boards of the dSPACE System. Gateway PCs provide the LIDAR and image data for each time step. A Control PC starts the system and logs the result data.

The same input data as in the MIL tests had been processed during the PIL tests. Hence, the same trajectories and the same image and LIDAR data were used.

A MATLAB Simulink model has been developed in order to validate the IRPN algorithms. It contains the IRP algorithms and the navigation function as well as the different interface blocks for reading the data from hard disk and saving the test results to files. The step time of the Simulink model had been set to 0.1 s, but on a Core-i7-4790K CPU (4x4.0 GHz) the algorithms performed faster than real-time.

All input data had to be read from hard disks and had to be provided in real-time to the dSPACE boxes. This was implemented by Gateway PCs, which were triggered to provide the sensor data only on request of the IRP algorithms and therefore simulate sensor hardware. All together this hard- and software was called Extended dSPACE System.

© F. Schnitzer

Fig. 10: Components of the Extended dSPACE System (left) and PIL Simulink model (right) with assignment of the IRPs to processor cores. Each of the four dSPACE DS1006 boxes hosts one of the IRPN algorithms. Three Gateway PCs provide the LIDAR and image data for each time step. A Control PC with the dSPACE ControlDesk Software starts the system and logs the result data.

In general, the PIL tests showed the same results as was shown in the MIL tests. LIDAR IRP allowed robust initialization at distance of 100 m. As expected, LIDAR IRP tracking was most robust due to the independence from external illumination. Combining LIDAR IRP with VIS and/or IR IRP algorithms improved the quality of the estimates, especially after the synchronization with the target rotation.

© F. Schnitzer

Fig. 11: Mean absolute relative position error over all experiments of two scenarios (x, y and z coordinate of target body frame w. r. t. chaser body frame; in m) and standard deviation (3-σ-sleeve). At t ≈ 800 s the chaser starts to move to the target's rotation axis; at t ≈ 1600 s the chaser starts to synchronize its rotation with the target rotation; after t ≈ 1750 s the chaser is moving towards the target with a direct approach. There are hold points during the approach at t ≈ 0 ... 500 s (100 m), at t ≈ 700 ...1200 s (50 m), at t ≈ 2000 ... 2400 s (11 m) and at t ≈ 2600 ... 4000 s (2 m). The estimation errors increase when starting the transfer to the rotation axis and decrease when synchronizing with the target's rotation. At a distance of approx. 10m - 15m (t ≈ 1800 ... 2500 s) the highest accuracy is reached. After this, the target is only partially visible by the sensors and hence the estimation errors are increased a bit.

© F. Schnitzer

Fig. 12: Results of a simulation experiment (overlying the model data (red, based on the estimated pose) over the IR image data).

Generally, the highest accuracy is reached when the target is at a distance of approx. 10m - 15 m, when the LIDAR scan pattern covers most of the target and most of the edges of ENVISAT can be seen in the images of the cameras. In general, the position error in those parts was less than 1-3 cm and the attitude error was less than 0.3° in these parts of the trajectory.

Tests with Images Generated with On-ground S/C Rendezvous Simulator

In a final stage of the activity, real image sensors (VIS cameras + IR cameras, no LIDAR) were used to generate real image data to test the IRP algorithms. The cameras were attached to TUD's Spacecraft Rendezvous Simulator MiPOS. The camera motion is based on the same trajectories as used in the MIL and PIL tests. During this image acquisition phase, the image data were saved on hard disk and only processed offline afterwards.

The results of the tests with the real camera images differed clearly from the results of the MIL and PIL tests. This was caused mainly by the edges of the silhouette and of other features which were better recognizable in the synthetic image data while being faded in the real image data. Furthermore, in the real image data there were bigger differences between the mock-ups and the a priori known model used for the pose tracking. Additionally, the barrel distortion of the IR images showed to be much bigger than expected. The differences of the test results showed that tests with real image data are necessary and that one should not rely on synthetic image data only.

© F. Schnitzer

Fig. 13: VIS (left) and IR (right) on-ground rendezvous simulator images for mid-range/close-range (top/bottom).

IMAGE GENERATION WITH ON-GROUND S/C RENDEZVOUS SIMULATOR

For the generation of on-ground image data, the Spacecraft Rendezvous Simulator MiPOS (Mini Proximity Operation Simulator) was used. Two cameras for VIS and two cameras for IR were mounted to the 6-DOF portal robot and moved step-by-step along an approach trajectory facing a mock-up of ENVISAT. At every step the four images were saved to hard disk. Due to the limited workspace of MiPOS and the resulting limitations in the cameras' depth of field, two down-scaled mock-ups of ENVISAT had to be used: One with scale 1:25 for mid-range (simulated target distance from 50m to 11 m) and another with scale 1:5 for close-range (simulated target distance from 5m to 2 m).

© F. Schnitzer

Fig. 14: Spacecraft Rendezvous Simulator MiPOS used for the HIL test image acquisition.

Due to the usage of thermal infrared cameras, the mock-ups must not only have a visual signature, but also a thermal signature. We used the effects that different materials heat-up with different gradients and that bare metal parts reflect the temperature of the surrounding, which is colder than the heated mock-up, to generate thermal signatures.

For the mid-range simulations, a detailed model of ENVISAT was provided by ESA. It has real MLI foil on its surface and sensor/payload models are made of metal. However, the internal structure the real satellite is not modelled. For illumination and heating during image generation Halogen spotlights were used.

For the close-range simulations, the mock-up was developed at TU Dresden. It comprises only those features of ENVISAT, which are important in this distance/scale, and mainly consists of two large aluminium plates representing the front side of the Payload Module and the ASAR antenna. Stickers on the front, which are up-scaled photographs of the surfaces of the 1:25 scale mock-up, add visible details to the model, the radiator plates were cut out from the stickers (bare metal surface). For heating of the stickers, resistors are mounted on the rear of the plates. A Halogen floodlight was used for the illumination.

© F. Schnitzer

Fig. 15: ENVISAT mock-up with 1:25 scale (left) for the mid-range HIL tests and ENVISAT mock-up with 1:5 scale (right) for the close-range HIL tests.

ACTIVITY EXTENSION FOR REAL-TIME ON-BOARD PROCESSING

Looking forward to future applications of algorithms for space craft rendezvous the project extension RTOP (Real-time on-board Processing) was performed by TU Dresden and Jena-Optronik GmbH. This extension was focused on real-time prototyping and testing of selected algorithms for processing of infrared images and LIDAR data into representative commercially available processing hardware: a representative flight computer, a representative FPGA. The algorithms were selected, taking into account their possible usage for real-time on-board image processing in the context of Active Debris Removal scenarios. Another objective of the extension was preliminary allocation of the IRPN algorithms to a (representative) hardware architecture. The preliminary allocation of the IRPN algorithms was iteratively tried considering several system configurations starting from single LEON4-N2X CPU (Cobham Gaisler AB) and ending with system consisting from multiple Freescale P4080 CPUs (NXP Semiconductors) and radiation hardened Virtex-5QV FPGAs (Xilinx, Inc.). The second part of the extension was aimed at the implementation and testing of selected basic LIDAR algorithms and infrared image recognition and processing (IR-IRP) algorithms in a PIL facility consisting of one LEON4-N2X CPU and one FPGA Virtex-4 XC4VLX200. The PIL facility was built on basis of RASTA development platform provided by ESA as Customer Furnished Item (CFI). LIDAR algorithms were provided, implemented and tested by the Jena-Optronik GmbH using symmetric multiprocessing on LEON4-N2X. FPGA implementations and tests of selected IR-IRP algorithms were accomplished by TU Dresden using High-Level-Synthesis approach. One IR-IRP algorithm was implemented on the LEON4 by Jena-Optronik GmbH for comparison and it was found that the FPGA implementation can run about 100 times faster than CPU of comparable clock frequency. The FPGA implementations can also be ported to radiation hardened FPGA Virtex-5QV.

© F. Schnitzer

Fig 16: RASTA Development Platform (ESA CFI)

ACKNOWLEDGMENTS

The activity Image Recognition and Processing for Navigation was funded by ESA/ESTEC. The Institute of Automation at Technische Universität Dresden was the prime contractor. The project consortium furthermore comprised Airbus Defence & Space (Friedrichshafen and Bremen), Astos Solution GmbH and Jena-Optronik GmbH. In any case, the view expressed in this publication can in no way be taken to reflect the official opinion of the European Space Agency.

Abstract

While today's robotic space missions are largely pre-planned and controlled by ground personnel, future flight vehicles are expected to act autonomously to a much greater extent. This concerns in particular the capabilities for navigation, environment sensing and motion planning while ensuring the highest level of operational safety. At the same time, costs are to be minimized, which requires "small" systems in terms of size, mass and power consumption. In this context, the project aims to develop a fully autonomous landing procedure in an unknown environment and demonstrate it using a functional model.  The demonstration is to be carried out with an octocopter (8-rotor aircraft) in challenging terrain. In this way, the results are also directly usable for terrestrial applications. This is of interest for various institutions such as disaster control, police or mountain rescue, which increasingly use unmanned aerial vehicles, so-called UAVs. Another challenge to be considered is the increased demands on the reliability and robustness of the system. Under the general constraint of limited size and on-board resources ("small"), classical redundant arrangements are not feasible. Instead, novel RAMS (Reliability, Accessability, Maintainability, Safety) approaches and methods of analytical redundancy are to be applied at the system level. A special feature is the explicit consideration of the inherent interaction between the navigation process and the guidance process (motion planning). Motion planning indirectly influences the quality of the environment map via the perceptual performance of the sensor system, which in turn is the basis for motion planning. These interdependencies are to be investigated for the synergetic optimization of both subprocesses.

Abstract

Autonomous satellite servicing operations represent one of the current challenges of space flight. A key technology in this context is relative navigation. The goal of this research project is the development and computer-aided verification of primarily image-based estimation methods for relative navigation between two spacecraft. The target spacecraft is assumed to be non-cooperative, unknown in its geometry a priori, and tumbling. As a result, not only its motion but also its 3D structure should be known.

3D shape reconstruction  from image-based point clouds   for unknown and non-cooperative spacecraft

• simulation experiment

3D shape reconstruction from image-based point clouds for unknown and non-cooperative spacecraft

• simulation experiment

3D shape reconstruction from image-based point clouds for unknown and non-cooperative spacecraft

• laboratory experiment with stereo-camera and MiPOS - Mini Proximity Operations Simulator at LS-AT

© F. Schnitzer

PDF Download

© A. Sonnenburg

PDF Download

Selected Publications

Main objectives of the project

• Detailed design of the navigation system based on the optical flow techniques with optical correlator and inertial measurements
• Performing the simulation campaign with an image simulator in the loop to evaluate the complete navigation chain performances under representative mission constrains
• Providing the optical correlator hardware development plan for the optical flow navigation experiment on a real-time test facility

© V. Chernykh

© V. Chernykh

Abstract

The feasibility of proposed concept of visual navigation system for planetary lander has been proved in open-loop simulation experiment, considering the Mercury landing scenario as a reference mission. During simulated Visual Guidance Phase, RMS error of position determination has been determined to be within 1% of distance to landing site; RMS attitude error ? about 0.5 degrees and RMS velocity error - about 1%. Obtained accuracy can be further improved by appropriate filtering of navigation data The effect of the resolution of the reference 3D model of landing site on the accuracy of the obtained navigation data has been investigated. Standard deviation of position error, caused by insufficient resolution of the reference model, has been found to be within one half of the sampling distance of the model. The effect of the errors of rough estimation of position/attitude/velocity prior to the start of visual navigation phase on the navigation data accuracy has been investigated. A good estimation convergence has been obtained, when the initial estimation errors were within the following limits:

• position estimation error - 22% of the initial distance to landing site; attitude estimation error - 14 degrees,
• velocity estimation error - 45% of initial velocity. In this case the effect of initial errors was practically compensated after 3 to 4 measurements.

© K. Janschek

Abstract

The trend in Earth observation is toward ever higher resolutions and more detailed measurements. On the satellite platform side, this requires an ever higher alignment accuracy of the instruments as well as an exact allocation of the image information to the earth's surface.  Based on the experience with image motion analysis, a concept for the use of multispectral images for the state estimation of geostationary observation satellites was developed with the specific inclusion of supplementary information.

© E. Zaunick

© S. Dyblenko

Abstract

A novel approach for autonomous navigation of satellites is presented. The proposed navigation concept uses robust image motion tracking for acquisition of navigational data and requires minimum a priori information. The system incorporates a single compact image sensor and a high speed optical image processing unit to provide real-time performances. With the possibility of determination of both position and attitude estimates this system could be a basis for full autonomous operation of a satellite. This can be of particular interest in context with missions to other planets: Mars, Europa, etc.

The results and products of the project are:

• concept of optoelectronic system for autonomous navigation of satellites;
• navigation and attitude determination algorithms based on image motion tracking;
• analysis of navigation system performances (theoretical and simulation studies);
• detailed simulation models of all system components and algorithms
  system limitations and constraints;
• general specification of the system hardware.

Abstract

Key topics of this research project are the development of the appropriate methods and algorithms those realize the information technology of remotely sensed data processing and analysis . This includes research activities in the following areas: segmentation, filtering of images obtained in conditions of prior uncertainty and irregularity of noises, particular in the frame of wavelet and fractal analysis; signal detection and estimation on the base of diffractive optical elements (DOEs) and spectral analysis.

Team

Objectives

Conceptual and technology development for aerial manipulation with free fyling fully actuated multicopters.

© K. Janschek

Download PDF

© Chao Yao

Download PDF

Selected Publications

Abstract

Future air and space vehicles should be able to act autonomously to a greater extent. The key functions required for this are navigation and motion planning. There have been and still are numerous research activities in both areas, but most of them consider one of the functions in isolation. The goal of this project is to investigate both functions in an integrative approach, as well as to explore their interactions and use them to improve motion planning. Autonomous flight through an unknown area with high obstacle density serves as a reference scenario. This area is to be traversed safely (guaranteed collision-free) while maintaining a sensible course. The flight platform is assumed to be a fixed-wing aircraft. This type of aircraft has additional trajectory constraints (flight level, minimum speed) compared to rotor-powered aircraft (helicopters, multicopters) and is therefore a special challenge for motion planning.The advantage of an integrated treatment of navigation and motion planning can be seen in the situation shown below. If the trajectory planning chooses a path far away from distinctive orientation features (left picture), the navigation quality suffers in such a way that in the later course of the flight a safe crossing of the opening between the two obstacles is no longer guaranteed. If, on the other hand, motion planning takes into account the requirements of the navigation process (right image), an improvement in navigation quality can also be expected, which in turn has a positive effect on motion planning. In the concrete example, navigation benefits from curved (accelerated) trajectory sections as well as sufficiently many distinctive environmental features in the sensor field of view in such a way that the initial "detour" is more than compensated for by a shortened route in the later course of flight.

Observability-driven 6DOF body-centric monocular SLAM   (M. Markgraf)

• here shown: full observability during accelerated motion with monocular vision

Hybrid Planning and Optimization Concept   (M. Seeman)

• here shown: RRT* based planning in dense environment for fixed wing aerial vehicle with nonholonomic constraints

© M. Markgraf

Download PDF

© M. Seemann

Download PDF

Selected Publications

Abstract

The junior research group ADFEX (Adaptive federative 3D exploration with multi-robot systems) pursues the goal of quickly and reliably acquiring, processing and relaying spatial information on unknown large-scale environments or changing scenes. The three-dimensional spatial explorations will be performed by a fleet of multiple flying robots. These flying robots will be equipped with different sensors and will act semi-autonomously, i.e. they will be coordinated and controlled via a simple control console. The pilot application is intended to be the detection of structural damage on objects that are difficult to access (bridges, large buildings) or construction monitoring on extensive construction sites. By using such cooperating multi-robot systems, spatial reconnaissance can be automated, greatly simplified and carried out with greatly improved quality. However, the use of such a fleet of flying robots is not limited to the construction industry, but can be transferred to a number of other application areas, e.g. agriculture (precision farming), environmental ecology or disaster management. The three flying robots are based on octocopters (8-rotor flying machines). Different (partly high-resolution) cameras (RGB, NIR), laser scanners, ultrasonic sensors as well as GPS and inertial sensors serve as sensor technology. The particular strength of the ADFEX junior research group lies in its interdisciplinary approach. Ten scientists from the scientific fields of automation technology, image processing, 3D visualization and real-time data processing in embedded systems are developing important key functions and key technologies for such multi-flight robot exploration systems in close cooperation with application specialists from the construction industry. The ADFEX junior researchers are also collaborating with a number of small and medium-sized enterprises from Saxony. The activities of the young scientists thus strengthen Saxony's visibility as a science location and innovation pool for the service robotics market segment, which is becoming increasingly important.

Selected Publications

Abstract

© F. Keppler

The vision of fully automated driving on public roads raises several challenges such as robust scene interpretation in unknown terrain and in various environmental conditions. In well-defined automation zones such as logistic centers, high-precision maps, infrastructure-side sensors and adapted legal provisions enables automated driving in very near future.

Many required functionalities – including cooperative trajectory planners and following controllers – have been studied and developed in the broad research field of mobile robotics throughout the past decades. However, most techniques for trajectory coordination assume circular or car-like robot geometries. This assumption falls short for complex vehicle structures such as articulated truck-trailer combinations.
Within this project, a cooperative trajectory planner, which takes account of the complex kinematics and chassis geometries of articulated vehicles, is developed and integrated into the Fraunhofer IVI’s yard automation framework helyOS. Strategies on how to repair coordinated motion plans upon disturbances are investigated and formalized analyses of algorithm sensitivity and performance are carried out.
 

© F. Keppler

Selected Publications

© S. Horn

Abstract

An autonomous mobile platform must be able to operate in a partially unknown and dynamic environment. This requires the acquisition of diverse enviromental information, which is generally affected by noise. Assuming that the probability distributions of the uncertainties are unknown, but bounded, interesting estimates can be modeled as sets. This thesis introduces a novel set-based navigation concept, which handles set-based information safely and productive.
To achieve a high degree of accuracy in the platform localization, multifarious information from different sources is gathered and merged to obtain a more precise pose estimation area. This merger is called set-based multifusion. Similarly the principle of multifusion can be applied to the platform guidance. Here diverse information has also to be combined to calculate a collision-free path to the goal. The verification of the presented set-based algorithms was conducted via real and simulated experiments.

© S. Horn

Poster Download

Selected Publications 

© M. Hennig

Abstract

The APECA framework provides a flexible control architecture for robotic systems for behavior-based control. The conceptual design combines different requirements identified in miscellaneous robotic control approaches. The main focus of the resulting concept is on a simplified engineering process for the controller design. This  approach is supported by the use of atomic system behaviors, the so called action  primitives, in a special module hierarchy.
For this purpose a distinction between a functional behavior based system model with hierarchically and also parallelly executed action primitives and a sequential control system model with a task-dependent activation of the primitives is proposed. These models are assigned to different users through a distinct user concept. An object-oriented implementation of the proposed architecture allows the utilization and synchronisation of multiple (sub-)systems within one framework. In this work the proposed framework will be discussed, a prototypical implementation will be presented and results based on different experimental scenarios will be shown.

© M. Hennig

Selected Publications

Abstract

• Proposed a bio-mimetic solution for optical flow-based perception of visual navigation systems
• Detailed design of the behavior-based controll system based on the optical flow-based perception
• Developing the system to the module-based control program applicable generally for autonomous mobile robots with various sensors.

© Ngoc Anh Mai

Optical flow-based perception, behavior-based control, and topological path planning for mobile robots using fuzzy logic concepts (Mai Ngoc Anh)

In this thesis, a new approach of optical flow-based perception through qualitative interpretations is developed. The qualitative optical flow interpretations provide mobile robots with visual perception a more detailed image of their 3D working environment, e.g. obstacle positions and indoor object types. By using fuzzy logic for the interpretations, the optical flow perception becomes simple and intelligent in a bioinspired manner and moreover gains robustness under noisy conditions in the working environment.
On the other hand, this thesis develops a generic modular structure of a behavior-based control system with three clearly separate modules for perception, motion control, and path planning. The system concept is independent of the specific type of perception. The designed behaviors are functionally classified into two separated modules, concerning collision-free motion control and goal oriented path planning. Some of the behaviors use fuzzy logic concepts, which result in flexible and smooth robotic motion.
Furthermore a new scheme for topological path planning in combination with fuzzy-based behaviors is developed for the goal-oriented navigation of a mobile robot. This combination allows a mobile robot to perform topological path planning in a real environment without metric information regarding its global and local positions.

© Ngoc Anh Mai

PDF Download

Selected Publications 

Abstract

• Simulation environment for the design of navigation strategies of two-wheeled autonomous robots;
• Integration of navigation strategies into the firmware of a vacuum cleaning robot;
• Methodology for the documentation of industrial software products.

Abstract:

• Development of a simulation environment
• Development of a sensor system for global localization
• Development of a navigation strategy
• Development of a novel motion control system.

© K. Janschek

PhD Thesis Abstract by Karim Bondoky Spacecraft (S/C) docking is the last and most challenging phase in the contact closure of two separately flying S/C. The design and testing of S/C docking missions using software-multibody simulations need to be complemented by Hardware-In-The-Loop (HIL) simulation using the real docking hardware. The docking software multibody simulation is challenged by the proper modeling of contact forces, whereas the HIL docking simulation is challenged by proper inclusion of the real contact forces. Existing docking HIL simulators ignore back-reaction force modeling due to the large S/C sizes, or use compliance devices to reduce impact, which alters the actual contact force. This  dissertation aims to design a docking HIL testbed to verify docking contact dynamics for small and rigid satellites by simulating the real contact forces without artificial compliance.
HIL simulations of docking contact dynamics are challenged mainly by:

• HIL simulation quality: quality of realistic contact dynamics simulation relies fundamentally on the quality of HIL testbed actuation and sensing instrumentation (non-instantaneous, time delays)
• HIL testbed design: HIL design optimization requires a justified HIL performance prediction, based on a representative HIL testbed simulation, where appropriate simulation of contact dynamics is the most difficult and sophisticated task.

The goal of this dissertation is to carry out a systematic investigation of the technically possible HIL docking contact dynamics simulation performances, in order to define an appropriate approach for testing of docking contact dynamics of small and rigid satellites without compliance and using HIL simulation. In addition, based on the investigations, the software simulation results shall be validated using an experimental HIL setup.
To achieve that, multibody dynamics models of docking S/C were built, after carrying out an extensive contact dynamics research to select the most representative contact model. Furthermore, performance analysis models of the HIL testbed were built. In the dissertation, a detailed parametric analysis was carried out on the available models’ design-spaces (e.g., spacecraft, HIL testbed building-blocks and contact dynamics), to study their impacts on the HIL fidelity and errors. This was done using a generic HIL design-tool, which was developed within this work. The results were then used to identify the technical requirements of an experimental 1-Degree-of-Freedom (DOF) HIL testbed, which was conceived, designed, implemented and finally utilized to test and validate the selected docking contact dynamics model.
The results of this work showed that the generic multibody-dynamics spacecraft docking model is a practical tool to model, study and analyze docking missions, to identify the properties of successful and failed docking scenarios before it takes place in space.
Likewise, the "Generic HIL Testbed Framework Analysis Tool" is an effective tool for carrying out performance analysis of HIL testbed design, which allows to estimate the testbed’s fidelity and predict HIL errors.
Moreover, the results showed that in order to build a 6DOF HIL docking testbed without compliance, it is important to study and analyze the errors’s sources in an impact and compensate for them. Otherwise, the required figure-of-merits of the instruments of the HIL testbed would be extremely challenging to be realized.
In addition, the results of the experimental HIL simulation (i.e., real impacts between various specimen) serve as a useful contribution to the advancement of contact dynamics modeling.

Docking example for two small spaccraft

• real behavior vs. HIL simulation with parasitic dynamics
• unrealistic physical HIL simulation may suggests wrong docking result because of parasitic dynamics
• here the first experiment suggests wrongly "failed docking"

© K. Bondoky

PDF download  Poster

Selected Publications

© K. Janschek

Download PDF

Abstract

CORRAL image registration technology developed by Institute of Automation at Technische Universität Dresden provides mutual registration of sequences of overlapping satellite images. High registration accuracy is achieved also in presence of non-rigid frame-to-frame distortions and image content alteration due to frame-to-frame variations of spectral sensitivity. In a precursor project the CORRAL image registration algorithms have been demonstrated successfully for the processing of the raw images produced by hyperspectral imager with Linear Variable Filter (LVF). The CORRAL technology has shown in particular as enabling technology for LVF-based hyperspectral imaging from loosely stabilized satellites. As a result of the recently finished CORRAL activity the CORRAL algorithms have been optimized to enable real time operation with limited computational resources and the implementation strategy has been developed. CORRAL image registration algorithms are also suitable for processing of the images affected by the other distorting factors, such as panoramic distortions and topography displacements, expanding the potential application area to High Resolution Earth observation missions, visual navigation for planetary landers and flying robots as well as correction of the imaging LIDAR data. The industrial partner Jena-Optronik (JOP) aims to use CORRAL technology for imaging solutions including earth observation and relative navigation applications. These applications comprise:

• Camera systems for tracking of e.g. fires / floods etc.
• Postprocessing of LIDAR point clouds for satellite service missions.
• Sensor fusion (Lidar provides excellent resolution in z, camera in x and y) sensor fusion provides a higher value for the customer.
• Station keeping during docking manoeuvres via camera sensor (in sub pixel niveau at high frame rates

© V. Chernykh

Download PDF

© K. Janschek

Abstract

Earth observation from the geostationary orbit (GEO) offers a number of advantages with respect to the Low Earth Orbit (LEO) imaging missions (short revisit time, continuous data downlink, possibility of real time observation; optimal using of clouds clearance, etc.). The disadvantages are the necessity to use a telescope with an approximately 45 times larger focal length to provide the same ground resolution at nadir as achieved with a LEO satellite(due to 45 times larger orbit altitude). Besides the difficulty of the large optics realization, it results also in 45 times larger sensitivity to telescope pointing instability. For the high resolution GEO observations (few meters ground sampling distance), an ultra-high pointing stability during integration (well below 0.1 arc second) will be required to avoid motion blur and MTF degradation. The project is devoted to an analysis of a group of alternative solutions for pointing instability compensation based on the precise real time measuring of pointing vector during integration time with onboard optical correlator. For each scene a sequence of short-exposed sub images will be taken, they will be mutually registered and stacked; thus the mechanical pointing error will be compensated. Using this technology the requirements for the satellite attitude stability can be significantly relaxed, what is particular essential for high resolution GEO imaging mission. The project makes use of the hardware model of the Optical Correlator (OCOR Model), developed and manufactured within the frame of previous activity (Optical Correlator for Push-Broom Imagers, funded by ESA / GSTP programm).

In particular, the project activity includes:

• Development of the concept of imaging system with optical correlator for a geo-stationary high resolution imaging mission.
• Development and validation of a testbed for real-time testing of an optical correlator for high-resolution imaging from geostationary orbit. The testbed will include a Test Image Generator to simulate scene images like they will be acquired by optical imagers in GEO orbit, hardware OCOR Model to perform mutual registration of the sub-images, and Image Processing Unit to integrate the sub-images.
• Development of the System Performance Model.
• Conceptual design of a vacuum compatible electronic assembly for OCOR Model.
• Vibration tests on the OCOR Model hardware.

Selected Publications

© K. Janschek

Project goals

• Development, test and validation of the elegant Breadboard Model of the Optical Correlator, suitable for demonstration of the optical flow navigation during the flight test onboard the Precision Landing Test Facility (ESA)
• Estimation of the space compatibility of the main optoelectronic components

Features of the model

• Based on the MEMS spatial light modulator

• Off-the shelf optoelectronic components

• Designed to be used as test bench for space compatibility estimation of the main optoelectronic components, not optimized for minimal size/mass/power consumption

• Compatible with the Precision Landing Test Facility (ESA) on the base of an unmanned helicopter (vibrations tolerant, dust protected)

Performances

• Dimensions: 200x200x400 mm
• Mass: 11.4 Kg (with navigation camera)
• Power consumption: 50 W
• Correlations rate: 1024 correlations/s
• Inputs: nav. camera images; reference image of the test site; estimated camera pose
• Output: corrected camera position/attitude

Abstract

The objective of this project is to make use of the existing optical correlator technique and to identify additional application areas, beside its application for the determination of the satellite pointing vector for pushbroom imagers. This could be any application in which an image processing is performed by a Fourier-transformation or a similar transformation, which can be optically processed. The scope of this activity comprises the identification of possible applications and their preliminary assessment. The most attractive application for utilization in space shall be proposed for a detailed design study in a Mid-term meeting. This selection requires the approval of the Agency. Within this detailed design study the application shall be defined to a conceptual design level. The benefits of the application for space utilization shall be highlighted. The emphasis of the detailed design work shall be put on the thorough study of the critical areas and all the constraints of the application for space utilization.

© K. Janschek

SmartScan - Airborne Test

• image motion recorded by sequential images from area image sensors in image plane
• image motion trajectory derived from image-motion tracking by onboard embedded optical correlator
• post-flight pushbroom image correction using recorded image motion trajectory

Project Overview

This contract addresses an innivative solution for image corrections of satellite pushbroom cameras due to satellite attitude instability effects. Satellite cameras with linear sensors are particularly sensitive to attitude errors, which cause considerable image distortions. A novel solution of distortions correction is presented, which is based on the real-time recording of the image motion in the focal plane of the satellite camera. This allows to use such smart pushbroom cameras (multi-/hyperspectral) even on moderately stabilised satellites, e.g. small sat's, LEO comsat's. The proposed concept uses in-situ measurements of the image motion with additional CCD-sensors in the focal plane and real-time image processing of these measurements by an onboard Joint Transform Optical Correlator.

Phase A

(1) build a hardware laboratory model for

• proof of concept, i.e. proof of functional system feasibility;
• performance estimation of the system performances (data processing rate, accuracy, sensitivity to noise and distortions, response time, minimum resolution under disturbing conditions).

(2) derive a system specification (1-st Order Model) for a space compatible system.

Phase B

The proof of feasibility of the concept and detailed performance figures have been derived with a optical processor hardware model and a pushbroom camera breadboard model, developed and manufactured under this ESA-contract.

Phase C

The project objectives are preparation and implementation of flight tests of a real time demonstrator of an advanced push broom camera. Due to image motion measurement technique this smart imaging system can provide high quality images even from moderate stable satellite platforms. The functionality and performances of the advanced push broom camera have been successfully tested during two probe flights on the aeroplane provided by Deutsches Zentrum für Luft- und Raumfahrt, Bonn.

Selected Publications

© optomechatronics

Download als pdf-Datei

Selected Publications

© K. Janschek

Abstract PhD Thesis R. Schroedter

Monocrystalline silicon microscanners are increasingly used in applications for image projection, distance sensors, spectroscopy and laser beam control, because they allow the miniaturization and mass production of optical systems. With the static beam positioning and linear raster scanning abilities in a wide range of zero to several hundred hertz the so-called quasi-static microscanners offer a much broader and more flexible range of applications compared to previous resonantly oscillating microscanners. However, the movement is superimposed with natural oscillations due to the extremely low damping of the system. Therefore, an open-loop and closed-loop control is necessary for the highly dynamic and accurate beam control with quasi-static microscanners.  In this thesis different concepts for open-loop and closed-loop control of quasistatic microscanners with electrostatic comb drives are designed and verified experimentally on a real-time system with optical feedback (see Figure). The nonlinear mechatronic model becomes completely parameterized and suitable trajectories with jerk limitation are derived. The control of the microscanner on a microcontroller with feedback by the on-chip integrated piezoresistive position sensors is validated realizing a 2D raster scan. As a result, conclusions for the system design are derived for quasi-static microscanners.

Selected Publications

© K. Janschek

© K. Janschek

Abstract PhD Thesis Th. Kaden

Optomechatronic systems encapsulate mechanical, electrical and optical components
to fullfill their specific product task. Examples are systems for optical signal
processing, interferometer and adaptive/active controlled telescopes. The design
process of such complex systems requires an integrative software tool for modeling
and simulation. The object oriented modeling language Modelica established over
the last years for modeling heterogeneous systems. In addition to various model
libraries there are several commercial or open source software tools available. Up to
now however there exists no model library for the simulation of optical components
or systems. Therefore in this work diffractive optical models based on the angular sectrum
of plane waves are integrated in a modular way into Modelica. To achieve this
goal a concept is used that allows Modelica to quickly process the neccessary
two-dimensional signal processing (in particular matrix calculation). Additionally
optical interfaces are introduced in form of cut planes in space and the behavioural
description of optical components is stated. Abstract classes are used to describe
the coupling between optics and the multibody system for transmissive and reflective
optical components. This functionality can be used by specialized optical
components due to inheritance.
The sucessful interaction of optical, mechanical, electrical and signal processing
components is proofed by representative optomechatronic systems. Those include a
Fourier optical 4f-system, an electrostatically driven micro mirror in a reflective
setup and a closed-loop controlled, actively stabilized Michelson-Interferometer.

© Th. Kaden

Download Poster

Selected Publications

Abstract

 The Iridium telephone network is an example of the use of satellite-based messaging systems. In future networks, however, broadband communication channels will increasingly have to be provided. For these applications, optical data links between different satellites are very well suited because they guarantee a large bandwidth together with a low mass and size as well as a low power requirement of the transmitting and receiving terminal. Another advantage is the high immunity to interference of the transmitted signal. In Europe, there have been research projects involving optical satellite terminals since the 1980s. The Swiss company Oerlikon Con-traves Space in Zurich is developing the SROIL terminal (Short Range Optical Intersatellite Link) under an ESA contract. This terminal is part of a future family that will cover all conceivable optical satellite links and can connect 2 satellites over distances of up to 2400 kilometers. Data rates up to 1.5 Gbps shall be achieved.
The goal of this diploma thesis was to investigate and optimize the transmission behavior of the SROIL terminal especially in the presence of interference. Here, the micro-vibrations generated by the satellite itself are to be mentioned above all. Furthermore, there are also internal sources of interference that can negatively influence the transmission behavior. Within the scope of this work, the existing system descriptions and mathematical equations were combined into a model that describes the beam steering system of two terminals and the optical link between them. Due to the complexity of the SROIL terminal, it was necessary to simplify this system description to such an extent that a design of an optimization strategy became possible. Furthermore, all sources of interference that have an influence on the transmission behavior were analyzed in order to be able to define the parameters of the optimization algorithm. The results found were verified by means of a system simulation created in Matlab/ Simulink.
Two promising algorithms were found to optimize the beam alignment of the SROIL terminal. Finally, one algorithm was selected, which is planned to be implemented in the terminal.

Abstract

 The aim of the project is to develop a high-speed scanning system based on optical Fourier transformation using scanning lasers for process optimization in the production and processing of (partially transparent) high-filament technical textiles (e.g. filter gauze, screen printing screens, etc.), which is associated with an increase in productivity and a reduction in waste, which in turn leads to a reduction in energy consumption and carbon dioxide emissions. A scanning laser generates a Fourier spectrum of the web material on a high-resolution image sensor without contact, from which the parameters, thread count and thread position are calculated. Another light source (e.g. LED, laser) generates images of the moving fabric, which are recorded by a low-resolution high-speed matrix sensor. A DSP calculates the speed of the web and the position of production defects from the variation of the optical flow. The system is used both in fabric production (circular knitting machines, weaving machines) and in thermal treatment (shrinking, heat setting) in the stenter frame.

© S. Dyblenko

Abstract

Paper, board and cardboard are very versatile products and play an important role in today's life. The global competitive situation as well as the strong price pressure in paper distribution place high demands on productivity and the quality of the paper offered. In modern production plants, efforts are therefore made to optimize paper production. Quality control plays an important role in this. The overall objective of the project is to improve quality assurance in the paper industry by using an innovative high-performance quality measurement technology. This measurement technology will enable a robust and accurate online assessment of the uniformity of the internal fiber distribution (paper formation) in the paper. This can be used to improve other quality parameters such as printability, surface finish, strength, stiffness, optical quality, etc. The image-based measurement method is based on powerful image processing algorithms which, together with the robust image acquisition to be developed, enable significantly better measurement performance compared to the state of the art. Within the scope of the joint project, the existing knowledge of formation measurement at the Institute of Automation Technology will be expanded and, using the practical knowledge and experience of dresden elektronik ingenieurtechnik gmbh in industrial electronics, translated into a design for a new powerful and robust formation measurement technique. The technical solutions to be developed in the project for robust image acquisition and powerful image evaluation can be used for quality monitoring of railroad goods in other industrial sectors (e.g. steel or textile industry). The hardware-technical solutions can also contribute to a considerable increase in performance in general image processing tasks. The technical solutions to be developed in the project for robust image acquisition and high-performance image evaluation can be used for quality monitoring of web products in other industrial sectors (e.g. steel or textile industry). The hardware solutions can also contribute to a considerable increase in performance in general image processing tasks.

© K. Janschek

© K. Janschek

Abstract

The research project includes methodological and application investigations in the field of optical automated testing for paper technology. The research results aimed at are:

•     Development of a solution for optical quality inspection in paper production using innovative technologies of optical information processing,
•     Development of an industrially suitable concept for an optical 2D spectral processor,
•     Construction and testing of a laboratory model of the optical 2D spectral processor under real operating conditions.

© A. Braune

Teilthema: Kommunikationssysteme und multimodale Bedienoberflächen

Beteiligte Forschungsstellen:

• TU Dresden, Institut für Textilmaschinen und Textile Hochleistungswerkstofftechnik, Professur Konfektionstechnik: (Prof. Dr.-Ing. habil. Rödel), Prof. Dr. Sybille Krzywinski
• TU Dresden, Institut für Akustik und Sprachkommunikation, Professur Akustik und Haptik: Prof. Dr.-Ing. habil. Ercan Altinsoy
• TU Dresden, Institut für Automatisierungstechnik, Lehrstuhl für Automatisierungstechnik: Prof. Dr. techn. Klaus Janschek
• GFaI - Gesellschaft zur Förderung angewandter Informatik e.V., Bildverarbeitung/Industrielle Anwendungen (BVIA): Dipl.-Ing. Frank Püschel

Projektinhalt:

Problemstellung:
Die Oberflächen von Konsumgütern ebenso wie die industrieller Produkte besitzen multimodale Eigenschaften, d. h. sie sind durch die Einheit visueller, auditiver und haptischer Eigenschaften gekennzeichnet. Produzenten und potentielle Käufer sind aber heute nicht in der Lage, solche Eigenschaften als multimodale Einheit über räumliche Entfernungen, d.h. mithilfe von Telepräsenzlösungen (Virtual-Reality (VR)-Anwendungen) zu erfassen und zu bewerten. Ursache sind fehlende akzeptable Geräte und plausible Modelle zur wahrnehmungsgerechten Aufnahme sowie zur Beschreibung und zur Wiedergabe von Produkteigenschaften.

Projektziel:
Im Vorhaben sollen Verfahren zur Texturerfassung, -beschreibung, -verarbeitung und wiedergabe für multimodale (haptische, visuelle und auditive) Human-Machine-Interface Lösungen (HMI-Lösung) zur räumlich entfernten Inspektion textiler Produktoberflächen entwickelt werden. Dabei spielen u.a. Aspekte der Datensynchronisation und -übertragung ebenso eine Rolle wie die Datencodierung, die unter Berücksichtigung des Auflösungsvermögens der technischen Wiedergabe und visuellen Erfassung gestaltet werden soll. Struktur der entwickelten Gesamtlösung zur multimodalen Inspektion von Produktoberflächen:

  Beispiel einer Multimodalen Wiedergabe textiler Produktoberflächen

© A. Braune

KONTRANS - Kontextabhängige Transformationsmodelle

Projektinhalt:

Im modellgetriebenen Entwicklungsprozess von Benutzungsschnittstellen sind  Transformationen das verbindende Element zwischen den formalen Modellen der verschiedenen Abstraktionsebenen. Ihre Qualität und Wiederverwendbarkeit sowie der Erstellungsaufwand bestimmen somit die Effizienz des gesamten Vorgehens. Trotzdem werden Transformationen heute noch meist spezifisch für die jeweils genutzten Metamodelle (Beschreibungssprachen der Modelle) bzw. für spezielle Kontexteigenschaften  ausprogrammiert. Der dafür erforderliche erhebliche Aufwand kann  unter Umständen sogar die Vorteile des modellgetriebenen Vorgehens in Frage stellen.

Zielstellung:

Ziel dieses Projektes ist die Entwicklung von Methoden zur Formulierung wiederverwendbarer universell nutzbarer Transformationsvorschriften. Entwickelt werden konfigurierbare Transformationsmodelle, die zulässige Abbildungen zwischen Modellelementen unabhängig von einer Implementierungssprache allgemeingültig und wiederverwendbar beschreiben. Die Abhängigkeiten von Kontexteigenschaften werden nicht mehr implizit in den Transformationsvorschriften programmiert, sondern ebenfalls explizit als Teil der Transformationsmodelle konfiguriert.

© A. Braune

Selected Publications

© A. Braune

Remote evaluation of product surfaces using 5G technologies

Abstract:

The remote testing of product surfaces can help to reduce  the high return rates in
online trade. For this purpose it is necessary to stream visual, auditory and haptic content
via the internet with minimal time delay between haptic inputs and outputs.
Here, the new communication standard 5G, is expected to gain promising approaches for to minimizing the occurring transmission latency in the context of
its use case “Critical Communications”. In this project, a streaming server for the transmission of multimodal data is being developed and tested.  

The aim is to reduce latency with the help of 5G concepts. First successes were achieved with the so-called network slicing concept.

© A. Braune

Beteiligte Forschungsstellen:

• TU Dresden; Institut für Textilmaschinen und Textile Hochleistungswerkstofftechnik
• TU Dresden, Institut für Feinwerktechnik und Elektronik-Design, TU Dresden
• TU Dresden, Institut für Automatisierungstechnik

Beschreibung:
Electronic combined textiles, sog. E-Textiles sind Stoffe oder Gestricke, in die leitfähige Garne und elektronischen Baugruppen eingebettet werden. Sie gewinnen zunehmend an Bedeutung für intelligente Bekleidungs-, Heim- oder technische Textilien. Eine besondere Herausforderung bei deren Herstellung stellen Technologien zur Kontaktierung dar, die sowohl den Eigenschaften der Textilien und der elektrischen Komponenten gerecht werden müssen als auch den Anforderungen der späteren Benutzung. Bisher ist die Marktdurchdringung noch gering, da die eingewebten oder eingestrickten elektrischen Leiter oft noch von Hand miteinander oder mit Kontaktelemente verbunden werden.
Inhalt dieses Projektes ist die Entwicklung einer automatisierten inline-fähigen Fügetechnik während der Herstellung am Beispiel von Strickwaren. Die Aufgabe dieses Teilprojektes besteht im Finden und Systematisieren relevanter Eigenschaften aller beteiligen Materialien, Technologien und Maschinen in Form von Digitalen Zwillingen. Diese bilden die Grundlage für die Entwicklung von inlinekontaktierbaren Gestrickkonstruktionen bis hin zur automatisierten Maschinensteuerung im Fügeprozess.

• System zur Onlinemessung der Abscheidegeschwindigkeit an Prozessen der industriellen Oberflächenbehandlung (SOMA-Projekt)
• Entwicklung und Erprobung einer Technologie zur Vermeidung von PFT-Emissionen über den Abwasser-, Abfall- und Abluftpfad in Produktionsstätten am Beispiel der Oberflächenveredlung
• Prozessmessverfahren zur Onlineüberwachung der Abscheidegeschwindigkeit an Chemisch-Nickel-Prozessen
• Entwicklung einer Prozess- und Anlagentechnik zur abwasser-technischen Behandlung des Teilstroms "Chemisch Nickel"
• Entwicklung eines Messsystems zur qualitätsgerechten Prozessführung von Chemisch-Nickel-Prozessen
• Steigerung der Materialeffizienz und Wirtschaftlichkeit bei der Metalloberflächen-Veredelung mit neuartiger Automatisierung der Produktionskette
• Modellierung und Simulation von Verfahrensprozessen in der Galvano- und Oberflächentechnik
• Abwasserteilstrombehandlung Chemisch Nickel
• Umstellung bestehender galvanotechnischer Anlagen auf eine stoffverlustminimierte Prozesstechnik bei gleichzeitiger Kostensenkung
• Automatisierung einer Produktionsanlage zum Säurepolieren von Bleikristall
• Schadstoffarme Prozesse in Galvanotechnik und Metallchemie

zurück

zurück

zurück

zurück

zurück

zurück

zurück

zurück

zurück

zurück

zurück

Programmier- und Inbetriebnahmesystem für speicherprogrammierbare Steuerungen

Projektleiter: Prof. Dr.techn. K. Janschek (ab 1995), Prof.(em.)Dr.-Ing.habil. E. Habiger
Mitarbeiter: Dr.-Ing. L. Hoffmann, Dr.-Ing. V. Hammer, Dr.-Ing. M. Köhler, Dr.-Ing. D. Trappiel, Dr.-Ing. R. Brunner, Dr.-Ing. A. Safeca, Dipl.-Ing. Th. Weise, Dipl.-Ing. F. Dehner, Dipl.-Ing. J. Hellmann, Dipl.-Ing. M. Seyfert
Finanzierung: INDRAMAT GmbH, Lohr a.M.
Wissenschaftliche Zusammenarbeit: INDRAMAT GmbH, Lohr a.M.
Laufzeit: 1990 - 2000
Beschreibung/Ergebnisse:
Weiterentwicklung und Pflege eines SPS-Programmier- und Inbetriebnahmesystems nach dem Sprachkonzept der IEC 1131-3. Systemausbaustufen befinden sich weltweit in der Anwendung (Bundesrepublik, Schweiz, USA, Großbritannien, Spanien, Südkorea)

Bedientheoretische Modelle

Aufgabe im Rahmen des DFG-Schwerpunktprogramms "Die elektromagnetische Verträglichkeit (EMV) moderner energietechnischer Systeme"
Projektleiter: Prof. Dr.techn. K. Janschek (ab 1995), Prof.(em.) Dr.-Ing.habil. E. Habiger
Mitarbeiter: Dr.-Ing. R. Vick
Finanzierung: DFG Wissenschaftliche Zusammenarbeit: DFG
Laufzeit: 09/94 - 12/99
Beschreibung/Ergebnisse:
Durch die Modellierung des Verhaltens von EMV-Prüfobjekten auf bedientheoretischer Grundlage wurde das Verhalten von rechnerbasierten Prüfobjekten gegenüber zufallsverteilten Prüfpulsfolgen erkundet.

Schirmwirkung von Metallbeschichtungen

Aufgabe im BMBF-Förderprojekt: "Metallisieren von Kunststoffgehäusen für den elektromagnetischen Umweltschutz"
Projektleiter Teilprojekt und Koordinator des Gesamtprojekts:  Prof. Dr.-Ing. habil. Ernst Habiger
Mitarbeiter:  Dr.-Ing. Johannes Wolf
Wissenschaftl. Zusammenarbeit:  SIEMENS AG Berlin und Erlangen sowie acht weitere Firmen
Finanzierung:  SIEMENS AG Berlin
Laufzeit: 07/94 - 12/96, Abschlußbericht 06/97
Beschreibung/Ergebnisse: Die erklärte Zielstellung bestand darin, Konstruktionshinweise für umweltfreundlich herstellbare und problemlos recyclierbare beschichtete Kunststoffgehäuse mit Abschirmwirkung gegenüber elektromagnetischen Einflußgrößen zu erarbeiten.

Grundlagen der EMV-Meß- und Prüftechnik

Projektleiter:  Prof. Dr. techn. Klaus Janschek (ab 1995); Prof.(em.) Dr.-Ing. habil. Ernst Habiger
Mitarbeiter:  Dipl.-Ing. Billy Akpebu
Wissenschaftl. Zusammenarbeit: ABB Forschungszentrum Heidelberg
Finanzierung:  ABB Forschungszentrum Heidelberg
Laufzeit:  01/95 - 06/98, Abschlußbericht 06/98
Beschreibung/Ergebnisse: Wissenschaftliche Untersuchungen zur Ausbreitung von Prüfstörgrößen in elektronischen Schaltungen und Systemen.

Analyse und Reduzierung von geleiteten Störgrößen in getakteten Stromversorgungen

Projektleiter: Prof.Dr.techn. K. Janschek (ab 1995), Prof.(em.) Dr.-Ing.habil. E. Habiger
Mitarbeiter: Dipl.-Ing. Ph. F. Okyere
Finanzierung: ABB Forschungszentrum Heidelberg
Wissenschaftliche Zusammenarbeit: ABB Forschungszentrum Heidelberg
Laufzeit: 08/96 - 06/00
Beschreibung/Ergebnisse:
Wissenschaftliche Untersuchungen zur Entstehung, Ausbreitung und Vermeidung leitungsgebundener Störgrößen in getakteten Stromversorgungen.