Semantic Robots is a strategic partnership between the Centre for Applied Autonomous Sensor Systems, which MPI is a part of, and several industrial partners, all leaders in their respective fields. This is a 6-year research profile funded by The Knowledge Foundation for a total of 36MSEK, matched by equal contribution from the industry. The main objective of this effort is to further develop our key research areas through close cooperation with leading industrial players.
MPI is a principle investigator of one of three research directions in this project, with the main goal to create methods and tools for discovering semantic knowledge from aerial imagery. The project is comprised of three major parts: image segmentation and classification, discovering semantic knowledge and building ontologies based on classification results and open available data sources, and visualization.
On the first step, we develop algorithms for classification and segmentation of satellite orthoimagery using convolutional neural networks. From multi-band true-ortho satellite imagery covering the range from 400 to 1040nm, along with high detailed digital elevation models (DEM), we produce rapid classification of the area into five classes: vegetation, ground, water, road, building. Although not perfectly precise, the results of classification allows to build reasoning on the data and incorporate other data sources, such as Open Street Map (OSM). The reasoning will in tern allow to improve classification (for instance, to solve confusion in the difficult areas of the map, for example, in shadows).
Example of SemMap Viewer 2.0 with flood simulation and flood affected areas. Labels show the semantic data of points, when not flooded.
Publications:
2019
Marjan Alirezaie; Martin Längkvist; Michael Sioutis; Amy Loutfi
@article{semreferee,
title = {Semantic Referee: A Neural-Symbolic Framework for Enhancing Geospatial Semantic Segmentation},
author = {Marjan Alirezaie and Martin Längkvist and Michael Sioutis and Amy Loutfi},
doi = {10.3233/SW-190362},
year = {2019},
date = {2019-09-26},
journal = {Semantic Web Journal},
volume = {10},
number = {5},
pages = {863-880},
abstract = {Understanding why machine learning algorithms may fail is usually the task of the human expert that uses domain knowledge and contextual information to discover systematic shortcomings in either the data or the algorithm. In this paper, we propose a semantic referee, which is able to extract qualitative features of the errors emerging from deep machine learning frameworks and suggest corrections. The semantic referee relies on ontological reasoning about spatial knowledge in order to characterize errors in terms of their spatial relations with the environment. Using semantics, the reasoner interacts with the learning algorithm as a supervisor. In this paper, the proposed method of the interaction between a neural network classifier and a semantic referee shows how to improve the performance of semantic segmentation for satellite imagery data.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Understanding why machine learning algorithms may fail is usually the task of the human expert that uses domain knowledge and contextual information to discover systematic shortcomings in either the data or the algorithm. In this paper, we propose a semantic referee, which is able to extract qualitative features of the errors emerging from deep machine learning frameworks and suggest corrections. The semantic referee relies on ontological reasoning about spatial knowledge in order to characterize errors in terms of their spatial relations with the environment. Using semantics, the reasoner interacts with the learning algorithm as a supervisor. In this paper, the proposed method of the interaction between a neural network classifier and a semantic referee shows how to improve the performance of semantic segmentation for satellite imagery data.
@workshop{IJCAI-LR,
title = {A Symbolic Approach for Explaining Errors in Image Classification Tasks},
author = {Marjan Alirezaie and Martin Längkvist and Michael Sioutis and Amy Loutfi},
url = {http://www.iiia.csic.es/LR2018/files/Alirezaie.pdf},
year = {2018},
date = {2018-07-14},
booktitle = {IJCAI Workshop on Learning and Reasoning: Principles & Applications to Everyday Spatial and Temporal Knowledge},
abstract = {Machine learning algorithms, despite their increasing success in handling object recognition tasks, still seldom perform without error. Often the process of understanding why the algorithm has failed is the task of the human who, using domain knowledge and contextual information, can discover systematic shortcomings in either the data or the algorithm. This paper presents an approach where the process of reasoning about errors emerging from a machine learning framework is automated using symbolic techniques. By utilizing spatial and geometrical reasoning between objects in a scene, the system is able to describe misclassified regions in relation to its context. The system is demonstrated in the remote sensing domain where objects and entities
are detected in satellite images.},
keywords = {},
pubstate = {published},
tppubtype = {workshop}
}
Machine learning algorithms, despite their increasing success in handling object recognition tasks, still seldom perform without error. Often the process of understanding why the algorithm has failed is the task of the human who, using domain knowledge and contextual information, can discover systematic shortcomings in either the data or the algorithm. This paper presents an approach where the process of reasoning about errors emerging from a machine learning framework is automated using symbolic techniques. By utilizing spatial and geometrical reasoning between objects in a scene, the system is able to describe misclassified regions in relation to its context. The system is demonstrated in the remote sensing domain where objects and entities
are detected in satellite images.
@article{s17112545,
title = {An Ontology-Based Reasoning Framework for Querying Satellite Images for Disaster Monitoring},
author = {Marjan Alirezaie and Andrey Kiselev and Martin Längkvist and Franziska Klügl and Amy Loutfi},
url = {http://www.mdpi.com/1424-8220/17/11/2545},
doi = {10.3390/s17112545},
issn = {1424-8220},
year = {2017},
date = {2017-11-05},
journal = {Sensors},
volume = {17},
number = {11},
abstract = {This paper presents a framework in which satellite images are classified and augmented with additional semantic information to enable queries about what can be found on the map at a particular location, but also about paths that can be taken. This is achieved by a reasoning framework based on qualitative spatial reasoning that is able to find answers to high level queries that may vary on the current situation. This framework called SemCityMap, provides the full pipeline from enriching the raw image data with rudimentary labels to the integration of a knowledge representation and reasoning methods to user interfaces for high level querying. To illustrate the utility of SemCityMap in a disaster scenario, we use an urban environment—central Stockholm—in combination with a flood simulation. We show that the system provides useful answers to high-level queries also with respect to the current flood status. Examples of such queries concern path planning for vehicles or retrieval of safe regions such as “find all regions close to schools and far from the flooded area”. The particular advantage of our approach lies in the fact that ontological information and reasoning is explicitly integrated so that queries can be formulated in a natural way using concepts on appropriate level of abstraction, including additional constraints.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
This paper presents a framework in which satellite images are classified and augmented with additional semantic information to enable queries about what can be found on the map at a particular location, but also about paths that can be taken. This is achieved by a reasoning framework based on qualitative spatial reasoning that is able to find answers to high level queries that may vary on the current situation. This framework called SemCityMap, provides the full pipeline from enriching the raw image data with rudimentary labels to the integration of a knowledge representation and reasoning methods to user interfaces for high level querying. To illustrate the utility of SemCityMap in a disaster scenario, we use an urban environment—central Stockholm—in combination with a flood simulation. We show that the system provides useful answers to high-level queries also with respect to the current flood status. Examples of such queries concern path planning for vehicles or retrieval of safe regions such as “find all regions close to schools and far from the flooded area”. The particular advantage of our approach lies in the fact that ontological information and reasoning is explicitly integrated so that queries can be formulated in a natural way using concepts on appropriate level of abstraction, including additional constraints.
@inproceedings{AlirezaieEtAl:AnSWeR2017,
title = {Exploiting Context and Semantics for UAV Path-finding in an Urban Setting},
author = {Marjan Alirezaie and Andrey Kiselev and Franziska Klügl and Martin Längkvist and Amy Loutfi},
url = {http://ceur-ws.org/Vol-1935/#paper-02},
issn = {1613-0073},
year = {2017},
date = {2017-05-29},
booktitle = {1st International Workshop on Application of Semantic Web technologies in Robotics (AnSWeR)},
number = {1935},
pages = {11--20},
address = {Portoroz, Slovenia},
series = {CEUR Workshop Proceedings},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Marjan Alirezaie; Martin Längkvist; Andrey Kiselev; Amy Loutfi
Open GeoSpatial Data as a Source of Ground Truth for Automated Labelling of Satellite Images Inproceedings
In: Proceedings of the Workshop on Spatial Data on the Web (SDW 2016) co-located with The 9th International Conference on Geographic Information Science (GIScience 2016), pp. 5-8, 2016.
@inproceedings{DBLP:conf/giscience/AlirezaieLKL16,
title = {Open GeoSpatial Data as a Source of Ground Truth for Automated Labelling of Satellite Images},
author = {Marjan Alirezaie and Martin Längkvist and Andrey Kiselev and Amy Loutfi},
year = {2016},
date = {2016-09-27},
booktitle = {Proceedings of the Workshop on Spatial Data on the Web (SDW 2016) co-located with The 9th International Conference on Geographic Information Science (GIScience 2016)},
pages = {5-8},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
@article{langkvist_classification_2016,
title = {Classification and Segmentation of Satellite Orthoimagery Using Convolutional Neural Networks},
author = { Martin Längkvist and Andrey Kiselev and Marjan Alirezaie and Amy Loutfi},
url = {http://www.mdpi.com/2072-4292/8/4/329},
year = {2016},
date = {2016-01-01},
journal = {Remote Sensing},
volume = {8},
number = {4},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
@inproceedings{ijcaiinteractive2016,
title = {Interactive Learning with Convolutional Neural Networks for Image Labeling},
author = { Martin Längkvist and Marjan Alirezaie and Andrey Kiselev and Amy Loutfi},
url = {https://a3eb7066-a-62cb3a1a-s-sites.googlegroups.com/site/ijcai2016iml/IJCAI2016-IML_paper_13.pdf?attachauth=ANoY7cpyC3pP98kVZSR7MiJJLQiQ6ZZtKOfnZBZf98h-2eb2DWfpIKmbpoq4n9pGpl9yRIsITA4h_JO5vA5ELLzLUwHWj1LnFdmzDjQ1CtJV1H2D6uLkWfXT20AsRKzrTuChgjfRVhcLGYohVRG9nswQLpvShDTMKV5vvQLvcr6NH3vkHVys9M5CtHYCS1n5ZaoFu3DDz8cZk1Vl1FZC14FoUxgf-8R4Bi1Ilqrl6OKJe-ObXuDkjcs%3D&attredirects=0},
year = {2016},
date = {2016-01-01},
booktitle = {IJCAI 2016 workshop on Interactive Machine Learning: Connecting Humans and Machines},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
In: Proceedings of the 24th ACM SIGSPATIAL International Conference
on Advances in Geographic Information Systems, GIS 2016, Burlingame,
California, USA, October 31 - November 3, 2016, pp. 11:1–11:4, 2016.
@inproceedings{DBLP:conf/gis/AlirezaieKL16,
title = {Knowing without telling: integrating sensing and mapping for creating an artificial companion},
author = {Marjan Alirezaie and
Franziska Kl{ü}gl and
Amy Loutfi},
url = {http://doi.acm.org/10.1145/2996913.2996961},
doi = {10.1145/2996913.2996961},
year = {2016},
date = {2016-01-01},
booktitle = {Proceedings of the 24th ACM SIGSPATIAL International Conference
on Advances in Geographic Information Systems, GIS 2016, Burlingame,
California, USA, October 31 - November 3, 2016},
pages = {11:1--11:4},
crossref = {DBLP:conf/gis/2016},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
