Page 25 - Forum-2021-JanToMarch

Floris Gisolf Zeno Geradts Marcel Worring

• Querying structure: Uses the created structure to test
hypotheses and answer questions.
To improve efficiency, several of these tasks can be (par-
tially) automated using computer vision. Computer vision
is the interdisciplinary scientific field of how computers can
make sense of digital images or video. It tries to automate
tasks that humans can do with their visual systems. Comput-
Figure 2. Side view left (top) and right (bottom). Identification of ers nowadays can learn to recognize objects and locations,
wreckage retrieved from the wreckage sites. (Source: Dutch Safety such as cats, cars, Paris, the beach, etc. However, to do so
Board) the computer needs a lot of examples to train on (approxi-
mately 1,000 images per object). While this is no problem for
actual investigation took place, we now switch to the postanalysis everyday objects and locations, airplane crashes and other
in which we consider methods that could have made the process accident sites are often unique in location and the type and
more efficient and that could form the basis for potential future state of objects. State-of-the-art computer vision techniques
investigations. are thus not yet able to do some of the most difficult parts of
The analysis process of a large image collection generally consists the analysis: determining what the object is and where it is
of two phases: located. In combination with the expertise of an investigator,
• Exploration, applicable when the investigator is faced with a however, it can make several tasks much easier.
collection they do not know much about beforehand and wants To assist the investigator in the analysis, and with the lim-
to discover what is inside and/or how the data is structured. itations of the current state of the art in computer vision in
An exploratory session typically takes time and involves a mind, the following tasks were sought to be automated and
dynamic model of the data, continuously refined as the analyst developed into an app:
iteratively gains knowledge. • Cluster images with similar content into groups.
• Search, applicable when the investigator has a clear idea what • Query an image, sorting all images based on their simi-
they are looking for and queries the system for items relevant larity with the queried image.
to certain attributes. A search session is then a sequence of • Query part of an image, sorting all images based on their
query-response pairs, and the analyst expects fast response. similarity with the queried part of the image.
The data model is static, since the investigator knows exactly
what they are looking for, and this can be communicated to the
system through a query.
Tasks related to these phases can be placed on an explore-search
axis (see Figure 5, page 26), with tasks on the left generally preced-
ing tasks more toward the right, but with a lot of switching back
and forth between the different tasks.
• Clustering: Groups images based on similarity to make it easy
for the investigator to find structure in the collection and rela-
tions between images.
• Browsing: Allows the investigator to quickly and intuitively
view the image collection.
• Structuring into categories: Brings relevant structure to the
image collection to more easily make sense of the data.
• Finding relevant items: Finds those images that give infor-
mation that supports hypotheses or answers questions of the
investigator.
• Searching additional relevant items: Images of the same object
or location but from a different angle can give new information.
• Ranking: Sorts images based on content or metadata. Figure 3. Example of finding the location through satellite images.
(Source of top image: Rob Stothard; source of bottom image:
• Querying item: Searches for a specific image. Google Earth)

January-March 2021 ISASI Forum • 25