Page 26 - Forum-2021-JanToMarch

A high thresh-
old will result in
many small clus-
ters, whereas a low
threshold will result
in fewer, but larger
clusters. This thresh-
old can be changed
by the investigator
to suit the task
and preferences. A
cluster overview can
be generated, which
shows the most rel-
evant image of each
cluster. This enables
the investigator to
quickly find relevant
clusters.
It is then up to
the investigator to
classify the clusters.
In ImEx, the images
in a cluster are dis-
played in a scrollable
canvas at the bottom
Figure 4. Overview of wreckage site 4 of the screen. The
and the location of the wreckage pieces. display size of the
(Source: Dutch Safety Board) images can be ad-
justed. The inves-
Figure 5. The exploration-search tigator can create
axis with example multimedia buckets for holding
analytics (sub)tasks. whole clusters or a
selection of images
Furthermore, the investigator should categories (such as cats, dogs, houses, cars, etc.) by extracting in order to structure
be able to features from images, such as shapes and textures. Features the image collection.
• Browse fluidly through the images. extracted from an image are represented by a value, where Relevant images or
• Place images in user-defined catego- a higher value means the feature is present more frequent- parts of images can
ry “buckets” to structure the image ly and more clearly in the image. In the training phase, the be queried to find
collection. neural network learns which features are best to discriminate additional images.
between categories. By finding these features, it can decide to By generating buck-
• Retrieve and filter images based on ets, and by adding
these buckets. which category an image belongs.
ImEx works slightly different. As noted, training a neural images to these
• Gain information about progress network requires a lot of training examples, which are usually buckets, the image
made in the structuring of the image not available for crash sites or other accident sites. Therefore, collection is given
collection.
rather than classifying images (deciding to which category structure by the user.
We developed ImEx (Incident Image an image belongs), ImEx only calculates whether images A second window
Explorer) with these tasks and features look similar or not. ImEx still makes use of a neural network shows the progress
in mind to assist investigators in inves- trained to classify everyday objects and scenes (such as differ- of structuring the
tigations with large image collections. ent types of animals, sceneries, intact airplanes, other modes image collection
To cluster images based on similarity, of transportation, instruments, etc.). and a Sankey
ImEx makes use of a convolutional neural The neural network used in ImEx extracts 2,048 features diagram to show
network. A brief description follows, as per image. The similarity between two images can then be relations between
a full explanation of neural networks calculated by correlating the 2,048 features of one image with the buckets. Based
goes beyond the scope of this paper. In the 2,048 features of another image. If this correlation is higher on images contained
short, convolutional neural networks are than a user-defined threshold, the two images are placed in in multiple buckets,
the current state of the art in computer the same cluster. If other images also correlate higher than the Sankey diagram
vision. By making use of large collections this threshold, these images are also placed in the same shows a breakdown
of labeled training data, a neural net- cluster. This process is repeated until all images are placed in of each bucket, e.g.,
work is trained to discriminate between a cluster. upon close inspec-
26 • January-March 2021 ISASI Forum