Trust-Based Fusion of Untrustworthy Information in
Crowdsourcing Applications
Matteo Venanzi, Alex Rogers, Nicholas R. Jennings
University of Southampton
Southampton, UK
{mv1g10, acr, nrj}@ecs.soton.ac.uk
ABSTRACT
In this paper, we address the problem of fusing untrustworthy re-
ports provided from a crowd of observers, while simultaneously
learning the trustworthiness of individuals. To achieve this, we
construct a likelihood model of the users’s trustworthiness by scal-
ing the uncertainty of its multiple estimates with trustworthiness
parameters. We incorporate our trust model into a fusion method
that merges estimates based on the trust parameters and we provide
an inference algorithm that jointly computes the fused output and
the individual trustworthiness of the users based on the maximum
likelihood framework. We apply our algorithm to cell tower local-
isation using real-world data from the OpenSignal project and we
show that it outperforms the state-of-the-art methods in both accu-
racy, by up to 21%, and consistency, by up to 50% of its predictions.
Categories and Subject Descriptors
I.2.11 [
Artificial Intelligence
]: Distributed Artificial Intelligence—
Intelligent agents, multiagent systems
General Terms
Algorithms, Performance, Design, Theory
Keywords
Crowdsourcing, Information trustworthiness, Data fusion
1. INTRODUCTION
The practice of outsourcing tasks to the public, more generally
known as
crowdsourcing
, has recently shown enormous potential
in solving highly decentralised target localisation tasks [1]. In such
a setting, a
task requestor
wants to determine the undisclosed lo-
cation of a point-wise target through collecting multiple observa-
tions from a networks of observers, normally referred to as crowd.
Examples of this kind include the DARPA Red Balloon challenge
which aimed to find 10 balloons placed at hidden locations leverag-
ing social networks
1
, and the crowdsourcing of cell tower locations
to help improve the positioning systems of mobile phones (see Sec-
tion 5 for more details) In both of these cases, and many others
1
archive.darpa.mil
Appears in:
Proceedings of the 12th International Conference on
Autonomous Agents and Multiagent Systems (AAMAS 2013), Ito,
Jonker, Gini, and Shehory (eds.), May, 6–10, 2013, Saint Paul, Min-
nesota, USA.
Copyright
c
⃝
2012, International Foundation for Autonomous Agents and
Multiagent Systems (www.ifaamas.org). All rights reserved.
beside, a key benefit is the inexpensive decentralisation of a com-
plex information gathering process broken into micro–tasks and
outsourced to individuals (possibly for small monetary rewards).
However, a key challenge in these domains is how to deal with
the unknown reliability or
trustworthiness
of information reported
from the crowd. The reasons motivating this concern are many-
fold. First, crowd members have different levels of accuracy re-
lating to their individual skills and subjectivities as lay observers.
Second, some of the users are only interested in maximising the
reward from executing as many tasks as possible, thus exerting the
minimum effort in the single task and submitting low quality data.
For example, in the Red Balloons challenge, 66% of the balloon
sights received by the winning team proved to be erroneous [10]
and, in the crowdsourced cell tower maps, cell tower detections of-
ten report out-of-date GPS locations.
The unreliability of crowdsourced data presents challenges when
multiple reports of the same phenomenon must be
fused
together.
Recently, this has been addressed through the design of computa-
tional agents that seek to estimate the reliability of the reports and
also compute their aggregated output [8]. In particular, existing
research in machine learning and multi-agent systems has mainly
concentrated on the problem of fusing multiple single-value obser-
vations combined with the assessment of a user’s trustworthiness in
a number of crowdsourcing applications, including image labelling
[17], galaxy classification [8] and IQ testing [2]. In such applica-
tions, observations are typically values corresponding to the class
label or the answer to a question selected by the user. Then, mul-
tiple observations are fused together using simple majority voting
and machine learning approaches based on probabilistic graphical
models [17, 18]. For example, both Whitehill
et al.
and Raykar
et al.
use expectation-maximisation to infer the expertise of each
user and the most likely aggregated answer in a classification task
[18, 11]. In a similar vein, Welinder
et al.
consider user trustwor-
thiness in a multidimensional space and estimates the competence,
expertise and bias of each user through Bayesian inference in an
image labelling task [17]. However, in recent years, new applica-
tions based on the deployment of mobile technologies have pro-
vided a new perspective on this problem. To date, people using
their smart phones as an mobile computing platform with a number
of sensors, such as image/video sensor and GPS sensor, are now
able to report not just single-value observations but rather they can
report
estimates
that more comprehensively include numerical in-
formation about to the uncertainty in an observation. For example,
uncertainty values can be reported by the user as the confidence
level about an answer to a classification task or as the variance of
a series of multiple measurements. Specifically, when users report
geo-referred data, the precision of a single location is automatically
provided by the GPS device itself on the basis of the number and
geometry of the satellites being used to generate the fix.
2
Alter-
natively, in crowd-powered prediction markets, the amount people
pay for a particular contract represents their confidence level in the
corresponding outcome [6]. Given this, we focus on the problem
of fusing untrustworthy estimates which we believe is relevant for
a large class or crowdsourcing applications where reported uncer-
tainties are part of the collected data.
In terms of addressing this challenge, a vast literature in the re-
lated multi-sensor fusion domain studies how to integrate multiple
estimates into a single output and there are standard techniques for
fusing estimates when these relate to stationary targets, i.e. co-
variance intersection, (CI), and to a moving targets, i.e. the co-
variance union (CU). However, their limitations when applied to
our problem is that they typically merge estimates without mod-
elling the trustworthiness of the user or they apply simple outlier
detection methods to the reports, such as kNN [16], SOD [9] and
LOF [3], which identify unreliable estimates but fail to attribute
these to the untrustworthiness of the individual user who supplied
them. This stems from the assumption that the noise in the data is
only introduced by uncalibrated or faulty sensors. However, noise
models developed in sensor fusion are often unsuitable for deal-
ing with untrustworthy information in crowdsourcing settings [4].
First, the range of human errors cannot be entirely characterised by
the concept of noise assumed in traditional sensor fusion in which
sensor noise is typically captured with predefined sensor fault mod-
els. Second, it is unrealistic to think that sensors can deliberately
misreport observations in a human-like manner with a strategic be-
haviour. In this field, the work of Reece
et. al
that considers a
model of sensor trustworthiness to deal with sensors with unknown
fault types offers a solution that is more applicable to our problem.
In their model, the estimates are aggregated using a consensus rule
and each sensor’s trustworthiness is measured by the Mahalanobis
distance of the sensor measurement from the fused estimate, after
appropriately setting a threshold parameter
β
to characterise trust-
worthy estimates [13]. However, since such a model is natively
defined for the sensor fusion domain, it has not been applied to
crowdsourcing problems in previous work. As such, we will also
contribute to provide its evaluation in a crowdsourcing setting us-
ing it as a benchmark for our approach. In addition, more flexible
approaches can possibly derive measurements of trustworthiness
purely relying on the observed reports without requiring any pa-
rameter tuning.
Against this background we developed a new trust-based fusion
method that combines trust modelling in the fusion of untrustwor-
thy information. In particular, we model user trustworthiness as an
uncertainty scaling
parameter of the user’s estimates and we incor-
porate such parameters in the computation of the fused output. This
is similar to the Dempster-Shafer belief fusion [15] which, how-
ever, only works when the trust degrees of the beliefs are known
in advance, while our approach learns these from the data. Then,
we construct a likelihood model user’s trustworthiness based on the
joint product of the probability densities of the user’s estimates and
their fusion. Putting these together, we provide an algorithm, called
MaxTrust, to estimate the users’ trustworthiness and the fused out-
put from the reports gathered from the crowd. We show the effi-
cacy of MaxTrust in the real-world crowdsourcing application of
cell tower localisation using a dataset provided by the OpenSignal
project (
opensignal.com
). In particular, we show that our al-
gorithm outperforms a set of benchmarks in providing more accu-
rate and more informative predictions of cell tower locations. In
summary, the contribution of this paper to the state of the art is
2
See
developer.android.com
and
developer.apple.
com
for more details.
Figure 1: Illustration of the scenario for a crowdsourced appli-
cation where users report GPS location estimates of the target
using smartphones.
stated as follows:
•
We introduce a new trust-based fusion model for jointly ag-
gregating estimates of untrustworthy users and estimating the
trustworthiness of each user within the crowdsourcing do-
main.
•
We provide the MaxTrust algorithm to efficiently compute
the fusion of the reports and the trustworthiness levels of
each users based on the maximum likelihood framework.
•
We show that our algorithm outperforms the existing meth-
ods in both making more accurate, by up to 42%, and more
informative predictions, by up to 80%, in a cell tower locali-
sation task using real-world data.
The reminder of this paper is structured as follows. Section 2 for-
mally describes our model and Section 3 provides the model anal-
ysis for the two-dimensional case that is of practical interest for its
application of location data. Next, Section 4 presents the MaxTrust
algorithm for estimating the model’s parameters and Section 5 pro-
vides is evaluation on the OpenSignalMaps dataset. Section 6 con-
cludes.
2. MODEL DESCRIPTION
In this section, we formally describe our model of untrustworthy
estimates (Section 2.1). Then, we detail the procedure for comput-
ing the fusion of the reports (Section 2.2) and estimating the user’s
trustworthiness (Section 2.3).
2.1 Modelling Untrustworthy Estimates
In this model, a crowd of
k
users
U
=
{
1
,
· · ·
k
}
observe an invari-
ant and unknown target feature
x
0
,
∈ R
n
(or simply target) defined
in an
n
dimensional space. Each user
i
reports
p
i
estimates of the
target, where each estimate
r
i;j
comprises the following values: (i)
the
measured value
x
i;j
∈ R
n
and (ii) an estimate of the
preci-
sion
of the user’s observation:
θ
i;j
∈ R
>
0
. In particular,
θ
i;j
is
the reported uncertainty that may be referring to the user’s confi-
dence level about its reported value, the precision of the measuring
tool, or the variance of some repeated measurements. Thus, the
report set is
R
=
{
r
i;j
|
i
= 1
, . . . , n
;
j
= 1
. . . p
i
}
and includes
p
=
∑
k
i
=1
p
i
reports where each report
r
i;j
=
⟨
x
i;j
, θ
i;j
⟩
denotes
that user
i
estimates
x
0
as
x
i;j
with precision
θ
i;j
. For example,
Figure 1 illustrates a typical scenario described by our model in
which users observe a specific target (e.g. a “red balloon” inspired
by the DARPA red balloon challenge) and report their observations.