Vai al contenuto

CONFRONT – Challenge ON wifi FRame fingerprinting for people cOunting aNd Tracking

    Results

    We have finally released the results of the challenge CONFRONT – “Challenge ON wifi FRame fingerprinting for people cOunting aNd Tracking”.

    Seven international research groups participated by applying their own algorithm to our dataset.

    We are very happy now to share the news that the winner is Tomás Santos (Iscte – Instituto Universitário de Lisboa, Portugal). The second place was achieved by Aleš Simončič (Jožef Stefan Institute, Slovenia).

    We thank all the participants for competing in our challenge:

    • Abhishek Mishra (Privatics / Inria Lyon, France)
    • Aleš Simončič (Jožef Stefan Institute, Slovenia)
    • Diego Gasco (Polytechnic of Turin, Italy)
    • Giuseppe Perrone (Polytechnic of Turin, Italy)
    • Tomás Santos (Iscte – Instituto Universitário de Lisboa, Portugal)
    • 2 members of an IT company in Germany

    The results will be presented during the 2024 EuCNC & 6G Summit in Antwerp (Belgium) at the beginning of June. This activity has been funded by MOST.

    Background

    The widespread use of personal mobile devices, including tablets and smartphones, created new opportunities for collecting comprehensive data on individual movements within cities while preserving their anonymity. Extensive research focused on turning personal mobile devices into tools for measuring human presence. To protect privacy, the data collected must be anonymous or pseudo-anonymous, leading to the preference for management data.

    A common approach involves analysing probe requests, which are Wi-Fi protocol messages transmitted by mobile devices while searching for access points. These messages contain media access control (MAC) addresses, which used to be unique identifiers. To safeguard the privacy of smartphone users, Google, Apple, and Microsoft have implemented algorithms that generate random MAC addresses, which change often and unpredictably.

    This challenge focuses on the problem of fingerprinting Wi-Fi devices by analysing management messages to overcome previous methods that relied on the MAC address and became obsolete. Detecting messages from the same source allows counting the devices in an area, calculating their permanence, and approximating these metrics with the ones of the humans carrying them.

    Challenge

    The Challenge consists of three tasks:

    • Task A: you must use a file obtained by merging individual device captures (capture_A.pcap). We will tell you the number of devices but not which are the sources of the individual probes. The output required from you is a CSV file with two columns: the IDs of the samples and the labels. If you wish to discard some samples you have to use -1 as label.
    • Task B: you must use a file obtained by merging individual captures (capture_B.pcap).  You won’t know the number of devices neither the sources of the individual probes. The output you will give us is a CSV file with two columns: the IDs of the samples and labels.  If you wish to discard some samples you have to use -1 as label.
    • Task C: you must use a file obtained by sniffing a group of devices inside the anechoic chamber (capture_C.pcap). The output you will give us is the number of devices and discarded samples.

    Registration

    To participate in the challenge, you must register by 20th January 2024 via the following link.

    Groups or individuals may participate.

    Timeline

    • 2024 January 20th – registration
    • 2024 January 31st – transmission of the dataset
    • 2024 March 15th – deadline for the transmission of obtained results
    • 2024 March 20th – ranking publication
    • 2024 March 29th – paper submission
    • 2024 April 8th – paper-acceptance notification
    • 2024 April 12th – deadline for final version of the papers

    Dataset

    The dataset provided for the challenge will consist of 3 files containing Wi-Fi probe requests. It will only be sent to those who register by the deadline. However, you can download two complete examples of captures in pcap format and the requested output in the following link.

    The probe requests in the example files will differ from those in the challenge tasks since they will be collected at different times and with different devices.

    Evaluation

    Every task will be evaluated with a score between 0 and 100. The score in Tasks 1 and 2 will be calculated as the v-measure multiplied by the percentage of discarded samples. The score in Task 3 will be calculated as the percentage error multiplied by the percentage of discarded samples. The total score will be the sum of the task scores.

    scoretot = score1 + score2 + score3

    score1 = vmeasure x discarded%

    score2 = vmeasure x discarded%

    score3 = error% x discarded%

    The vmeasure is calculated as follows:

    vmeasure = ((1 + beta) x homogeneity x completeness ) / ((beta x homogeneity) + completeness)

    Where beta is a ratio of weight attributed to homogeneity versus completeness and is set to 1. The homogeneity is a metric to evaluate how clusters calculated by an algorithm contain only samples which are members of a single class. This metric is independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score value in any way. Homogeneity is not symmetric: switching the true labels with the predicted labels will return the completeness score which might be different in general. More details about the vmeasure can be found in “V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure” by Rosenberg and Hirschberg (EMNLP-CoNLL 2007).

    The discaded percentage is calculated as the ratio between the number of samples and the number of discarded samples multiplied by 100. The percentage error is calculated as follows:

    error% = (|devicesestimated – devicesreal|) x 100 / devicesreal

    The scores will be published on this website.  All teams will be invited by email to write a paper containing the methodologies used to achieve that score.

    Conference

    We are in the process of organizing a special session at the EuCNC & 6G Summit 2024 conference. All teams will have the opportunity to publish a paper with the methodology used to achieve their results in this special session. These papers will be reviewed by our committee. Accepted papers will be part of the Conference Proceedings, but they will not be submitted to IEEE Xplore.

    Organisers

    Luigi Atzori is a Full Professor at the Department of Electrical and Electronic Engineering at the University of Cagliari (Italy), where he leads the activities of the Networks for Humans laboratory. His research interests are in multimedia communications and computer networking, with emphasis on multimedia QoE, multimedia streaming, NGN service management, service management in wireless sensor networks, architecture and services in the Internet of Things. Luigi Atzori has been the coordinator of the Marie Curie Initial Training Network on QoE for multimedia services (2015-2018), involving ten European Institutions in Europe and one in South Korea. He has been the associate editor for major journals, including ACM/Springer Wireless Networks, IEEE IoT Journal, Elsevier Ad Hoc Networks, and Advances on Multimedia. He has been the guest editor for the IEEE Communications Magazine, the Springer Monet Journal, and the Elsevier Signal Processing: Image Communications Journal. He is currently the associate editor of the Digital Communications and Networks and the IEEE Open Journal of the Communications Society.

    Lucia Pintor has been a researcher at the Italian National Inter-University Consortium for Telecommunications (CNIT – Consorzio Nazionale Interuniversitario per le Telecomunicazioni) since November 2023. She is also a PhD student at the University of Cagliari, where she also worked as a researcher from 2018. Her research concerns mobility as a service (MaaS) and methods to anonymously track the flows of people moving in a Smart City. In the last years, she specialised in the fingerprinting of Wi-Fi devices through the analysis of management messages with the aim of tracking mobile devices. She also has experience in modeling and developing web platforms, APIs, and user interfaces.

    Contacts

    lucia.pintor[at]unica.it

    Acknowledgements

    This challenge is partially supported by the European Union under the Italian National Recovery and Resilience Plan (NRRP) of NextGenerationEU, “Sustainable Mobility Center” (Centro Nazionale per la Mobilità Sostenibile), CNMS, CN 00000023.