Threshy: supporting safe usage of intelligent web services

Alex Cummaudo, Scott Barnett, Rajesh Vasa, John Grundy

Research output: Chapter in Book/Report/Conference proceedingConference PaperOtherpeer-review

1 Citation (Scopus)

Abstract

Increased popularity of 'intelligent' web services provides end-users with machine-learnt functionality at little effort to developers. However, these services require a decision threshold to be set which is dependent on problem-specific data. Developers lack a systematic approach for evaluating intelligent services and existing evaluation tools are predominantly targeted at data scientists for pre-development evaluation. This paper presents a workflow and supporting tool, Threshy, to help software developers select a decision threshold suited to their problem domain. Unlike existing tools, Threshy is designed to operate in multiple workflows including pre-development, pre-release, and support. Threshy is designed for tuning the confidence scores returned by intelligent web services and does not deal with hyper-parameter optimisation used in ML models. Additionally, it considers the financial impacts of false positives. Threshold configuration files exported by Threshy can be integrated into client applications and monitoring infrastructure. Demo: <a>https://bit.ly/2YKeYhE</a>.

Original languageEnglish
Title of host publicationESEC/FSE'20 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
EditorsPrem Devanbu, Myra Cohen, Thomas Zimmermann
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages1645-1649
Number of pages5
ISBN (Electronic)9781450370431
DOIs
Publication statusPublished - 2020
EventJoint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2020 - Virtual, United States of America
Duration: 8 Nov 202013 Nov 2020
Conference number: 28th
https://dl.acm.org/doi/proceedings/10.1145/3368089 (Proceedings)
https://2020.esec-fse.org (Website)

Conference

ConferenceJoint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2020
Abbreviated titleESEC/FSE 2020
Country/TerritoryUnited States of America
CityVirtual
Period8/11/2013/11/20
Internet address

Keywords

  • Decision theory
  • Intelligent services
  • Thresholding
  • Tooling

Cite this