Model-based Whitebox Fuzzing for program binaries

Van-Thuan Pham, Marcel Böhme, Abhik Roychoudhury

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

31 Citations (Scopus)


Many real-world programs take highly structured and complex files as inputs. The automated testing of such programs is non-trivial. If the test does not adhere to a specific file format, the program returns a parser error. For symbolic execution-based whitebox fuzzing the corresponding error handling code becomes a significant time sink. Too much time is spent in the parser exploring too many paths leading to trivial parser errors. Naturally, the time is better spent exploring the functional part of the program where failure with valid input exposes deep and real bugs in the program. In this paper, we suggest to leverage information about the file format and data chunks of existing, valid files to swiftly carry the exploration beyond the parser code. We call our approach Modelbased Whitebox Fuzzing (MoWF) because the file format input model of blackbox fuzzers can be exploited as a constraint on the vast input space to rule out most invalid inputs during path exploration in symbolic execution. We evaluate on 13 vulnerabilities in 8 large program binaries with 6 separate file formats and found that MoWF exposes all vulnerabilities while both, traditional whitebox fuzzing and model-based blackbox fuzzing, expose only less than half, respectively. Our experiments also demonstrate that MoWF exposes 70% vulnerabilities without any seed inputs.

Original languageEnglish
Title of host publicationProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering
EditorsDavid Lo, Sven Apel, Sarfraz Khurshid
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages11
ISBN (Electronic)9781450338455
Publication statusPublished - 2016
EventAutomated Software Engineering Conference 2016 - Singapore Management University (SMU), Singapore, Singapore
Duration: 3 Sep 20167 Sep 2016
Conference number: 31st (Conference website) (Proceedings)


ConferenceAutomated Software Engineering Conference 2016
Abbreviated titleASE 2016
Internet address


  • Program Binaries
  • Symbolic Execution

Cite this