A cost-effective random testing method for programs with non-numeric inputs

Arlinta C. Barus, Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Robert Merkel, Gregg Rothermel

    Research output: Contribution to journalArticleResearchpeer-review

    Abstract

    Random testing (RT) has been widely used in the testing of various software and hardware systems. Adaptive random testing (ART) is a family of random testing techniques that aim to enhance the failure-detection effectiveness of RT by spreading random test cases evenly throughout the input domain. ART has been empirically shown to be effective on software with numeric inputs. However, there are two aspects of ART that need to be addressed to render its adoption more widespread - applicability to programs with non-numeric inputs, and the high computation overhead of many ART algorithms. We present a linear-order ART algorithm for software with non-numeric inputs. The key requirement for using ART with non-numeric inputs is an appropriate "distance" measure. We use the concepts of categories and choices from category-partition testing to formulate such a measure. We investigate the failure-detection effectiveness of our technique by performing an empirical study on 14 object programs, using two standard metrics - F-measure and P-measure. Our ART algorithm statistically significantly outperforms RT on 10 of the 14 programs studied, and exhibits performance similar to RT on three of the four remaining programs. The selection overhead of our ART algorithm is close to that of RT.
    Original languageEnglish
    Article number7442567
    Pages (from-to)3509-3523
    Number of pages15
    JournalIEEE Transactions on Computers
    Volume65
    Issue number12
    DOIs
    Publication statusPublished - 1 Dec 2016

    Keywords

    • Random testing
    • Adaptive random testing
    • Category-partition method

    Cite this

    Barus, A. C., Chen, T. Y., Kuo, F-C., Liu, H., Merkel, R., & Rothermel, G. (2016). A cost-effective random testing method for programs with non-numeric inputs. IEEE Transactions on Computers, 65(12), 3509-3523. [7442567]. https://doi.org/10.1109/TC.2016.2547380
    Barus, Arlinta C. ; Chen, Tsong Yueh ; Kuo, Fei-Ching ; Liu, Huai ; Merkel, Robert ; Rothermel, Gregg. / A cost-effective random testing method for programs with non-numeric inputs. In: IEEE Transactions on Computers. 2016 ; Vol. 65, No. 12. pp. 3509-3523.
    @article{396d05ce74ac4be5b1ab2198a131dd30,
    title = "A cost-effective random testing method for programs with non-numeric inputs",
    abstract = "Random testing (RT) has been widely used in the testing of various software and hardware systems. Adaptive random testing (ART) is a family of random testing techniques that aim to enhance the failure-detection effectiveness of RT by spreading random test cases evenly throughout the input domain. ART has been empirically shown to be effective on software with numeric inputs. However, there are two aspects of ART that need to be addressed to render its adoption more widespread - applicability to programs with non-numeric inputs, and the high computation overhead of many ART algorithms. We present a linear-order ART algorithm for software with non-numeric inputs. The key requirement for using ART with non-numeric inputs is an appropriate {"}distance{"} measure. We use the concepts of categories and choices from category-partition testing to formulate such a measure. We investigate the failure-detection effectiveness of our technique by performing an empirical study on 14 object programs, using two standard metrics - F-measure and P-measure. Our ART algorithm statistically significantly outperforms RT on 10 of the 14 programs studied, and exhibits performance similar to RT on three of the four remaining programs. The selection overhead of our ART algorithm is close to that of RT.",
    keywords = "Random testing, Adaptive random testing, Category-partition method",
    author = "Barus, {Arlinta C.} and Chen, {Tsong Yueh} and Fei-Ching Kuo and Huai Liu and Robert Merkel and Gregg Rothermel",
    year = "2016",
    month = "12",
    day = "1",
    doi = "10.1109/TC.2016.2547380",
    language = "English",
    volume = "65",
    pages = "3509--3523",
    journal = "IEEE Transactions on Computers",
    issn = "0018-9340",
    publisher = "IEEE, Institute of Electrical and Electronics Engineers",
    number = "12",

    }

    Barus, AC, Chen, TY, Kuo, F-C, Liu, H, Merkel, R & Rothermel, G 2016, 'A cost-effective random testing method for programs with non-numeric inputs', IEEE Transactions on Computers, vol. 65, no. 12, 7442567, pp. 3509-3523. https://doi.org/10.1109/TC.2016.2547380

    A cost-effective random testing method for programs with non-numeric inputs. / Barus, Arlinta C.; Chen, Tsong Yueh; Kuo, Fei-Ching; Liu, Huai; Merkel, Robert; Rothermel, Gregg.

    In: IEEE Transactions on Computers, Vol. 65, No. 12, 7442567, 01.12.2016, p. 3509-3523.

    Research output: Contribution to journalArticleResearchpeer-review

    TY - JOUR

    T1 - A cost-effective random testing method for programs with non-numeric inputs

    AU - Barus, Arlinta C.

    AU - Chen, Tsong Yueh

    AU - Kuo, Fei-Ching

    AU - Liu, Huai

    AU - Merkel, Robert

    AU - Rothermel, Gregg

    PY - 2016/12/1

    Y1 - 2016/12/1

    N2 - Random testing (RT) has been widely used in the testing of various software and hardware systems. Adaptive random testing (ART) is a family of random testing techniques that aim to enhance the failure-detection effectiveness of RT by spreading random test cases evenly throughout the input domain. ART has been empirically shown to be effective on software with numeric inputs. However, there are two aspects of ART that need to be addressed to render its adoption more widespread - applicability to programs with non-numeric inputs, and the high computation overhead of many ART algorithms. We present a linear-order ART algorithm for software with non-numeric inputs. The key requirement for using ART with non-numeric inputs is an appropriate "distance" measure. We use the concepts of categories and choices from category-partition testing to formulate such a measure. We investigate the failure-detection effectiveness of our technique by performing an empirical study on 14 object programs, using two standard metrics - F-measure and P-measure. Our ART algorithm statistically significantly outperforms RT on 10 of the 14 programs studied, and exhibits performance similar to RT on three of the four remaining programs. The selection overhead of our ART algorithm is close to that of RT.

    AB - Random testing (RT) has been widely used in the testing of various software and hardware systems. Adaptive random testing (ART) is a family of random testing techniques that aim to enhance the failure-detection effectiveness of RT by spreading random test cases evenly throughout the input domain. ART has been empirically shown to be effective on software with numeric inputs. However, there are two aspects of ART that need to be addressed to render its adoption more widespread - applicability to programs with non-numeric inputs, and the high computation overhead of many ART algorithms. We present a linear-order ART algorithm for software with non-numeric inputs. The key requirement for using ART with non-numeric inputs is an appropriate "distance" measure. We use the concepts of categories and choices from category-partition testing to formulate such a measure. We investigate the failure-detection effectiveness of our technique by performing an empirical study on 14 object programs, using two standard metrics - F-measure and P-measure. Our ART algorithm statistically significantly outperforms RT on 10 of the 14 programs studied, and exhibits performance similar to RT on three of the four remaining programs. The selection overhead of our ART algorithm is close to that of RT.

    KW - Random testing

    KW - Adaptive random testing

    KW - Category-partition method

    UR - http://www.scopus.com/inward/record.url?scp=84998880895&partnerID=8YFLogxK

    U2 - 10.1109/TC.2016.2547380

    DO - 10.1109/TC.2016.2547380

    M3 - Article

    VL - 65

    SP - 3509

    EP - 3523

    JO - IEEE Transactions on Computers

    JF - IEEE Transactions on Computers

    SN - 0018-9340

    IS - 12

    M1 - 7442567

    ER -