Thoughts on “Hybrid Verification: Mixing Formal Methods and Testing”

This article, by Dr. Ben Brosgol, focuses on a mixture of formal methods and testing practices (together called “hybrid verification”) and the use of “contracts” that consist of preconditions and postconditions in order to formalize the assumptions made by critical code.

I chose to write about this article because it highlights some of the limits of testing, shows how to provide additional security for critical code, and introduces contact-based programming.

Brosgol defines a “contract” in programming as a set of preconditions and postconditions that wrap a subprogram (function, method, etc.) call.  The subprogram cannot begin unless its preconditions are met, and cannot return until its posconditions are true.  This provides a contract between a program and its subprograms that guarantees a certain state at critical times.  There are tools written for some languages (he uses SPARK as an example) that can do both static and dynamic contract testing and provide proof that the code will work as specified.

Brosgol then details ways to mix testing and formal verification.  If formally verified code calls subprograms that were tested rather than proven, the formal analysis tools will attempt to show that the preconditions are met, and assume that the tested code satisfies the postconditions.  This also requires that either the contracts are checked at runtime, or that sufficient testing was done such that the developer is confident the contracts will be fulfilled by the tested code.  If formally verified code is called from tested code, there need to be runtime checks for the preconditions (because the tested code does not guarantee those in the way the formal verification requires), but because the postconditions have been proven there is no need for checks at that point.

Next, Brosgol mentions the need for good choice of postconditions.  Strong, extensive postconditions make it easier to provide proof, but may have unacceptable overhead if they need to be checked dynamically.

He concludes that the relatively new combination of formal proof tools and contract verification on both a static and a dynamic basis opens up new avenues to create code that secures its critical sections.

This article helped me to understand that there’s a much wider world of testing beyond what we covered in class.  We didn’t talk about proof-based testing at all, and that’s a subject area that I believe I should learn more about.  It also highlights the way that our understanding of testing is ever-expanding.


Posted in Uncategorized | Tagged , , | Leave a comment

Thoughts on “Getting Started with AI for Testing”

In my last post, I wrote about an article that dove into the uses of AI in software testing.  Given the volume of search engine results that turned up when I started doing some research into the subject area, I thought it was worthwhile to write another piece about it.

The post I chose to write about this time is an introduction to AIST – Artificial Intelligence for Software Testing.  It is defined by Tariq King (the author of the post) as “an emerging field aimed at the development of AI systems to test software, methods to test AI systems, and ultimately designing software that is capable of self-testing and self-healing.”  Most intrigueing to me is the last part — self-healing software.

The organization hosting this blog (of which King is a founding member) is called AISTA, or the Artificial Intelligence for Software Testing Association.  Their mission is to pursue what they call the “Grand Dream” of testing: software that tests and updates itself with little need for human intervention.

King’s post is more of a survey than an in-depth piece.  He identifies three areas to explore when looking to get into AIST: artificial intelligence, software testing, and self-managing systems.  I know a little about the first two, but the third I haven’t touched on much.  Self-managing systems also appear to be the focus of AISTA.  King claims that there is “a general lack of research in the area of self-testable autonomic software”, but that recent technological developments appear to bring solutions closer practicality.

Ultimately, self-managing and self-healing systems are designed to adapt to their environment, modeled (originally by IBM) after the autonomatic nervous system in living creatures.  A self-healing system should be able to maintain homeostasis alongside self-optimization.  And that necessitates self-testing: before making changes to its own code, an autonomous system needs to ensure the change won’t do more harm than good.

So, what does a world of self-testing software mean for software testers?  It means that we may become more like teachers for software systems, moving them out of local pitfalls so that they can continue to grow.  Of course, these systems may be a long way off, and will need extensive human-driven testing and validation before they can start to test themselves.

The robots aren’t coming to take software testing jobs.  Yet.

Posted in Uncategorized | Tagged , , | Leave a comment

Thoughts on “AI Test Automation: The AI Test Bots Are Coming”

In this article, Greg Sypolt talks in brief about the role of AI as a software testing assistant.  I chose this piece because it combines a field I’m interested in (AI and machine learning) with the content of my software testing course.  I am interested in AI task automation already, so a piece that dovetails these two topics is a perfect fit.  The author has chops as well — he oversaw the conversion from manual to automated testing at his company, and offers training to help other teams transition to automation.

Sypolt starts off by outlining three uses of AI in testing:

  1. Automatic test case generation that reduces level of effort (he abbreviates as “LOE”) while maintaining consistent standards.
  2. Generating test code or pseudocode from user stories; a user story is a use case or use sequence for some kind of product (software or hardware).
  3. Codeless test automation, or testing without writing tests as code.

He then outlines the necessity of properly training the testing bots, and some of the difficulties that may be involved:

  1. Identifying the proper training algorithms.
  2. Collecting a massive quantity of useful data.
  3. Ensuring that bots behave in a reasonable fashion from a given set of inputs, and ensuring that they exhibit new behavior (generate new tests) when the inputs change.
  4. The training process never really ends; each new set of tests and results gives the bots another data point to learn from.

I firmly believe that we are at the very start of a revolution in machine learning.  Why not apply those principles to software testing?  There are, of course, a couple of issues that arise which Sypolt didn’t mention: quality of tests and accuracy of tests.

A point firmly pushed by other articles and texts I’ve read is that quality is more important than quantity.  There can be little difference between running ten tests and running one hundred tests if the extra ninety don’t teach us much of anything.  The trick isn’t to create an AI that merrily runs thousands on thousands of unit tests; it’s to create on that identifies the important tests which reveal faults we don’t know about and confine itself to executing exactly those.

It’s also very important to ensure that the AI has learned properly and is up to standards — and that means testing the AI.  Which means testing the AI that tests the AI, and it’s digital turtles all the way down.

I can take away from this article two things: Firstly, it’s reasonable to combine two fields that I’m interested in (AI and testing) and that resources exist or will exist to support the two together.  Secondly, the field of testing is constantly and rapidly changing.  Additional learning is crucial, just like AI systems continue to learn from each new piece of data.

Article link

Posted in Uncategorized | Tagged , , | Leave a comment

Thoughts on “Six Things That Go Wrong With Discussions About Testing”

In this article, James Bach talks about the ways in which conversations about testing skew away from the reality of what it means to actually do software testing, and what it means to be a skilled tester.  As is clear from the title, he divides this broad topic into six smaller pieces:

  1. Size doesn’t matter.  The number of test cases that you run is not a meaningful number, just like the number of lines of code in a project is not meaningful.  What matters is what they cover and what they can teach us.
  2. Tests are performances.  It’s an activity, not an object.  The person implementing test cases is more important than the cases themselves.
  3. Testing strategy needs to evolve.  Testing is a process of constant interrogation (I’ve written about this before in response to other articles) of both the code and the tester.
  4. Automation does not define testing.  Automation is a tool with which to run tests.  It takes human judgment and skill to design and implement quality tests.  Automation is a way to do a lot, but it takes a tester’s skills to do a lot with as little as possible.
  5. There are many kinds of test coverage.  No one type of coverage is truly comprehensive, and making changes to tests to give additional coverage by one metric may change what is covered by another metric.
  6. Testing is not static.  It’s an activity that’s fundamentally about learning.  Some things are predictable, but many things are not.  Testing is essetially formulating and then running experiments.  Much can be gained by deviation from established procedures.

The biggest takeaway for me from this article is the notion of testing as design and execution of experiments.  It’s not something I’d ever really thought about before, and it makes a lot of sense.  Testing is the process of formulating a hypothesis (these inputs in this context should result in this output) and then trying it out to see whether it’s right.  It even involves fairly thorough hypothesis testing, so that we can claim with confidence that a particular outcome of use is guaranteed (or close enough).

This article is also a discussion of the ways in which one can have a useful conversation about testing, and the ways in which those conversations can turn useless.  I think this is important to take away too, when I’m looking for jobs in the software development field.  I want to avoid making conversational mistakes in interviews by going down the wrong path, and I also may want to avoid working at a company where those conversational mistakes are part of the corporate culture.

article link

Posted in Uncategorized | Tagged , , | Leave a comment

Thoughts on “Top Ten Factors to Consider When Choosing a Testing Technique”.

In this article, Manoj Verma describes (in his own words) “an exhaustive list of factors to consider” in the choice of testing technique.

But let’s back up a bit.

I chose this article to write about (and this topic) because in our work last week we were tasked with testing a small play problem; and shortly after starting I realized that for all of my practice with different testing methods, I didn’t know where to begin.  So when I went to write my weekly blog post, that’s where I looked.

Verma’s list of factors is as follows:

  1. Risk assessment — how tolerant to failure is the product?
  2. Client requirements — did the client specify a testing technique?
  3. Time/budget constraints — how long do we have to test?  How much money can we spend on testing?
  4. Industry guidelines — does the product need to fit into some kind of regulation? Which ones?
  5. Documentation — does documentation for the product exist?  Does it contain testing history?  Does it contain logical constructions such as decision tables or state graphs?
  6. Objective of test — what are we testing for?  What does the product need to do, or need to never do?
  7. Software development lifecycle — how is the development process managed?
  8. Models used to develop the system — how was the software system built?  Are there logical models that can be adapted into testing techniques or cases?
  9. Tester experience — how experienced is the tester?  How accurate is their assessment likely to be?
  10. Flexibility — does the devlopment process model involve a lot of flexibility?

Verma also concludes that choosing the right technique “is not child’s play”.  It isn’t easy, and he doesn’t offer hard metrics, but his guidelines serve as a way for me to ask myself a series of questions and discover the method that may work the best.

My search for answers, as it turns out, did not give me any that are truly satisfying.  It led me to more questions.  Software testing in general can be seen as the act of asking questions of both the design and implementation.  It seems natural that software testing techniques and methods should also be subject to questioning; in fact, software testers should subject themselves to similar questions.

So what can I take away from this?  How can it help me move past the brand-new-problem paralysis?  That I should start by asking questions of the problem I’ve been presented with, and of myself and my own testing experience.

Article link.

Posted in Uncategorized | Tagged , , | Leave a comment

Thoughts on “TDD Guided by Zombies”

In “TDD Guided by Zombies”, James Grenning shares a seasonally-appropriate acronym for developing unit tests — just in time for our class to wrap up its unit testing section!  ZOMBIES stands for Zero, One, Many/More, Boundary behaviors, Interface definition, Exercise exceptional behavior, Simple scenarios/Simple solutions.  The first three letters (Zero, One, Many) describe, in order, the complexity of the behavior undergoing testing — for example, a queue with zero, one, and then many entries.  The second set of letters (Boundary, Interface, Exercise Exceptions) give the order in which to build test cases — start with boundaries, design the interface undergoing testing (or ensure that the code meets the interface requirements), and then test exceptional behavior.  The last letter tells the tester to create simple scenarios with simple solutions — add the minimal possible behaviors to production code in order to pass the tests generated by the first six parts of the acronym.

Grenning spends the bulk of the article on an example of a circular queue implementation in C++, showing how he progresses through each step.  He gives clear examples of both what to do and not to do.  However, I’m more concerned with the underlying principles and process, so in the interest of brevity I’ll skip a detailed review of his example and simply say that it’s thorough and worth a longer period of study.

I chose this article not just because of the season (or that it coincides neatly with our in-class work), but because it helps to answer the question of how.  We’ve learned many methods for generating unit test cases, but how do we pick the order?  How do we work backwards from tests to code in a way that makes sense and satisfies the specification that we’re handed?  This confirms what I learned in 343 last year: ZOMBIES are the key to TDD and to successful unit testing.

And what can I, as a learner, take away from this?  Firstly, and perhaps most trivially, it can help to wrap important concepts in cute acronyms.  It makes sharing and remembering knowledge easier.  Secondly, it clarifies the most important sets of values to test, and the most important times at which to test newly-instantiated objects (or other language-apprpriate constructs): when they are fresh and empty, when they have a single value, and when they contain many values.  Thirdly, the article really drives home the importance of minimalism in testing and test-driven coding.  If a simple solution is all that it takes to meet the specification, then use it.  If a complex solution can be simplified, simplify it.

This semester and onward, when I need to develop unit test cases, I’ll be thinking ZOMBIES.

Article link.

Posted in Uncategorized | Tagged , , | Leave a comment

Thoughts on “Deeper Testing (3): Testability”

In his article “Deeper Testing (3): Testability”, Michael Bolton defines the testability of a product not only in terms of how it can be manipulated (although that’s one of his categories), but as “a set of relationships between the product, the team, the tester, and the context in which the product is being developed and maintained”.  He breaks this into five somewhat overlapping regions:

  • Epistemic testability — how we, as testers and developers, find the “unknown unknowns” related to the product.
  • Value-related testability — understanding the goals of everyone with a stake in the product and what they intend to get out of it.
  • Intrinsic testability — designing a product to facilitate easier and more meaningful testing.  This includes the ability to manipulate and view the product and its environment.
  • Project-related testability — how the testing team is supported by and supports other teams or team members attached to the product.
  • Subjective testability — the skills of the tester or testers.

Bolton then details how he thinks a tester on an agile team should operate: by bringing up and advocating for the testability of the product, and trying to keep the growth rate of the product sustainable in relation to testing.

As I learn more in the course, I find it increasingly important to understand not just how we test, but why and in what context.  Bolton’s article is especially helpful in understanding the context, and that is why I chose to write about it.  The article highlights aspects of the environment that surrounds a product, and the ways that those aspects contribute to or detract from the feasability of testing.  It also speaks to a tester’s role in an agile team, which is in practical terms useful to know about as many companies use some form of agile development.

With any resource I find in relation to my coursework, I look at what I can take away and what I can apply in my life (sometimes not limited to software development or computer science).  This piece gives me a better understanding of how to begin testing a product — not by writing test cases, or even determining the optimal testing tools to use, but by looking at the bigger picture.  I need to ask myself what I don’t know about the product, how it is going to be used, how it can be designed (or was designed) to facilitate testing, how I as a tester can and should engage other team members, and what skills I have that can make testing easier or what skills I need to hone that I don’t already possess.

Article link. 

Posted in Uncategorized | Tagged , , | Leave a comment