API Test is delivered as a service, hosted and managed by IBM® “in the cloud". It supports multiple users and is currently considered Alpha-level software. IBM is opening up access to API Test through this early program to share ideas and direction and to solicit feedback to help shape the final product.
Examples are in brackets.
Application (crm) 1___N Service (phonebook, inventory)
Service (phonebook) 1___N Operations (create, update, query etc)
Operation (create) 1___N Outcome (OK, Not OKE, not authorised etc)
It's possible to have full coverage without any confidence.
Figure: Coverage in API Test (green says there is a test associated, not that the test has passed!).
API Coverage is a metric that concerns the existence of Tests that exercise Operations and validate for specific Outcomes. An Outcome is said to be “covered” if at least one Test exists that invokes the containing Operation and Validates for that Outcome.
Confidence is a metric that helps give a numeric, quantified value to the confidence level of a System as a whole, helping to answer the question, “Is this software ready to be promoted to the next stage in the development process?” Our confidence in a service is informed by the presence and the status of tests(*1). Confidence is also specific to a particular environment for a SUT; tests have to be executed in that environment against specific versions of the services in that environment for them to be considered for the Confidence metric.
In API Test, an Importance value is a statement of relative importance for a Service, Operation, or Outcome. Importance factors into the Confidence metric. By raising or lowering the Importance, you raise or lower the Confidence level.
Table: Coverage, Confidence and Importance (Source dW)
*1 - Confidence is based on executed tests. More on this in Part 2
Questions related above table:
Does a non executed test influence the coverage?
Confedence based on the available services or the number of tests.
Duplicate test not run has no effect on Confidence.
Duplicate test which has outcome does has effect on Confidence.