Overcoming Test Coverage Challenges in API Testing

What is an API?

API stands for Application Programming Interface.

API is an interface that is used by the programmers to make communication from your system’s application to other applications/third-party servers. The developers connect to the APIs to make requests and get responses from other applications as required.

APIs in Action:

How is it possible to make an online flight booking from your credit card? Your bank must have provided the APIs for an online transaction that is internally called by the flight booking website. The flight booking websites make REQUEST and receive RESPONSE; based on the response (successful/failure of the transaction) the ticket gets booked.

If your bank’s APIs does not work properly, the transaction becomes unreliable and can cause potential loss of business. Hence, APIs used in such cases should be tested thoroughly before rolling out.

API Testing:

Testing an API is no different than testing for anything else. You still follow the same steps of the test method, but some of them have a different flavour.

The GUIs would have items like menus, buttons, checkboxes, and combo lists that would trigger the event or would require an action to be taken. Similarly, the APIs have input parameters as request and output as a response; both need to be tested.

Each API carries its own set of input parameters/request and based on the combination of input parameters/request we send, the API responds accordingly.

Test Coverage:

While testing APIs, input/output schema (request/response) coverage becomes important. We still probably care about all the same coverage used to do with non-API testing (scenario coverage, requirements coverage, and application data coverage), but in API testing schema coverage comes into the picture.

Test coverage is simple when the API carries fewer parameters. For example, consider an API named MULTIPLY with 2 parameters.

API Name	MULTIPLY
Parameters	2 Mandatory Parameters: int A and int B
Parameter Selection	A, B	Input Values: 10, 10
	A, B	Input Values: -6, 10
	A, B	Input Values: 6, -5
	A, B	Input Values: 2.8, 10
	A, B	Input Values: -8.589, 5.789

These are relatively simple to test since the input can be defined with various value pairs and results can be validated against the expected return value.

The challenges come around when we deal with APIs which got heavy request/response with more number of parameters, especially the optional parameters.

Challenges in Test Coverage:

Inadequate test coverage might cause huge problems. The quality of the product will suffer and the product will become bug prone. There are quite a few challenges that are ecountered to ensure enough test coverage in the API based applications. Following is the list:

Schema Updates
Parameter Combination
API Call Sequencing
Request data combination

Schema Updates:

The input/output schema (request/response) skeleton may vary from time to time due to the business requirements. The coverage should be extended for that. For example, If a new parameter getting added to the request, the coverage has to increase with all the request combinations using the new parameter and possible data combination.

Parameter Combination:

Each API will take various input parameters and there will be a huge number of combinations of input params. Verification on missing mandatory parameters is quite simple. The combination will get increased with more optional parameters. There might be Boolean input params which may carry value 1 or 0 and each of the combinations among these parameters might make difference in the business case and the response which is to be covered and tested.

API Call Sequencing:

Apart from testing each API, the sequence of APIs to be called also a challenge and may end up with inadequate coverage. For example, consider the sample shopping cart APIs as below:

create_user
update_user
update_shipping_address
update_order
add_item_to_bag
update_bag_items
delete_bag_item
merge_bag_items
check_out

Apart from testing each API, there can various sequences of flow can be made out of it. The output of an API can be used as an input of another API. All the sequence flow should be covered to deliver bug-free applications.

Request data combination:

Each parameter might be designed to carry a specific value and it should be validated. It can be a number range, max length restrictions, acceptance of a specific string/value. These kinds of parameters might get less attention while test case preparation and ends with a lack of coverage. It will become problematic from an end-user perspective.

Overcoming the Challenges:

Whenever there is a change in the request/response of the API, the related test cases should be revisited for modification/including more coverage. The change might in addition/removal of few parameters, request body change, response body change.

For example, let us take the MULTIPLY API got an optional parameter int C:

You can find in the below table the parameter combination has been increased as there is a new parameter got introduced in the API.

API Name	MULTIPLY
Parameters	2 Mandatory Parameters: int A and int B	1 Optional Parameter: int C
Parameter Selection	A, B	Input Values: 10, 10
Parameter Selection	A, B	Input Values: 10, 10
	A, B	Input Values: 6, -5
	A, B	Input Values: 2.8, 10
	A, B	Input Values: -8.589, 5.789
	A, C	Values: 10, 5 Should get a mandatory parameter missing error
	B, C	Values: 10, 5 Should get a mandatory parameter missing error
	A, B, C	Values: 4.5, 6, -7.8
	...

Parameters:

In the above example, we increased the coverage for 1 additional parameter. What if there are 50 parameters getting added to the API? We should come up with test coverage that includes parameters with all the permutation of combinations so that it will help us to discover all the failures. We can use any tool such as “All Pairs” to derive all the combinations.

The input parameters should be tested with the valid and invalid values using strategies like boundary value analysis and equivalence partitioning. Shuffle the parameters with all possible combinations to find the failures.

Grasp the purpose of the APIs, the business need in which they are to be used. How the wrapper UI going to pull the data from the APIs? How it will be real-time scenarios? Keep all such things in mind. Once all these parameter selections and combinations are designed, different call sequences need to be explored.

1.fromStation*
2.toStation*
3.classType*
4.trainType (default type express)
5.quotaCode (default quota General)
6.seniorCitizen (default as No)
7.gender (default as Male)

* are the mandatory parameters.

The fare is going to be calculated differently on each parameter combination you choose. Hence it is important to cover all the possible combinations.

Case ID		Mandatory Parameters				Optional Parameters
	fromStation	toStation	classType	trainType	quoteCode	seniorCitizen	gender
TC-1	x	x	x	-	-	-	-
TC-2	x	x	-	-	-	-	-
TC-3	x	-	x	-	-	-	-
TC-4	-	x	x	-	-	-	-
TC-5	x	x	x	x	-	-	-
TC-6	x	x	x	x	-	x	x
TC-7	x	x	x	x	x	x	x

Refer to the above table. We added the test cases for verification of missing mandatory parameter (TC-2,3,4) and still, we can add more combos.
The test cases for various optional parameter combinations are added in TC-4,5,6. In this way, we can track the coverage for all the combinations of the parameters. We can design a separate table for the parameters for which boundary values need to be analyzed.