The financial services industry has always been at the fore-front to adapt to the changing technology trends, if history is anything to go by. ATMs, core banking services, internet banking etc. are a testimony to this. The demography of financial services users is fast changing with the Millennials purported to be the top populace in a decade or so followed by the Gen Z. While both these demographics are inclined to show brand loyalties, their loyalty to convenience cannot be ignored. According to Javelin Strategy & Research, 50% of millennials will switch banks if their bank’s mobile application is complex to use. And with the advent of various fintechs servicing niche areas in the financial lifecycle traditionally serviced by banks, the onus is on banks to develop new attractive features, engage with the users and ensure availability of the financial platform to attract and retain the largest growing populace – the millennials and Gen Z.

With decoupling of banking services happening at an alarming pace, banks are no longer competing with other banks; but rather their competition is now with niche technology players. This in effect causes a total revamp of IT strategies of banks with one or many of these new features such as Chatbots, omni channel delivery, face recognition algorithms, two factor authentications, Robo Advisors, P2P Payments, Digital KYC, Cryptocurrencies, Block chain etc., to name a few.

Chatbots are one of the most popular touchpoints for millennials since they are available 24/7 and do not make the customer wait. As per Retale, more than 60% of millennials have used a Chatbot at least once and more than 70% of them reported a positive response. And not to mention the fact that having Chatbots enables organizations to divert their first level call center touchpoints to more critical areas of work. Unlike most touchpoints, Chatbots vary with their levels of interaction complexities. There are static menu based Chatbots that direct the user towards a certain part of the website through keyword recognition and there are dynamic Chatbots that use conversational AI and provide a human-like interaction experience with the user, and perform user validations, back-end financial operations, etc.

Depending on the type of Chatbot an organization uses, the testing strategy must be customized, while the basics can remain the same. All Chatbot testing strategies should aim for coverage of the following parameters:

  1. Usability
  2. Performance under load
  3. Conversation flow
    • a) Spelling & Grammar
    • b) Keyword recognition
    • c) Response appropriateness
    • d) Logical conclusion
    • e)Warning/Error messages & Notifications
  4. Session handling
  5. Integration to product functionalities
  6. Security

Testing Static Chatbots

Static Chatbots perform pre-defined actions and are primarily used to direct the user towards a feature/functionality of the application. They work by recognizing keywords in the user’s message and respond with an appropriate action. In most cases they do not involve in performing any business operation such as verifying a user’s identity, displaying an account balance, funds transfer, etc.

Such Chatbots are not too difficult to test and can be tested with an organizations existing test infrastructure. The key would be to generate loads of test data with appropriate but varying keywords that cover the end-to-end functionality of the Chatbot and using that to verify the response of the Chatbots. Being static Chatbots, the expected responses could be passed as expected result and compared with the actual response and take the conversation to a logical closure. The same could be accomplished using test automation through UI or in ideal situation through the Chatbot API if there is one. Any standard API or UI test automation tool could be used to automate the request and responses. Or specialized bot testing frameworks and tools such as Botium, TestMyBot, Watson Assistant, etc.

Additionally, negative tests and the non-functional aspects such as performance of the Chatbot, security vulnerabilities assessments, etc. must be part of the test strategy and can be evaluated using the standard tools such as JMeter, Owasp ZAP, SOAP UI, Rest Assured.

Testing Dynamic Chatbots

Dynamic Chatbots are much more complex to test. In addition to validating the usability, performance, security, etc., the functional testing must provide coverage to handle the cognitive engine that powers the conversation. These Chatbots typically understand the conversation flow, (and in rare cases the mood of the user as well) and respond accordingly in a human-like nature. They can perform backend operations too as requested by the users. Just like any other web or mobile app, Chatbots too must be tested for functionality, usability, performance, security etc., and at no cost should be ignored. However, there are a few aspects specific to Chatbots that form the critical success factors to testing Chatbots.

Critical Success Factors:

  • Test Data: With dynamic Chatbots, the problem typically lies in the accuracy and size of the test data that needs to be selected to interact with the Chatbot. The data set typically indulges the Chatbot to cover typical positive flows that traverse end-to-end as well as a good amount of negative flows. Typically, the variations in test data should be good enough to provide confidence to the testers. This type of testing is more subjective in nature and really depends on the skill level and creativity of the tester. But the most important part is to continuously monitor and fine tune the test data.
  • Conversational Score: The most complex aspect of testing Chatbots is making the testing objective when it comes to evaluating the conversation. The results are not usually black and white with a simple pass or fail. The right approach would be to assign a rating for each of the three key parameters – Behavioral Score, Intelligence Score and Responsiveness Score by answering a standard set of questionnaire.
    • The behavioral aspect should deal with the personality fitment with the target audience as well as the conversational tone of the Chatbot.
    • The Intelligence Score should deal with the ability of the Chatbot to understand the user’s question, his/her personality, mood and expectations and provide appropriate responses and perform backend operations. Also, the crucial part is logical handling of unexpected situations.
    • The Responsiveness Score should deal with the ability of the Chatbot to respond in a logical manner to resolve users’ queries with shortest possible flows.

The questions that probe the Chatbot for the above said parameters must be agreed upon with the testing and product teams for best results. And the important part here too is to continuously monitor and review the resulting trend. For the conversational score analysis, there is no readymade automated solution but rather the testers have to rely on their manual testing skills and their creativity.

  • Seamless Integration: The makers of an intelligent Chatbot would definitely hope for the Chatbot to handle business operations to a certain extent as well in addition to just making conversations. In order for this to happen, the Chatbots should have seamless integrations to business APIs. Testing these integrations can indeed be automated and can be handled at the API level rather than using the Chatbot GUI. But then the important part here is to verify the negative flows as well. A few examples of such integrations are
    • Ordering checkbooks through Chatbots
    • Funds transfer
    • Add payees
    • Bill Pay

While Chatbots themselves have been in existence for a while, there is still no standardized approach to testing them, let alone using test automation. And this is evident by the fact that there are scarce test automation tools for Chatbot testing. While we wait for the market to mature, the wise thing to do is to use automation for testing the integration, performance and security while relying on the creativity and imagination that a human being can bring to test the conversational flow.

Latest posts by Srinivasan Sankar (see all)