Testing the digitized supply chain – A comprehensive report on the strategies implemented for testing a mobile application
Abstract
A brief about Manhattan TMS: (1)
· 40% of the top 20 retailers in US (by revenue) use our Transportation Management System
§ 40+ percent of all U.S. domestic grocery products consumed in the U.S. are transported with our TMS
§ Through SaaS offering of TMS we host close to 1,300 - 1,500 carrier users. They interact with 15 shipper companies on a daily basis
§ 8,000 – 10,000 shipments are built and tracked daily in SaaS TMS
End-to-end visibility in supply chain is very critical and this helps the entire supply chain in making more informed business decisions based on real-time information. Last mile delivery in any supply chain very crucial. TMS Mobile application was built kept in mind catering to solve last mile delivery problems by reaching out to large number of carriers in the network. This application provides a platform for carrier companies to easily communicate with shippers and provide real-time information on their shipment deliveries.
This paper provides an insight into some of the relevant challenges faced while testing such a mobile app, while also shedding light on the solutions undertaken to overcome these hurdles.
Key Features
· This application is used by non- technical users, Testing had to be catered to the roles were directly tied to the end customer (E.g. – driver, customer service representative etc.)
· The application is first of its kind to be uploaded to App Store & Google Play store, adherence to standards had to be maintained in test strategy. Security of the application was also important as it would be used outside the domain (2)
· Innovative tools were used to ensure right automation in place for us to be able to make frequent release to market. Used a combination of backend and front end automation integrated with a Continuous Integration (CI) environment
· In order to provide scalable solution to adapting newer devices, UI automation with real devices in an in-house device lab was implemented (3)
· Load testing was important to handle many carrier users logging in to the app. The server had to be capable of handling multiple requests at a time including concurrent users
· Innovative non-functional testing techniques were implemented to cater to the subtle needs of customer, which are typically unstated in requirements. This includes testing for battery life, weak network connectivity, mobile data consumption, RAM & internal storage, UI loading time and offline tracking
Table of Contents
Background Objective & Challenges 4
Holistic Testing Strategy 4
Functional Testing 5
Location testing with customer data 5
Non-functional testing 5
Automation Testing 6
Integration with Continuous Integration 6
Load Testing 7
Security Testing 7
Measurable Impact 8
Resources & Tools Implemented 8
Conclusion 8
References & Appendix 9
Background Objective & Challenges
TMS Mobile application itself was built to focus on a key challenge of supply chain management, i.e. visibility and real time tracking. Prior to building this solution, team had to rely on web based application and on board computer systems in trucks to integrate with web application. There are a large portion of carrier companies who are small scale. They do not possess the infrastructure as well as the technical expertise to handle this. Also, many carriers were using fax, telephone and other outdated manual processes for tracking in the absence of appropriate infrastructure. This mobile Application provided the small-scale carrier companies to be able to communicate to shippers using the mobile devices of their drivers without requiring any additional investment.
There were various challenges in testing the application
· A new version of the client side transportation management system is released four times in a year and each client side mobile build must be compatible with all versions of the backend application.
· Primarily the driver users are non-technical by nature and the app needs to be simple in usage
· Testing location based updates and geo-fencing was a challenge and mock locations were used to overcome this
· Offline mode testing as driver might go on long distances on highways without mobile internet connectivity
Holistic Testing Strategy
The testing strategies were directly tied by prioritizing the customer’s needs for the entire framework. There were several areas to consider for holistically testing the mobile application. Some of the key areas are represented within Figure 1. The document will talk about the key challenges within each of the individual areas. (3)
Functional Testing
To thoroughly test the functional requirements, a role-based testing strategy was implemented. The pages the user could access and the actions that could be performed was governed by the permissions or roles assigned to that user. It was important to ensure that there was no way for a user to access content or perform actions that they were not authorized to.
1) Carrier administrator – Manages tender responses and driver assignment
2) Driver – Manages shipment delivery and tracking
3) Owner operator – Manages tender responses, driver assignment and tracking (typically in smaller carrier companies)
Location testing with customer data
It was important to check that the app was correctly displaying the live GPS location retrieved from the device. To test this out, it was important to perform live tracking operations as well as simulated runs.
We used advanced technology through simulating shipment movement using mock GPS locations. This made it easier to test with real customer data as this could be done from a stationary location in Bangalore, India while the device was simulated to move between two locations in America. Everything from the exact route to be followed, wait time at various stops and the speed of travel could be simulated to mimic a real-life scenario.
Non-functional testing
There were many non-functional testing techniques which were adopted however few key ones include:
1) Offline Tracking
During drives on the highways for deliveries, there could be several instances when the phone loses network connectivity and hence offline tracking testing becomes very important.
2) User experience & compatibility testing
As mentioned earlier the user base of this application is non-technical by nature. We had to keep the experience of the users in mind to be able to test the app thoroughly
· User experience was tested by asking questions such as:
o Was the next action clearly indicated to the user?
o Can the user complete a function or transaction with minimum clicks and less training?
o Hierarchy of information on all pages - Is it shown in a clear and sensible manner?
o Mistakes made by the user - could it be easily rectified
· Compatibility testing was done through test execution on a wide variety of popular real devices with different versions of operating systems
o Real devices often show device specific issues that do not get come up on an emulator software
Apart from the points above, the following non-functional testing were also done:
· Battery life & CPU / RAM consumption (4)
· Mobile data & local storage consumption (5) (6)
· UI loading time & weak network connectivity
· Location services & low power mode testing
Automation Testing
Automation testing was done by implementing the principles of agile testing by performing in-sprint automation. To be able to make stride towards faster delivery to market, automation and CI integration is the key.
1) Server side automation:
· Various new API services were developed for the mobile application
· Automation script written for the transactional go through an end-to-end flow across various API calls
· Response from rest calls was tested against the expected outcome
2) Client-side automation
· Mobile application was developed using an ionic framework with code written in Angular JS
· Client side automation was done using Protractor due to its ease of implementation with Angular JS applications
· Protractor is a powerful browser automation framework that combines technologies from individual frameworks such as Mocha, Jasmine, Selenium and WebDriver (7)
· Functions with Selenium to create an automated test infrastructure that can simulate the user’s interaction with an Angular application run either on a web browser or within a real device
Integration with Continuous Integration
· Test cases are integrated with the CI repository and run twice daily with the latest code check-ins
· Extended the automation framework to run on real-devices within an in-house device lab
·
The devices would be running on different versions of operating systems and firmware versions to identify device specific failures that may not be caught using an emulator
Figure 2 - Continuous Integration for mobile applications
Load Testing
1) Server side:
· Individual rest calls should handle many users performing simultaneous actions without any delays in processing
· JMeter scripts for simulating load to test various rest calls
· Processing time from API calls checked to ensure multiple users performing the same action simultaneously can be handled
2) Client side:
· Handle multiple users logging in from individual devices and performing tracking operations concurrently
· Run above scenarios by extending the automation framework developed using Protractor
· Multiple concurrent browser sessions will be used instead of real-devices
Security Testing
· Data within the application was kept secure through a tight system of user authentication
○ User authentication based on roles assigned to individual users - only authorized users could access certain data
○ Tight security policies for password creation
· Static scanning was performed on code using a Veracode scanner
○ No exposed vulnerabilities that could be exploited by hackers or malicious individuals
Measurable Impact
There were several benefits that were observed through the testing strategies undertaken. Some of the key advantages include:
· Faster release to market - new releases of the mobile app are uploaded to play store and app store every quarter as opposed to yearly product builds
· Enabled the app to be SaaS ready by ensuring security and volume aspects thoroughly and customers in SaaS platform started using as soon as it is released to market
· Scalable solution offering by able to adopt newer devices in the market
· Able to create a huge market pull from small scale carrier segments
Resources & Tools Implemented
1) Functional & non-functional tools:
· Mock GPS location for geofencing testing & functional testing of shipment tracking with customer data
2) Automation tools:
· Protractor for UI automation of Angular JS mobile application
· Groovy / Test NG for Non-UI automation
3) Load testing tools:
· Jmeter for API load testing
4) Security testing tools:
· Veracode scan
Conclusion
Testing of mobile applications presents a range of complications that arise from the diversity within mobile operating systems, firmware versions, processing power and screen size variations. These differences make it extremely challenging for testers to holistically test the mobile application and guarantee that the app will deliver quality on all devices available within the market.
This paper provides an insight into our experiences while testing a mobile application while also shedding light on the solutions undertaken to overcome several testing challenges. Numerous areas were detailed including the functional and non-functional aspects as well as the strategies undertaken for automation, load and security testing.
References & Appendix
1. Manhattan Associates. Transportation Management System - Overview. [Online] Manh, 01 September 2017. [Cited: 01 September 2017.] http://www.manh.com/products/transportation-management/overview.
2. Saifi, Rahis. The 2017 Mobile App Market: Statistics, Trends, and Analysis. Business 2 Business Community. [Online] 6 January 2017. http://www.business2community.com/mobile-apps/2017-mobile-app-market-statistics-trends-analysis-01750346#UwdaEOz1Jwq7KokQ.97.
3. Mobile Application Testing. Aziz, Scott. 1, s.l.: UST Global, 2012.
4. Data Usage & Monitor. Google Play. Data Usage Monitor & Manager. [Online] 20 February 2017. [Cited: 20 May 2017.] https://play.google.com/store/apps/details?id=com.gn.datamonitor.
5. Battery Doctory. Google Play. Battery Doctor (Power Saver). [Online] 25 May 2017. [Cited: May 25 2017.] https://play.google.com/store/apps/details?id=com.ijinshan.kbatterydoctor_en&hl=en.
6. My Data Manager. My Data Manager Application. [Online] 20 February 2017. [Cited: 20 February 2017.] http://www.mydatamanagerapp.com/.
7. Protractor. Protractor - End to end testing for Angular JS. [Online] 20 May 2017. [Cited: 20 May 2017.] http://www.protractortest.org/#/.
Author Biography
Anukampa Pattnaik (Senior Manager, Quality Engineering)
15 years of Industry experience in all phases of software development life cycle and 10 years of experience in quality engineering and supply chain expertise.
· Rich experience in Transportation, understanding business challenges of transportation and how to leverage technology platform to provide solution
· Very strong product development expertise using Agile methodologies
· Strong experience in Automation, Good knowledge on TestNG, Junit, Selenium,Protrator REST API & Web services automation Expertise in Performance engineering using Jmeter
· Actively contributing to external forums by presenting white papers and formal presentation
Co-Author - Aashish Prasad (Quality Analyst)
Aashish Prasad has been working at Manhattan Associates as a quality analyst for the last 2 years. He has been primarily working on software quality testing for mobile applications as well as web applications. Currently, he is working within the Transportation & Logistics product suite and has utilized his functional expertise to enhance the product so that it is tailored to meet the customer’s requirements. He has completed his master’s degree in manufacturing engineering with business management from the University of Warwick in the U.K.
THANK YOU!