American Evaluation Association
October 16-19, 2013
Presenter: Bram Piot (part of the session entitled, “Mobile Technology in Developing Countries in the Field of Evaluation – Where are we?”
Presentation Title: Going mobile to get data used: lessons for evaluation from the development of mobile data collection systems
Authors: Bram Piot, Aleck Dhliwayo, Marie Solange Ngoueko
Population Services International (PSI) has implemented a number of mHealth solutions for monitoring and evaluation in order to generate more accurate, real-time data that can be applied for programmatic decision-making. Several systems have been developed and successfully scaled up in recent years. We highlight the advantages and challenges in implementing and scaling up mobile technology use. In Zimbabwe, PSI uses tablets to manage and report client records for sexual and reproductive health services provided in franchise clinics and by mobile teams. In Cameroon, an SMS-based system is used to collect weekly data on community case management and drug supplies for a large-scale child survival project. The adoption of these systems has improved the management and quality of routine monitoring data, reduced operating costs, and improved efficiency of our interventions through increased information use by program managers. Improved data will support evaluation of these programs in the future.
Session Title: Mobile technology in developing countries in the field of evaluation – where are we?
Mobile phone technology is poised to change the field of evaluation, especially for quantitative methods. Every month cellular voice and data networks reach further in developing countries. International non-governmental organizations (INGOs) are beginning to use mobile technology in their evaluations. This session will look at the lessons learned from using mobile technology for evaluation in developing country contexts from the perspective of four international NGOs, with examples from more than 10 uses of mobile technology for evaluation. The session will discuss both the advantages and disadvantages of mobile technology for evaluation and review the state of the field in 2013.
INGOs are using mobile technology in evaluation in developing countries. In evaluation of programs, INGO mobile technology use currently takes the following main forms:
• SMS for data collection uses a system that allows for limited data collection via SMS (1-10 fields) and is most useful for rapid data collection or crowd-sourced data collection • Electronic forms that collect data or responses in the format of a form rather than through voice or SMS.
• Pairing External devices with mobile phones / tablets such as medical instruments (e.g., devices to test blood or blood sugar), scanners, radio frequency identification tags, barcodes or fingerprint scanners with mobile phones or tablets to make measurements for an evaluation.
When designed and used properly, mobile technology eliminates the need for paper data collection; reduces the need for secure transport of paper instruments from the source to a central data warehousing location; and lessens the demands of data entry. Mobile data collection platforms also make data available to an audience that is much wider than could previously access it in real time while encrypted transmission and password protection of aggregated data safeguard data security. Compared to the previously used Windows-based personal digital assistants (PDAs), mobile phones are much more familiar to users, communications capability is integrated, and capturing images and voice as part of a dataset is feasible.
With a little forethought given to the system design, mobile technology can help overcome many issues plaguing traditional quantitative evaluation methods. First, the cloud-based technology underpinning mobile systems removes barriers of geography and function that have long impeded access to data and data sharing and extends data access to a wider brain trust of individuals who can help analyze, disseminate and use data.
Second, by consolidating data collection and entry, mobile technology also expedites digitization of data and faster data processing no longer delays evaluation results cleaning, analysis, dissemination and use.
Third, because of the systems of checks and skip patterns built into mobile technology data collection systems, data are cleaner; mobile technology can ensure that all the data are entered in the same format, thus allowing for seamless download into formats that usable by most INGO staff.
Fourth, mobile technology can also help to harness “big data” more effectively—permitting data aggregation, standardization and analysis across projects—and thus can facilitate systematic examination of the results of an overall approach rather than of discrete projects.
However, mobile technology is not a panacea. Evaluators must begin by carefully assessing the skill sets of those who will be using the technology as well as the availability of networks and power supply, and then carefully design the system and plan for training as required. Mobile technology may not be universally appropriate. For example, although Africa is the second largest mobile market after Asia, with an estimated 650 million subscribers and 44 million broadband Internet connections, vast territories are still not covered via mobile phones and power for charging mobile technologies also continues to be an issue.
Mobile technology may not be appropriate for all types of data collection. Qualitative data are still better collected using traditional methods (although this might soon change). Critically, the technology does not preclude the need for a careful system design, including piloting before launch, training of staff collecting the data and supervision of data collection. The indicators and linkages among the data must be planned to ensure that data still have face validity . Issues with systems interoperability and whether to use open standard or open source systems have been identified as major obstacles.
Before mobile technology becomes an expected intervention in evaluations, we should learn lessons about its advantages and disadvantages to the field of evaluation. This session will provide an overview of some of these lessons.