The Government of Togo, in collaboration with academic research partners from the University of California Berkeley Center for Effective Global Action (CEGA), the Data-Intensive Development Lab, J-PAL, and Northwestern University, conducted a systematic evaluation of the machine learning model used to target emergency cash transfers under the Novissi social protection programme. This evaluation component -- distinct from the targeting model itself (documented separately as TGO-001) -- focused on monitoring the performance, fairness, and error behaviour of the phone-based poverty prediction algorithms that were used to prioritise the poorest individuals for COVID-19 relief payments between November 2020 and March 2021.
The Novissi programme was launched in April 2020 by Togo's Ministry of Digital Economy and Digital Transformation (MENTD) as an emergency cash transfer programme for informal sector workers affected by the COVID-19 pandemic. The first phase distributed payments to 572,852 informal workers in the greater Lome area (World Bank, 2021). In a second phase, 57,000 new beneficiaries in the 100 poorest rural cantons were identified using machine learning algorithms trained on anonymised mobile phone metadata -- specifically call detail records (CDR) -- to predict individual consumption levels for 5.7 million subscribers, representing approximately 70 percent of Togo's population (World Bank, 2021; Aiken et al., 2022).
The performance monitoring and evaluation effort assessed how accurately the machine learning targeting model identified the poorest individuals compared to alternative targeting approaches. The evaluation used ground-truth consumption and wealth data collected through two independent datasets: a nationally representative field survey conducted in 2018-2019 (n = 6,171 households) and a large phone survey of active mobile subscribers conducted in 2020 (n = 8,915 individuals). These ground-truth datasets provided the benchmark against which model predictions were compared (Aiken et al., 2022).
The evaluation framework applied multiple performance metrics including classification accuracy (proportion of correctly identified poor and non-poor), precision (proportion of targeted individuals who are truly poor), exclusion error (proportion of truly poor individuals excluded from benefits), and inclusion error (proportion of non-poor individuals incorrectly receiving benefits). These metrics were calculated across different targeting thresholds and compared against several counterfactual targeting methods: geographic targeting at the prefecture and canton level, occupation-based targeting, and proxy means testing (PMT) -- the latter representing a hypothetical benchmark since no comprehensive social registry existed in Togo at the time (Aiken et al., 2022).
The evaluation found that the phone-based machine learning targeting approach reduced exclusion errors by 4 to 21 percent relative to the geographic targeting alternatives that were considered by the Government of Togo. However, relative to methods that would require a comprehensive social registry -- specifically proxy means testing based on detailed household survey data -- the machine learning approach increased exclusion errors by 9 to 35 percent. This finding was significant for informing future programme design, as it demonstrated that while phone-based ML targeting outperformed the feasible alternatives available during the crisis, it would not outperform traditional PMT if registry data were available (Aiken et al., 2022).
A critical component of the evaluation addressed algorithmic fairness and potential bias. The research team planned in-person surveys for 2021 specifically designed to detect inadvertent predictive bias against vulnerable subgroups, including women, illiterate individuals, and other marginalised populations. The fairness analysis disaggregated performance metrics by demographic characteristics including gender, literacy status, and other marginalisation indicators to identify whether the model systematically underperformed for specific groups (World Bank, 2021; Aiken et al., 2022).
The machine learning models evaluated were classical ML algorithms -- specifically gradient boosted decision trees and regularised logistic regression models -- trained on 857 to 1,042 CDR features describing aspects of each subscriber's mobile phone behaviour. These features included call and SMS activity patterns, mobility indicators derived from cell tower locations, social network characteristics, mobile data consumption, and expenditure on airtime and mobile money (Aiken et al., 2022). The evaluation results were published in the peer-reviewed journal Nature in 2022, enabling independent external scrutiny of the methodology and findings. The research was conducted under academic institutional review board (IRB) protocols, though no specific regulatory or legal framework governing the monitoring component was documented in the primary sources.