The term ‘human-in-the-loop’ refers to humans actively taking part in the operation, training, or decision-making of an artificial intelligence (AI) system. When using AI for social impact research, Laterite is a proponent of including our human experts in the loop at critical junctures in the process. However, this presents an enormous challenge:
How can our human team keep up with the tireless efficiency and the sheer volume of information produced by AIs, ensuring our research outputs are rigorous, useful, and not ‘slop’?
This is a question we have been grappling with at Laterite for a few years now, as we’ve been developing our suite of AI-assisted research tools, called LateriteAI. AI tools are incredibly capable and powerful, but often create extremely comprehensive outputs, like a Python script with thousands of lines of code, or – as in a recent Laterite project – 6,000+ unique sources and claims to review. While we can (and do) intentionally build verification steps in our AI workflows, the sheer volume of outputs is often the main challenge to review.
The task of reviewing this AI-generated work is both daunting and important, and we believe others are asking themselves similar questions. This blog shares some of our approaches to managing the AI review burden, and how we try to keep human beings ultimately in control of our research work.
Do we even need a human in the loop?
Short answer: for now, of course, yes. AI systems are progressing at astonishing rates, and are currently capable of solving decades-old math problems, predicting protein structures, and exploiting cybersecurity vulnerabilities. What these examples share in common, though, is that they have definite and verifiable answers – truths – that can be proved to be correct or incorrect.
Social science research has elements of this type of verifiable work, particularly when working with statistics and quantitative data. For example, you can arrive at a final, verifiable figure for the sum of a continuous variable from data you collected in a household survey – there is only one right answer. However, our research work is often more nuanced than that, involving subjectivity, requiring expert judgement and awareness of local context. It’s possible that two human researchers may look at the same set of data and evidence and arrive at different conclusions. This nuance is everything in our work, and although today’s AI is immensely capable, an expert human opinion is still crucial. Furthermore, an AI is only as good as the direction it is given. A well-intentioned AI may also arrive at an erroneous or misguided conclusion due to human error in instruction. A human review is therefore useful for catching human errors, too.
Where is human review most needed?
Along many of the dimensions relevant to the work of social researchers, today’s AIs are already more intelligent than any individual human being on the planet – a daunting thought. However, AIs still have strengths and weaknesses compared to human beings. From our experience using these tools for research at Laterite, and from building our own in-house AI-powered tools, including LateriteAI, AutoQuant, and AutoQual, we’ve observed these strengths and weaknesses up close.
We’ve observed some particular strengths in action. AIs excel at coding across multiple languages (Python, R, and Stata), producing and interpreting rich visual representations, and handling complex econometric tasks such as power calculations, regression analyses, and post-estimation robustness checks. They also offer powerful verification capabilities for catching code errors or report inconsistencies, process high-resource natural languages (like English, French, Chinese) effortlessly, and synthesize massive volumes of information quickly.
At the same time, we’ve noticed areas where AI is weaker. Models can display over-confidence, exaggerate claims, or occasionally invent insights. Without carefully designed instructions, an agentic tool might introduce fallback rules or truncate inputs unnoticed, impacting research quality. AI also tends to make unstated assumptions without seeking clarification, and often outputs robotic, verbose text with recurring, awkward phrasing. Finally, working in lower-resource languages like Swahili, Amharic, or Kinyarwanda remains a challenge, though performance continues to improve steadily.
When reviewing an AI’s work, it’s important to be aware of these strengths and weaknesses, and target your review specifically on areas where we know AI tends to struggle. For example, we will spend more time reviewing an AI-generated interview transcript in Amharic compared to one in English, as we know that the English one will be almost perfect.
What strategies does Laterite use to facilitate a human review?
Build in human review time from the start
Human review takes time and should not be an afterthought. Take care to plan human review of AI outputs into your timelines and work plans.
Ask AI to call out assumptions and thin evidence
AI will often make assumptions or judgement calls based on your request, which can lead an analysis astray if left unchecked. Your AI workflow should be guided to be explicit and identify thin or shaky evidence, or insights that are based on layers of assumptions. These flags should be prominent in the outputs of your AI workflow, allowing for an easier human review. Build these checks directly into your workflow, prompts, and code, not after.
Don’t fully rely on autonomous agents like Claude Code or Codex – build your own workflows with verifiability mechanisms
2026 has seen the rise of incredibly powerful agentic AI tools like Claude Code or Codex. It can be tempting to pass a research task off to one of these agents and have them get the job done. For simple tasks, this can work very well. But if you’re not careful, and when tackling a large mixed methods analysis, these tools can introduce serious bias into your work.
Anthropic recently decided that “Auto” agent mode would be the default configuration for Claude Code users, whereas previously the default was for the human user to ‘Accept Edits’. This update means that – for most Claude users – the agent will no longer stop to ask you permissions or questions unless it absolutely has to. Of course, you can change this setting, but it points to the direction of travel. Agents are becoming more and more autonomous, and users seemingly want to ‘set it and forget it’.
The major concern for those using these tools for social science research is that they are becoming more and more capable of accomplishing long-run tasks, and asking fewer questions along the way, burying critical assumptions into a black box. Questions of verification become lost in the AI’s workflow, and unless prompted carefully, might not even be considered at all.
To avoid this over-automation, we at Laterite choose to build our own AI workflows, with purpose-built modules for each stage of the research process, including verification. This allows us to review our AI’s work along the way in a consistent and structured way, and not have to search inside of the blackbox.
All of our LateriteAI tools follow this principle – they are purpose-built for the research function they are meant for, and have standard human verification checks built-in. In a sense, we are building a researcher harness for AI.
Improve the human review user experience
The task of reviewing AI outputs can be tedious. We can make the review process as painless as possible by generating a simple user interface to help human reviewers navigate the task. The latest AI tools are highly capable of building user interfaces for this purpose. Just explain your goal and the human review process, and have your AI build a review system that helps streamline the process.
On a recent project, we built a simple web platform that allowed human reviewers to comb through thousands of AI-generated claims and accept or reject them in a structured way.
Use random sampling
The principles of random sampling need not only apply to selecting study participants. We can also use random sampling to ease the burden of human review. When faced with thousands of AI-generated claims to review, for example, taking a random representative sample of the claims can allow you to determine the level of errors or incorrect claims in your AI-generated output without having to go through each and every one.
Track verifiable metrics where possible
On some AI-assisted tasks it is possible to measure the quality of outputs in a quantitative way. For example, when Laterite developed AutoQual, our AI-assisted qualitative research application, one functionality pertained to thematic coding of transcripts in softwares like MAXQDA or Nvivo. Inter-coder reliability (ICR) is a metric – measured from -1 to +1 – that is commonly used to measure the consistency of coding between human coders. We used this metric to measure the quality of our AI-assisted coding, to ensure it was performing at least as well as the human team. Having a stable and quantifiable metric like this allows you to make tweaks in your AI workflow and measure the resulting improvements. These metrics can be invaluable in maximizing the quality of AI outputs.
Ask AI to review AI
You can employ different AI models to do some of the heavy lifting for you. By passing the output of an AI workflow to a different AI, you might find gaps in the output that a human would not catch. Ask another AI to poke holes in your output and identify where your human review should begin.
Iterating and learning as we go
Faced with the daunting task of reviewing thousands of unique sources or lines of code, most of us agree that a human-in-the-loop is still a requirement when it comes to AI-assisted research. The real question therefore becomes “What are the appropriate strategies to do that review rigorously and without requiring hundreds of human hours?”
How is your organization dealing with reviewing AI outputs? We’d love to get in touch to compare notes.
This blog is written by John DiGiacomo, Director of Analytics at Laterite.