Evaluating how people interact with software and AI has long presented practical difficulties.
Traditional human studies provide valuable feedback but tend to be slow and expensive to organize, especially when the goal is to capture variation across large and varied groups. Offline benchmarks can process many cases quickly yet frequently overlook the range of backgrounds, preferences, and interactive behaviors that shape real outcomes.
A research effort led by investigators at Harvard University and the Massachusetts Institute of Technology (MIT) has developed an infrastructure called MatrAIx to address parts of this gap.
The system centers on Persona 8B, a collection of 8.3 billion records in which each entry is described by 1,290 categorical dimensions covering background, psychology, capability, behavior, and lifestyle.
That number is roughly the same as the entire human population on Earth.
The idea, after all, according to its GitHub page:
MatrAIx is a population-scale, persona-driven infrastructure for evaluating AI systems and interactive products with heterogeneous simulated users. Instead of testing against a generic or interchangeable user, MatrAIx instantiates sampled persona records as LLM agents and runs them through reproducible tasks across four environments — Survey, AI Chatbot, Web, and App (native desktop and mobile, including macOS and iOS).
These records combine two sources.
Some are produced by sampling from a directed acyclic graph that encodes conditional dependencies among attributes, drawing on demographic and survey data to maintain realistic correlations.
Others are extracted from existing human materials such as biographies, review histories, and survey responses, then mapped into the same schema with attributes filled only where supporting evidence exists.
A filtered subset of roughly one million records, including both human-grounded and synthetic examples, has been released for research use.
The infrastructure pairs these personas with a playground that runs them as agents driven by large language models.
Long story short, MatrAIx is a population-scale simulated-user evaluation infrastructure that tests AI systems and digital products by simulating the world with all of its population.
The agents operate inside four environments: structured surveys, conversations with chatbots, interactions with websites, and sessions inside applications running on simulated or containerized platforms.
A library of 1,010 tasks spans more than twenty-five domains and supplies concrete scenarios ranging from pricing decisions to feature discovery and privacy adjustments.
In reported experiments, more than 18,000 trials were completed across a selection of these tasks.
Validation work examined whether the agents reliably reflected assigned traits.
In one controlled set of 400 trials that tested ten behavioral attributes across the four environments, the agents expressed or correctly withheld the designated behaviors in 91.5% of cases.
Separate assessments of extraction quality for the human-grounded records produced mean scores around 4.1 out of 5 from human raters, with language-model judges showing substantial agreement on most metrics.
The practical uses center on pre-deployment examination of systems and products.
Researchers or developers can sample cohorts defined by specific attribute combinations and observe how those cohorts respond to a given interface, pricing change, or conversational flow.
The recorded trajectories support subgroup comparisons that aggregate metrics often conceal, such as differences in hesitation after a cost increase or willingness to continue after an assistant error.
Domains already represented include commerce, software tools, finance, and healthcare-related interactions.
Results can be inspected for consistency across different underlying language models, since outcomes have been shown to vary with the choice of model that animates the personas.
Several implications follow from the design.
The approach makes it feasible to generate large numbers of controlled interactions in parallel, reducing the time required to explore edge cases or demographic variation relative to recruiting human participants. It also creates a reproducible record of both the persona configuration and the interaction path, which can aid later analysis or comparison of product versions.
At the same time, the authors note clear limits.
The released coreset is calibrated only on selected marginal distributions and does not constitute a probability sample of any real population. Synthetic generation rests on priors that may simplify or distort joint patterns, while human-sourced records inherit the biases of their original materials.
Agent behavior remains an approximation whose fidelity depends on the language model employed.
Consequently, the system is positioned as a tool for hypothesis generation and screening rather than a substitute for studies with actual users. Any findings intended to inform consequential decisions still require confirmation against human data.
The work therefore occupies an intermediate position between purely synthetic benchmarks and full-scale human evaluation.
It expands the set of questions that can be asked at population-like scale while leaving open the necessity of grounding those questions in observed human responses.
