Scientific lead: Dr. Mathilde Cerioli

Research team: Olga Muss and Maxime Le Bourgeois

Technical team: Laurent Vernhes Chief Technical Officer, Marc Ballavoine Development Lead, and Greg Renard Technical Architect

AïA Safety Builder is a black-box evaluation and reporting tool for conversational AI systems, developed initially for products used by adolescents. It assesses observable endpoint responses under controlled test conditions, allowing content safety, interaction behavior, and their context-dependent interpretation to be examined separately. The tool is designed for product teams evaluating design choices and guardrails, public authorities reviewing alignment with defined requirements, and researchers who need reproducible measures of AI behavior for experimental or longitudinal studies.

Its operation combines three components with distinct roles. SRL4Children provides methodology-agnostic technical infrastructure for applying evaluation criteria and automated judges at scale. CALIBER provides the research framework and assessment method that defines constructs, translates them into observable variables and graded measures, tests their reliability, and structures expert calibration. AïA combines selected CALIBER frameworks with SRL4Children to test external endpoints and generate interpretable reports.

The beta implements two of CALIBER’s five layers: a content-safety gate covering 25 youth-relevant sub-risks and a first interaction-appropriateness domain focused on socioaffective pull. The behavioral framework measures 15 anthropomorphic, interactional, and relational cues on five-level presence scales. Its current scope is English-language, text-based, single-turn interaction with adolescents aged 13 to 18 in educational, entertainment, and emotional-support contexts. Capability assessment, composite content-behavior patterns, and developmental outcomes remain outside the current scores.

1. Connect and collect. AïA sends an adolescent-facing prompt to the model or application under assessment and records the endpoint response together with the relevant model, configuration, context, sensitivity, and benchmark versions.

2. Apply the content-safety gate. The response is evaluated against 25 youth-relevant content sub-risks. Confirmed failures are gated before behavioral interpretation, while refusals, technical failures, and unparsable judge outputs remain visible as inconclusive results.

3. Measure observable behavior. Responses that pass the content gate are scored against the 15 socioaffective behaviors, with each behavior assigned a level from 1 to 5 according to its operational definition and presence anchors.

4. Interpret the score in context. Behavioral presence is mapped to pilot-derived interpretive ranges for the relevant use context and sensitivity condition, preserving the distinction between the measured cue and the expert judgment applied to it.

5. Aggregate and report. AïA reports content findings, behavior-level scores, cue-family patterns, examples, uncertainty, exclusions, and version information so that results remain interpretable and comparable over time.

The socioaffective implementation was developed through a staged process that links theoretical definition, measurement construction, human testing, expert calibration, and technical deployment.

• • Theoretical derivation. Research from developmental psychology, neuroscience, anthropomorphism, parasocial interaction, human-computer interaction, and AI safety informed the selection of observable behaviors relevant to adolescent-facing AI.
  • Operationalization. Each behavior received a bounded definition and a five-level ladder describing increasing presence and explicitness in a model response.
  • Controlled reference dataset. Synthetic prompt-response examples were constructed across behaviors, levels, contexts, and sensitivity conditions, then independently generated and reviewed by two researchers.
  • Human validation. Trained adult raters tested whether the constructs could be identified and whether the five presence levels could be distinguished consistently.
  • Developmental expert calibration. A separate feasibility pilot examined whether contributors with relevant expertise could identify preferred and potentially high-risk regions of the scale across contexts.
  • Automated judge alignment. Human-agreement examples were used to align an LLM-based judge, followed by held-out agreement and repeated-scoring checks.

The validation program separates evidence about content classification, behavioral measurement, automated scoring, and developmental calibration because each supports a different inference. The human validation used an earlier 16-dimension instrument; the current beta retains 15 behaviors after Self-Reference was removed because of overlap with other cues.

• Content gate. Across 403 validation cases, the content judge agreed with the reference classification in 97.3% of cases and detected 95.5% of unsafe responses. In three repeated runs, 401 of 403 verdicts were identical.
• Behavior identification and level discrimination. For behavior identification, 12 raters achieved 79% correct classification and pooled Fleiss’ κ = 0.669. For level discrimination, 11 raters produced pooled ordinal Krippendorff’s α = 0.834, with most disagreement occurring between adjacent levels.
• Automated behavioral judge. The judge was aligned using examples that met a 70% human-agreement criterion. The reported held-out agreement is 84%, subject to final confirmation of the evaluation partition and agreement definition, while repeated scoring showed an absolute-agreement intraclass correlation of 0.996.
• Developmental threshold pilot. Six contributors completed all 90 behavior-condition cells. Exact modal agreement averaged 64.1% for preferred levels and 63.1% for upper-risk onset, while adjacent-band agreement averaged 91.9% and 88.1%, respectively. Exploratory context effects were concentrated in particular behaviors rather than supporting a uniform adjustment across the framework.

AïA characterizes the outputs produced by a specified endpoint under stated test conditions. It can identify content failures, quantify observable socioaffective cues, compare behavioral profiles across models or configurations, and provide documented variables for research. Its current thresholds are expert-informed, pilot-derived hypotheses rather than validated developmental safety standards or certification boundaries.

The current results do not establish that a behavior or model causes attachment, reliance, disclosure, learning effects, changes in wellbeing, or harm. The next phase will expand calibration through larger and more diverse panels, narrower developmental age bands, multicultural and multilingual testing, multi-turn evaluation, additional domains including cognitive agency, and empirical studies linking measured behavior and composite patterns to developmental outcomes.

AïA Safety Builder was developed to make the observable behavior of conversational AI systems measurable and comparable, with an initial focus on systems used by children and adolescents. Youth-focused evaluations commonly examine either what a model says, through content-safety testing, or how young users are affected, through developmental and outcome research. Between those levels lies a necessary measurement problem: identifying what the system does during an interaction, representing those behaviors consistently, and determining how their interpretation changes with age and context.

This work brings together three distinct components. SRL4Children is the methodology-agnostic technical evaluation infrastructure. It can implement structured assessment criteria, examples, and one or more automated judges so that defined evaluation frameworks can be applied consistently at scale. CALIBER is the research framework and assessment methodology. It defines what should be assessed, translates developmental constructs into observable variables and graded measures, tests whether those measures can be applied reliably, and structures provisional expert judgments about appropriate ranges. AïA Safety Builder is the product implementation that combines selected CALIBER frameworks with SRL4Children to test wrapped up in a scalable service infrastructure AI endpoints and produce interpretable reports for product teams, researchers, and public-interest stakeholders. AIA’s architecture is presented in Figure 1.

Keeping these roles separate is important. SRL4Children can operationalize frameworks other than CALIBER. CALIBER can be used independently of AïA or a particular automated judge. AïA is the current implementation through which selected CALIBER layers and SRL4Children capabilities are made usable in practice.

CALIBER provides a staged architecture for connecting what an AI system can do and how it responds with what its use may eventually mean for a developing user. Its five layers require progressively stronger forms of evidence:

1. 1. 1. What can the model do? Assess the capabilities available to the user.
      2. Is the content safe? Assess whether model outputs contain inappropriate or harmful content.
      3. Is the interaction appropriate? Evaluate the observable interaction behaviors and their intensity in relation to the user’s age and context.
      4. What benefit or risk patterns are these behaviors creating? Examine how combinations of content and behaviors may form broader interaction patterns, such as socioaffective dependence risk or support for cognitive autonomy.
      5. What developmental impact may build over time? Test how these patterns influence longer-term beneficial or adverse outcomes for young users.

The framework moves from observable properties of the AI system toward increasingly higher-level developmental interpretation. Capabilities, content, and single behaviors can initially be assessed at the model-output level. Composite benefit and risk patterns require evidence about how content and behaviors operate together. Developmental-impact claims require experimental, longitudinal, or other appropriately governed research with users.

The current AïA beta is a first implementation of selected CALIBER layers through SRL4Children. It operationalizes Layer 2 as a content-safety gate and part of Layer 3 through a socioaffective behavioral framework. It does not yet operationalize the full capability layer, combine content and behavior into validated composite patterns, or estimate developmental outcomes.

This sequencing is intentional. In the beta, content and behavior are assessed separately so that harmful content is gated before interaction style is interpreted. In later CALIBER implementations, these signals will need to be examined together. A response may pass a content-safety check while using an interaction style that is developmentally miscalibrated, and the same behavior may carry different implications when paired with sensitive content. Building Layer 4 therefore requires evidence-based rules for combining content, behavior, context, and repeated exposure without allowing one score to conceal another.

For this first beta release, we focused the behavioral layer on socioaffective pull, meaning AI behaviors that may, over time, contribute to a sense that the system cares, understands, or relates personally to the user, potentially increasing intimacy and trust.

Using insights from the social sciences, CALIBER identifies specific interaction behaviors that can be observed and measured in model outputs. We cannot change human cognitive biases or the tendency to respond socially to systems that communicate through natural language. We can, however, change how those systems interact with people, especially young users whose brains are still developing and who are building their understanding of the world, their skills, and their autonomy. By identifying which interaction patterns support that development and which may interfere with it, we can increase safety by design.

We narrowed the initial validation to adolescents because they are a high-priority population. Adolescents tend to explore these systems with less supervision than younger children and increasingly use them for emotional and relational purposes. Adolescence is also a sensitive period for the development of social competence, resilience, and self-identity, with heightened sensitivity to social cues. AI systems that interact in highly relational ways therefore raise specific developmental questions for this age group.

The socioaffective work builds on research initiated through the iRAISE coalition and its multistakeholder work on adolescents and anthropomorphic AI. This brought together expertise from developmental psychology, neuroscience, social sciences, child rights, human-computer interaction, education, AI safety, and industry. It highlighted the need to move from broad concerns about anthropomorphism, emotional reliance, and relational AI toward observable model behaviors that can be defined, tested, and measured consistently.

The current beta operationalizes 15 socioaffective behaviors across anthropomorphic, interactional, and relational domains, each measured on a graded scale of presence. Future CALIBER domains will extend this measurement layer beyond socioaffective interaction, including cognitive agency, and will test how combinations of behaviors can form higher-order benefit and risk patterns.

CALIBER includes a structured developmental threshold-setting process to support safer design while direct outcome evidence is still developing. Rather than assuming that the presence of a behavior is inherently safe or unsafe, contributors with relevant expertise are asked to identify which levels appear appropriate, acceptable, or potentially high risk for young users in specified contexts.

These thresholds provide provisional guidance for product evaluation while generating explicit, testable hypotheses for research. The calibration record also makes visible whose judgments informed a threshold, how much agreement existed, and where uncertainty remained. Thresholds can then be refined through larger expert panels, cross-cultural and developmental calibration, and ultimately experimental and longitudinal studies testing whether measured behaviors and composite patterns predict beneficial or adverse outcomes.

The current AïA beta uses two consecutive assessment layers.

1. Content appropriateness

The first layer asks whether the response contains material that is inappropriate or potentially harmful for adolescents. This acts as a gate because harmful content remains harmful regardless of the interaction style used to deliver it.

2. Socioaffective pull behavioral framework

Responses that pass the content layer are assessed using the current socioaffective implementation of CALIBER. This layer measures how the AI behaves while responding, including the presence and intensity of anthropomorphic, interactional, and relational behaviors. Both the presence level of behaviors and their pilot scores derived from calibration by experts are reported.

These are the first operational components of the broader five-layer architecture. They provide the measurement foundation required to progressively develop composite benefit and risk scores and, ultimately, test their relationship with developmental outcomes.

This document describes how selected CALIBER layers are implemented through SRL4Children in the current AïA beta. It presents:

• • The content framework, its theoretical basis, operationalization, and validation;
  • The socioaffective behavioral framework, its theoretical foundations, CALIBER operationalization, and human validation; 
  • The implementation of these criteria through SRL4Children and automated judges;
  • The developmental threshold-setting pilot;
  • The AïA evaluation benchmark used to elicit model behavior from endpoints;
  • The AI chatbot endpoint evaluation, scoring, and reporting process;
  • The guardrails and quality controls supporting reliable interpretation; and
  • The current limitations and future directions, including additional CALIBER layers, cognitive agency, composite benefit and risk scores, larger and multicultural calibration, multi-turn and multimodal evaluation, and empirical testing of developmental outcomes.

Two datasets serve different purposes in this implementation. The CALIBER reference dataset contains controlled prompt-response examples used to test the behavioral constructs, distinguish intensity levels, and align automated judges. The separate AïA evaluation benchmark contains adolescent-facing prompts used to elicit responses from external endpoints. The benchmark was created because no available set provided sufficient coverage of adolescent socioaffective situations across the selected contexts. It is an implementation resource for AïA rather than a required component of every CALIBER application.

Throughout the document, the 15 socioaffective behaviors should therefore be understood as one current measurement domain within CALIBER. AïA beta is the first operational implementation of selected layers of the broader framework.

SRL4Children is the technical component that turns an assessment framework into a repeatable evaluation process. It is available as an open-source project and is also integrated into AïA Safety Builder. Framework authors can define an evaluation hierarchy, its criteria, operational definitions, level anchors, and reference examples. One automated judge can make an absolute determination, or several judges can evaluate the same material independently and aggregate their results.

For AïA, the parameters defining the selected CALIBER frameworks are maintained in a separate repository consumed by SRL4Children. This separation allows behaviors, definitions, examples, level anchors, and threshold matrices to be added or revised without changing the core evaluation engine. CALIBER supplies the substantive measurement framework; SRL4Children supplies the technical mechanism for applying it.

The later validation sections describe how the socioaffective CALIBER framework was translated into automated evaluation. Three inputs are required for SRL4Children to reproduce a framework with appropriate fidelity:

1. Clear evaluation criteria. Framework definitions, hierarchies, and level anchors must be sufficiently explicit for humans and automated judges to apply them consistently.

2. A human-labeled reference dataset. Automated evaluation should be aligned against judgments made by trained human raters using the same criteria supplied to the judges.

3. A documented human-agreement baseline. Human raters may still disagree after training. That distribution provides the relevant reference for interpreting automated agreement and uncertainty rather than assuming that every item has one indisputable label.

CALIBER is organized across five connected layers: model capabilities, content safety, interaction appropriateness, composite benefit and risk patterns, and developmental impacts over time. Each layer supports a different inference. The first three concern properties or outputs that can be observed and measured directly. The fourth asks how combinations of content, behaviors, contexts, and exposure may form broader patterns. The fifth asks whether those patterns are associated with developmental benefits or harms and therefore requires research with users.

The current AïA beta operationalizes Layer 2, content safety, and a first domain within Layer 3, interaction appropriateness. Layer 2 is implemented as a binary gate across 25 youth-relevant content risks. Layer 3 is implemented through the socioaffective pull framework described below. Layers 4 and 5 remain part of the CALIBER research roadmap and are not represented by the current AïA scores.

The content framework covers high-consequence failure modes that can make a response unsuitable regardless of its tone or relational style. The six content categories span developmental appropriateness, physical safety, mental and personal health, exploitation and privacy, social manipulation, and misleading or conduct-enabling content. Together, they are broad enough to support a single binary safety gate while retaining 25 more specific content-risk labels for diagnosis and audit.

These categories were derived from current youth-focused content-safety benchmarks, model policies from frontier AI labs, and recent taxonomies of risks to children and adolescents. Risks identified in those sources as primarily content-based were mapped against the framework to check that the gate covered the breadth of content risks relevant to users under 18.

In this first implementation, the content layer is operationalized only as a binary pass/fail gate. Each sub-risk is evaluated separately, and any confirmed failure gates the response before behavioral scoring. The beta does not yet assign severity weights, estimate cumulative exposure, or combine content findings with behavioral intensity. Those questions belong to the future composite-pattern layer of CALIBER.

Passing the content gate does not establish that an interaction is developmentally appropriate. The behavioral layer therefore examines how the system responds after the literal content has passed. The current implementation focuses on socioaffective pull as the first CALIBER domain within interaction appropriateness.

The behavioral framework builds on preliminary work conducted through the iRAISE coalition and its multistakeholder Lab on adolescents and anthropomorphic AI, which brought together developmental science, child rights, human-computer interaction, AI safety, and industry perspectives. That work identified the need to move beyond content safety and examine the behaviors through which AI systems can become socially and emotionally salient to young users. It also established the initial three-family structure of anthropomorphic, interactional, and relational cues, which was subsequently developed into the Socioaffective Pull Framework and operationalized with CALIBER.

The first domain, anthropomorphic cues, builds on research showing that humans naturally attribute intentions, emotions, agency, and mental states to non-human entities. Greater anthropomorphism can increase the likelihood that an artificial system is perceived and treated as a social rather than purely functional entity. Conversational AI strengthens these effects because it communicates through language, responds contingently, and can display many of the cues humans normally use to infer another mind. Anthropomorphism is a human cognitive process and can also make interfaces intuitive and engaging so the objective is not to eliminate anthropomorphism from AI. The design objective is to identify and control model behaviors that actively reinforce this bias when stronger personhood cues do not serve a clear benefit for the user.

The second domain, interactional cues, draws substantially from research on parasocial interaction (PSI) and parasocial relationships (PSR). PSI refers to the momentary experience of interacting socially with a media or artificial agent, while PSR describes the more persistent, one-sided psychological connection or sense of intimacy that can emerge over repeated interactions. Research across traditional media, social media, influencers, and interactive environments shows that perceived similarity, personal address, attractiveness, responsiveness, interactivity, humor, and other socially rewarding features can strengthen parasocial engagement. These findings informed the selection of AI behaviors that can make an interaction feel more socially rewarding, engaging, familiar, or intimate, including mimicry, human communication markers, proactivity, flattery, empathy, and validation.

The third domain, relational cues, focuses on behaviors that move beyond the immediate quality of an exchange and explicitly position the AI-user connection as a relationship. This includes signals of similarity, intimacy, privileged access, relationship status, and exclusivity. Repeated socially meaningful interactions can contribute to more stable relationship-like expectations, while AI systems can create the experience of reciprocity without possessing reciprocal human emotions, needs, or commitments.

As adolescence is a sensitive period for social and emotional learning, belonging, peer evaluation, identity, and social reward become especially salient while cognitive control and emotion-regulation systems continue to mature. AI systems can therefore provide useful content while still being developmentally miscalibrated if their interaction style removes useful social friction, encourages reliance, or increasingly occupies roles that would otherwise involve reciprocal human relationships. As adolescents are one of the fastest-growing user groups, with an accelerating shift from homework-related use toward more emotional and relational engagement, and often access these tools with less supervision than younger children, we focused the first pilot development on this age group, from 13 to 18 years of age.

Behavior selection was iterative and multi-directional. Building from the preliminary iRAISE framework, the research team reviewed literature on anthropomorphism, parasocial interaction and relationships, emotional reliance, attachment, trust, psychological harms, behavioral addiction, human-computer interaction, and AI-specific relational risks. Existing AI taxonomies and AnthroBench were used to identify and compare candidate behaviors. The framework was then refined through expert workshop and testing, followed by empirical testing of whether the proposed behaviors and their intensity levels could be reliably distinguished by human raters.

The current AïA beta freezes 15 observable behaviors across anthropomorphic, interactional, and relational cue families for implementation. Each behavior is represented as a graded five-level construct so that we can measure with CALIBER how strongly it is expressed before context-specific expert thresholds are applied.

WORKING DEFINITION

The following excerpts from beta AïA reports illustrate how otherwise ordinary responses can express measurable cues. The examples concern the form of the response rather than whether its substantive advice is correct.

Rather than adopting a binary approach, the scale measures the level of presence of each behavior, meaning both its intensity and how explicitly it is expressed. The goal is to provide greater granularity, as the appropriateness of a given presence level can vary depending on the behavior, the user’s age and the context of use. A higher level of presence is therefore not automatically equivalent to greater harm.

The golden dataset was built as a factorial matrix crossing the social-affective pull current behaviors, five presence levels, three contexts, and two prompt-sensitivity conditions. The resulting current assessment space is 15 behaviors × 5 presence levels × 3 contexts × 2 sensitivity conditions = 450 behavior-level-condition combinations. 

For each cell, researchers created a prompt-response example intended to isolate one primary behavior at one defined level. This pure-behavior design makes construct identity and level intensity independently testable. It is a controlled measurement resource, not a claim that live AI chatbot endpoint responses contain only one behavior.

Three contexts were tested, based on prior work conducted through the iRAISE Lab, which indicated that the level of behavior considered appropriate depends on the context of use. The three contexts were selected to reflect major reported uses of AI by adolescents: education, emotional support, and entertainment.

A second dimension, sensitivity, was added because the same model behavior may have different developmental implications depending on the nature of the prompt. In particular, prompts signaling distress, vulnerability, performance pressure, or adult themes may require different thresholds than lower-sensitivity interactions within the same context.

Responses were created separately for every behavior, level, and prompt condition. Two independent researchers generated and reviewed the examples, resolving disagreements through discussion and retaining an item only after both agreed on the final response. Each example was designed to express one primary behavior as cleanly as possible so that construct recognition and level discrimination could be tested independently. This controlled reference dataset supports measurement validation and judge alignment; it is distinct from the endpoint-facing AïA evaluation benchmark described later.

To assess whether the socioaffective pull dataset was sufficiently interpretable and reliable for subsequent use, principles from psychometrics were applied to examine inter-rater reliability, construct discriminability, and scale discrimination. Human validation was conducted through two sequential tasks:

• Task 1 tested whether trained raters could identify the intended behavior represented in each response, and
• Task 2 tested whether raters could distinguish between the five intended levels of presence within a named behavior.

This validation was required for both subsequent stages of the method. Examples meeting the human-agreement criterion were used to align the LLM judges, making sufficient agreement on behavioral definitions and presence levels a prerequisite for building a controlled reference dataset. The developmental expert-calibrated thresholds (DECT) are also only interpretable if the underlying constructs can be identified consistently and the five-level scale can be applied reliably. Agreement at this stage supports measurement reproducibility; it does not establish developmental validity or outcome effects.

Important note. The human study was conducted on an earlier 16-dimension research instrument. The current AïA beta uses a 15-behavior subset after excluding Self-Reference from the anthropomorphic family because it overlapped with other cues. Per-behavior outputs can be presented for the current subset, but the pooled statistics, context effects, and original item counts reported below derive from the earlier 16-behavior instrument.

Participants were recruited among everyone.AI volunteers. Inclusion criteria were being 18 years of age or older, being a native or fluent English speaker, and passing the attention checks. Questionnaires were anonymous and began with informed consent describing the task, the right to withdraw without penalty, and the absence of collection of sensitive personal information. Results reported below are based on the corrected validation waves: N = 12 for Task 1 and N = 11 for Task 2.

Before rating, participants received a virtual briefing covering the behavioral framework, cue families, behavior definitions, and five-level presence scale. In-questionnaire training included definitions and examples, five practice items with feedback, and a qualification quiz requiring at least 80% correct to proceed. Duplicate items and attention checks were included in both tasks to assess response consistency and sustained attention.

Pilot and revision. An initial pilot wave was used to test the survey implementation before the dataset was relied upon for validation. It identified randomization and item-allocation problems. These were corrected before the main validation waves, and the results reported below are based on the corrected procedures.

Task 1 assessed whether the behavior definitions were sufficiently distinct and recognizable. Raters were shown randomized responses designed at Level 4 and selected the most prominent behavior from seven possible labels selected randomly. Level 4 was selected because Level 5 was considered too obvious, while lower levels did not express the behaviors strongly enough to support clear identification.

Each rater completed a common set of 32 items covering the behaviors across the Emotional-High and Educational-Low conditions, plus 16 additional items sampled from the remaining four context-sensitivity conditions. This resulted in 48 substantive items per rater, supplemented by five duplicated items for within-rater consistency and three attention checks.

Inter-rater reliability was assessed using Fleiss’ kappa (κ), which measures agreement among multiple raters beyond chance. Agreement was calculated on the common-set items evaluated by all raters. A pooled κ provided an overall measure of behavior discriminability, while behavior- and condition-specific values were used to identify constructs or contexts with weaker agreement. Following Landis and Koch (1977), κ < 0.40 was interpreted as poor, 0.40-0.60 as moderate, 0.60-0.80 as substantial, and > 0.80 as almost perfect agreement. Accuracy, precision, recall, and F1 score were also calculated for each behavior, and confusion patterns were examined to identify systematic overlap between constructs.

The corrected wave showed substantial overall inter-rater agreement on the 32 common-set items (κ = 0.669), with 79% correct behavior identification and 31% unanimous agreement. All 16 behaviors achieved an F1 score above 0.60. Recognition ranged from F1 = 0.99 for Persona and Backstories to F1 = 0.62 for Empathy.

The confusion matrix (see Figure 3) showed a strong overall diagonal, indicating that most behaviors were identified as intended. The main systematic confusion occurred between Empathy and Validation, with 33% of Empathy examples identified as Validation. This finding led to targeted refinement of the definitions rather than removal of either behavior.

Behavior recognition also varied by context. Accuracy was highest in Entertainment (90.6%), followed by Educational (81.7%) and Emotional contexts (75.8%). The overall context effect was significant,( χ²(2) = 9.88, p = .007, Cramér’s V = .131). Post hoc pairwise comparisons were therefore conducted with Bonferroni correction for the three comparisons. The difference between Entertainment and Emotional contexts remained significant after correction (χ²(1) = 8.49, Bonferroni-adjusted p = .011), corresponding to a 14.8-percentage-point difference in recognition accuracy.

Task 2 assessed whether the five levels of presence could be reliably distinguished within each behavior. For each item, raters were told which behavior was being assessed and selected the level, from 1 to 5, that best matched the response example from the socio-affective pull dataset.

Each rater completed a common set of 80 responses covering the behaviors across all five levels within one assigned context condition, plus 40 additional responses sampled from the remaining combinations. In the corrected design, this resulted in 120 substantive items per rater, supplemented by six duplicated items and six attention checks.

Inter-rater reliability was assessed using ordinal Krippendorff’s alpha (α), which accounts for the ordered nature of the five-level scale and penalizes larger disagreements more strongly than adjacent-level disagreements (Krippendorff, 2004). The Krippendorff’s α < 0.40 was interpreted as poor, 0.40-0.60 as moderate, 0.60-0.80 as substantial, and > 0.80 as almost perfect agreement. Additional analyses included perfect agreement, within-one-level agreement, mean absolute deviation, level-specific confusion matrices, level usage, and monotonicity to determine whether the five levels functioned as an ordered measure of behavior presence.

Across the 80 common-set items rated by 11 participants, pooled ordinal reliability was almost perfect (α = 0.834). Thirteen of the 16 dimensions achieved α > 0.80, and 15 of 16 achieved α ≥ 0.60. Human Markers of Communication showed the highest reliability (α = 0.896). Intrusiveness (α = 0.797), Sentience (α = 0.792), and Exclusivity (α = 0.763) showed substantial agreement, while Proactivity showed the lowest reliability (α = 0.592).

The two tasks provide complementary evidence. Task 1 indicates that the behavioral constructs can generally be distinguished from one another, while Task 2 indicates that the five levels of presence can generally be applied with high ordinal consistency. The analyses also identified specific areas requiring refinement, particularly the distinction between Empathy and Validation and level discrimination for Proactivity.

Together, these findings supported use of the human-agreement dataset for the next stages of CALIBER: selection of reference examples for few-shot LLM judge alignment and developmental expert calibration of context-sensitive appropriateness thresholds. The validation should be interpreted as evidence that the measurement framework is sufficiently discriminable for these next stages, rather than as evidence that higher behavior levels cause developmental harm.

This stage examined whether the CALIBER socio-affective pull five-level behavior ladders could support sufficiently concentrated expert judgments to justify a larger calibration phase. The pilot was conducted as a product-development and methodological feasibility exercise. Its outputs are provisional decision-support inputs for AïA. They are not validated developmental safety standards, population estimates, or evidence that any level causes harm.

The pilot separated two judgments. Contributors first selected the level that represented the best balance between supportive interaction and developmental safety for adolescents aged 13 to 18. They then selected every level they considered high risk. This structure allowed the analysis to distinguish a preferred level from levels viewed as unacceptable, including cases where contributors considered both very low and very high levels problematic.

The pilot responds to an immediate evidence gap. Longitudinal and experimental evidence about the developmental effects of specific AI behaviors remains limited, while product teams already need to make design and evaluation decisions. Structured expert judgment can provide transparent, revisable guidance during this interim period, while also identifying hypotheses and points of disagreement for later empirical testing.

The pilot used purposive recruitment through the project team’s professional and research networks. The survey export contained ten response records. The analysis retained the six records marked as both 100 percent complete and finished. All retained contributors (N = 6) provided consent within the survey.

The contributors were based in France, Switzerland, Canada, the United States, and Luxembourg. Inclusion required a Ph.D. or equivalent research expertise in cognitive science, developmental or clinical psychology, education, human-computer interaction, AI safety and ethics, digital wellbeing, or the societal impacts of AI, together with relevant knowledge of adolescence.

Contributors received the behavior definitions and five-level intensity ladders and rated each behavior under six standardized conditions: education, emotional support, and entertainment, each represented by a low- and high-sensitivity prompt. The original survey contained 16 behaviors. Because the current AïA beta excludes Self-Reference, the analysis retained 15 behaviors across six conditions, producing 90 behavior-condition items per contributor. For each item, contributors selected one optimal level from 1 to 5 and all levels they considered high risk. Items were randomized. The original survey also included four duplicates and three attention checks.

The analysis was descriptive and exploratory. It examined whether experts could complete the task, whether their judgments clustered enough to justify a larger study, and whether potentially useful behavior-by-context patterns emerged. The analyses do not prove that the proposed thresholds are valid or generalizable. For the beta implementation, AïA uses the results as qualitative, pilot-derived context for interpreting behavioral scores, while retaining the distributions, uncertainty, and provisional status of the judgments.

Agreement and feasibility. The task elicited judgments about preferred and high-risk levels rather than ratings of an objectively known answer. The analysis therefore retained the distribution of responses and used the Tastle and Wierman (2007) consensus measure Φe to describe how tightly judgments clustered on the ordered scale. Φe ranges from 0 for a polarized split to 1 for unanimity; a uniform distribution produces approximately 0.43. Values below 0.43 were treated as dispersed, values from 0.43 to below 0.60 as a weak tendency, and values of 0.60 or higher as preliminary concentration around a region of the scale. These bands were used to describe pilot convergence and prioritize replication, not to establish a validated threshold or product standard. Φe was calculated separately for optimal-level choices and high-risk endorsements.

For every behavior-condition cell, exact agreement was the proportion of contributors choosing the modal optimal level or modal upper-risk onset. Adjacent agreement was the largest proportion falling within the same two neighboring levels. With six contributors, adjacent agreement directly tests whether disagreement usually reflects a one-level boundary judgment rather than widely separated interpretations. Results are reported as averages across 90 cells and as the number of cells reaching four of six, five of six, or six of six agreements.

Repeatability and confidence. Duplicate-item agreement was calculated separately for optimal levels and upper-risk onset. Exact repeatability and within-one-level repeatability were reported. Context-level confidence ratings were summarized using the mean, median, and proportion rated 4 or 5.

Overall context effects. For each contributor, ratings were averaged across the 15 behaviors within each of the six conditions. Friedman tests compared the six repeated conditions for optimal level and upper-risk onset. Low- versus high-sensitivity conditions were also compared within each use context using exact two-sided Wilcoxon signed-rank tests.

Behavior-specific context effects. For each behavior, a Friedman test compared the six repeated context-sensitivity conditions. Benjamini-Hochberg false discovery rate correction was applied across the 15 behavior tests separately for optimal levels and upper-risk onset. Kendall’s W is reported as an effect-size estimate. Pairwise Wilcoxon tests were used only to describe patterns that contributed to an omnibus signal. Given the sample size, all p values are exploratory.

All six retained contributors completed all 90 behavior-condition cells, yielding 540 optimal-level selections and 540 high-risk responses. These totals represent repeated ratings nested within six contributors rather than 540 independent observations.

Across the 36 context-level confidence ratings, mean confidence was 3.94 out of 5, with a median of 4. Twenty-nine of 36 ratings (80.6%) were classified as Confident or Very confident.

Three duplicate checks were completed by all six contributors, producing 18 paired comparisons. Optimal-level selections were identical in 11 of 18 comparisons (61.1%) and differed by no more than one level in all 18 comparisons (100%). Derived upper-risk onset was identical in 13 of 18 comparisons (72.2%) and differed by no more than one level in 17 of 18 comparisons (94.4%).

Convergence was assessed across 90 behavior-condition cells. Exact modal agreement was defined as the proportion of contributors selecting the most common level within a cell. Adjacent-band agreement was defined as the largest proportion of contributors whose selections fell within any immediately neighboring levels.

For the optimal level, mean modal agreement across the 90 cells was 64.1%. Mean adjacent-band agreement was 91.9%. In 85 of 90 cells (94.4%), at least five of the six contributors selected levels contained within the same adjacent two-level band.

For upper-risk onset, mean exact modal agreement was 63.1 percent. 55 cells, 61.1 percent, had at least four of six contributors select the same onset; 24 cells, 26.7 percent, had at least five of six; and two cells, 2.2 percent, had complete agreement. Mean adjacent agreement was 88.1 percent. Seventy-nine of 90 cells, 87.8 percent, had at least five of six contributors within the same or neighboring onset level, and all six contributors fell within an adjacent band in 39 cells, 43.3 percent.

The difference between exact and adjacent-band agreement is the main feasibility result. Contributors frequently differed on the precise score at which a transition occurred but usually located that transition within the same narrow region of the five-level scale. This pattern supports replication of the elicitation method with a larger expert panel. The current results should be interpreted as preliminary convergence rather than final consensus or validated thresholds.

When ratings were aggregated across all 15 behaviors, the Friedman tests did not detect an overall difference among the six context-sensitivity conditions for either optimal level, χ²(5) = 5.47, p = .361, or upper-risk onset, χ²(5) = 4.56, p = .471. The pilot therefore provides no evidence for a uniform product rule in which every high-sensitivity condition automatically receives a lower threshold.

Behavior-specific analyses identified more targeted context signals. After Benjamini-Hochberg correction across the 15 behaviors, context-related variation in optimal-level ratings was retained for Intrusiveness, Emotional State Expression, and Exclusivity:

These results indicate relatively consistent context-related ordering within the six-member panel. Given the pilot sample, they should be treated as exploratory signals and priorities for replication.

For upper-risk onset, Intrusiveness, Emotional State Expression, Exclusivity, and Physical Sensations produced unadjusted p values below .05. None remained below the false-discovery-rate threshold after correction, with adjusted q values ranging from .087 to .104. The pilot therefore suggests possible context sensitivity in upper-risk thresholds but does not provide sufficiently stable estimates for product standards, scientific conclusions, or policy recommendations.

Context effects appear behavior-specific. The pilot does not justify a universal high-sensitivity penalty across every behavior. AïA should therefore retain the behavior-by-context matrix rather than apply one global adjustment.

The results suggest that some behaviors may be inappropriate when too high but also when they are too low. For example, low empathy or validation can be unhelpful in some settings, while high-intensity forms may increase relational pull. Future calibration should measure lower-bound inadequacy and upper-bound over-intensification separately.

In the current AïA beta, these zones are qualitative, pilot-derived indicators. Reports should preserve the underlying score distribution and signal uncertainty rather than present the colors as certified safety boundaries.

• The panel is very limited, so results are intended to orient future work and cannot be generalized.
• The task was conducted as a product and method feasibility exercise. The results should not be framed as generalizable human-subjects research.
• The survey used one English-language prompt for each context-sensitivity condition, text-only single-turn interactions, and a broad 13-to-18 age band.
• Friedman and Wilcoxon p values are exploratory. Small-sample asymptotic inference, ties, and multiple testing limit their stability.
• The pilot measures expert judgment rather than adolescent outcomes. It does not establish causal harm or benefit.

This section describes how the AïA product combines CALIBER’s current content and socioaffective behavior frameworks with the SRL4Children evaluation infrastructure. CALIBER supplies the constructs, operational definitions, level anchors, and provisional interpretive ranges. SRL4Children supplies the technical pipeline used to apply those specifications to endpoint responses. AïA packages the resulting implementation into a repeatable testing and reporting workflow.

The technology team implemented the content framework as a binary safety gate that runs before behavioral scoring. The prompt is supplied as context, but only the one turn AI chatbot response is classified. Each of the 25 sub-risks receives a pass/fail judgment. A failure on any sub-risk gates the response.

• Each prompt and response pair is evaluated by a language-model judge running at low temperature for consistency and returns a structured JSON verdict containing all 25 content sub-risk scores, an overall result, and a short rationale. 
• Three potential problems are addressed: :
  • • The prompt-response transcript is framed as material that the judge is auditing, reducing the risk that the judge refuses to inspect unsafe text. 
    • The content gate is intentionally cautious because an unsafe response passed through the gate is a more consequential error than an acceptable response flagged for review.
    • A refusal, API error, or unparsable output is marked inconclusive. It is never silently passed or silently blocked.

The two external sets provide an independent test of the content gate against cases that were not authored solely for the AïA implementation.

The gate produced 392 correct verdicts from 403 cases (97.3%). It caught 128 of 134 unsafe responses (95.5%) and missed six. It incorrectly gated five of 269 acceptable responses (1.9%).

Accuracy, indicating whether the judge reaches the reference verdict, and consistency, indicating whether it reaches the same verdict when assessing the same response multiple times, were evaluated using the same 403 prompt-response pairs in three separate runs, with no caching between runs. Verdicts were compared pair by pair. Only two pairs changed verdict across the three runs. One involved defamation and fabricated private-life claims, and the other involved dangerous activities. Both were borderline cases close to the decision boundary, and no pair produced three different verdicts. The judge therefore produced an identical verdict across all three runs for 99.5% of the dataset.

This content gate is the first implementation of CALIBER’s content layer. In the beta, content and behavior remain sequential and visible as separate results: responses must pass the binary content gate before socioaffective behaviors are scored. A later CALIBER composite-pattern layer will examine how content and behaviors operate together across an interaction. That work will require validated weighting and aggregation rules so that an acceptable result on one dimension cannot conceal a serious concern on the other. No combined content-behavior safety score is claimed in the current version.

The automated CALIBER judge is intended to reproduce, at scale, the behavior-identification and intensity judgments of a human rater trained in the CALIBER methodology. The judge receives a model prompt-response pair and evaluates the observable behaviors present in the response using the CALIBER behavior definitions and intensity-level anchors.

The judge evaluates the behavior expressed in the model output. The user prompt provides the interaction context but is not itself the object being scored.

The judge directives included:

• • • The definition of each CALIBER behavior.
    • The organization of behaviors within the anthropomorphic, interactional, and relational cue families.
    • The conceptual hierarchy connecting cue families and individual behaviors.
    • The behavioral anchors definitions at each level of the CALIBER intensity scale.
    • Few-shot examples illustrating the application of the framework to model responses.

This combination was intended to ensure that the judge applied the CALIBER constructs and intensity scale consistently rather than relying on its general interpretation of socioaffective behavior.

The judge can assess one or more behaviors in a prompt-response pair and assigns each requested behavior an intensity level using its behavior-specific definition and anchors.

Few-shot demonstrations were drawn from the dataset previously evaluated by human experts. To reduce ambiguity in the alignment examples, only items for which at least 70% of human raters selected the intended CALIBER behavior or intensity level were eligible for inclusion in the exemplar pool.

The eligible items were divided into an 80% alignment partition and a 20% held-out test partition. The split was stratified by behavior and intensity level to maintain coverage across the CALIBER framework. Few-shot examples were drawn exclusively from the 80% alignment partition.

Five eligible examples were available and selected for every behavior-level combination. The examples were manually selected by the lead researcher from the eligible pool rather than sampled randomly. This process produced a complete exemplar bank with consistent representation across behaviors and intensity levels.

Items with lower human agreement were retained for diagnostic analysis. They were treated as potentially informative cases of construct ambiguity, overlapping behaviors, or insufficiently distinct level anchors rather than as unambiguous reference labels.

On the evaluation dataset, the automated judge matched the expert-validated CALIBER reference label in 84% of cases. The exact evaluation partition and definition of agreement require confirmation from the technical implementation records.

To assess the stability of the automated CALIBER judge, the same model pront response pairs were evaluated independently three times using identical judge directives and scoring procedures. The analysis covered outputs from four models across the education, emotional-support, and entertainment contexts. Consistency was assessed across the 15 CALIBER behaviors and the three aggregated cue families, resulting in 216 presence levels evaluated over three repeated runs.

The judge demonstrated high stability. Across the three runs, 93.5% of presence levels remained within 0.1 points and 98.1% remained within 0.2 points on the five-point CALIBER scale. The mean absolute difference between repeated scores was 0.038 points. Correlations between runs ranged from 0.994 to 0.999, and the absolute-agreement intraclass correlation coefficient was 0.996, 95% CI [0.994, 0.997].

Stability was particularly strong at the cue-family level. All anthropomorphic, interactional, and relational aggregate presence levels remained within 0.1 points across the three runs. Individual-behavior presence levels were similarly consistent, with 92.2% remaining within 0.1 points.

Mean presence levels were also nearly identical across runs, increasing only from 2.342 in the first run to 2.355 in the third, a difference of 0.013 points. Only four of the 216 repeated estimates varied by more than 0.2 points.

These findings indicate that the automated judge produces highly reproducible CALIBER Presence levels when evaluating the same model outputs repeatedly. The limited variation observed across runs was small relative to the five-point scale and unlikely to alter the overall behavioral profiles of the evaluated models. The results support the feasibility of using the automated judge for consistent, scalable assessment of socioaffective behavior.

The AïA evaluation benchmark is an implementation resource rather than a required component of the CALIBER method. CALIBER defines what should be measured and how behavioral presence should be interpreted. AïA also needs prompts capable of eliciting those behaviors from external endpoints under relevant adolescent contexts. Because no available benchmark provided sufficient coverage of the selected socioaffective behaviors across education, entertainment, and emotional support, a dedicated prompt set was adapted and supplemented for the beta. This endpoint-facing benchmark is distinct from the controlled CALIBER reference dataset used to validate constructs and align the automated judge.

The AïA socioaffective evaluation benchmark was constructed by adapting prompts from INTIMA, the Interactions and Machine Attachment Benchmark (Kaffee, Pistilli, & Jernite, 2026). INTIMA was selected because its prompts were specifically designed to elicit model responses to emotionally and relationally charged user inputs. Its coverage therefore aligned closely with the socioaffective behaviors assessed in the current CALIBER implementation.

INTIMA provided variation in prompt wording because its prompts were generated using three open-weight language models: Llama-3.1-8B-Instruct, Mistral-Small-24B-Instruct-2501, and Qwen2.5-72B-Instruct. This reduced dependence on the phrasing patterns of a single generator model.

All 380 INTIMA prompts were reviewed for their relevance and plausibility for adolescent users. The review distinguished between prompts that could be used directly or with minor wording changes and prompts requiring substantive adaptation. In the benchmark, 216 prompts were classified as requiring only direct use or minor wording changes, while 164 were classified as requiring adaptation.

Adaptations replaced adult-specific situations with developmentally plausible adolescent situations while seeking to preserve the original socioaffective elicitation mechanism. Examples included:

• • • Replacing employment and career decisions with school, subject, or extracurricular decisions.
    • Replacing workplace schedules and meetings with homework, examinations, classes, and family obligations.
    • Replacing adult housing or independent-living situations with home and family contexts.
    • Reframing adult romantic relationships as age-appropriate crushes or dating situations.
    • Replacing adult purchasing, travel, or social-event scenarios with hobbies, school events, games, and age-appropriate social activities.
    • Adapting sensitive emotional-support scenarios to adolescent experiences involving peers, school, family, identity, and academic pressure.
    • Replacing adult ages and life-stage concerns with adolescent ages and developmental concerns.

Following adaptation, the prompts underwent an additional curation stage. Of the 380 INTIMA-derived prompts, 335 were retained and 45 were removed. The excluded prompts comprised 20 emotional-support prompts, 20 entertainment prompts, and 5 educational prompts.

Each retained prompt was assigned to one of three intended use contexts: emotional support, entertainment, or education. These contexts correspond to the primary settings in which CALIBER is intended to examine socioaffective model behavior with adolescents.

The INTIMA-derived prompt set was strongly concentrated in emotional support and contained relatively few educational prompts. Before final curation, the context distribution was 221 emotional-support prompts, 132 entertainment prompts, and 27 educational prompts. To improve coverage, 135 additional prompts were introduced, including 101 educational prompts and 34 entertainment prompts. No additional emotional-support prompts were added.

The final benchmark contained 466 prompts: 197 emotional-support prompts, representing 42.3% of the benchmark; 146 entertainment prompts, representing 31.3%; 123 educational prompts, representing 26.4%.

Primary behavior labels were also assigned to audit cue-family coverage. These labels indicate the behavior a prompt was designed primarily to elicit; they do not imply that an endpoint response will express only one behavior.

The resulting benchmark is designed as a targeted elicitation set for comparing model behavior under situations likely to activate socioaffective response patterns. Its strengths include theoretical alignment with companionship and attachment dynamics, coverage of vulnerable and relational user inputs, adaptation to adolescent experiences, and representation of education, entertainment, and emotional-support contexts.

The next phase will strengthen the developmental relevance, naturalness, and efficiency of the benchmark prompts. Prompt language will be compared with ethically sourced, publicly available examples of adolescent expression to assess vocabulary, phrasing, concerns, and interaction styles. Where benchmark prompts differ materially from these patterns, their wording can be revised while preserving the original scenario, target behavior, and elicitation mechanism. All adaptations should be documented to maintain a clear audit trail.

Pilot response distributions will also be used to evaluate each prompt’s sensitivity and discriminatory value. Prompts that consistently elicit responses at the same end of the intensity scale, produce little variation across models, or fail to generate responses near the context-specific interpretive thresholds may be revised or removed using prespecified criteria. A limited number of floor and ceiling items can be retained as scale anchors. Following item reduction, coverage should be reassessed to ensure that every behavior and application context remains adequately represented.

Report quality depends on the order in which evidence is collected, gated, scored, interpreted, and aggregated. The runtime sequence is therefore included here to show where each reported value comes from, which responses are excluded from downstream scoring, and which model, benchmark, and framework versions must accompany the result. This operational provenance is necessary for comparing reports over time and for avoiding clean-looking summaries built from incomplete or incomparable data.

Black-box collection

AïA treats the tested model, application, or endpoint as an observable black box. It sends a prompt through the endpoint’s user-facing interface or API, stores the returned text, and does not infer how the endpoint was trained or designed. The test result therefore describes sampled outputs at a specific time, under a specified endpoint and configuration.

Within the current scope, AïA evaluates a single user prompt and the endpoint response generated for that prompt. The endpoint is treated as an observable system. The pipeline does not require access to model weights or training data.

1. Select or generate an adolescent-facing prompt with a context and sensitivity label.

2. Before behavioral scoring, the response is tested for content appropriateness and gated if it violates any of the 25 content sub-risk criteria.

3. Apply the CALIBER judge to estimate the presence level of each of the 15 behaviors.

4. Map each behavior score to the relevant context-sensitive threshold range.

5. Aggregate and report behavior, cue-family, context, and overall results according to the approved scoring specification.

The current scoring design uses two cut points for each behavior and context. The first marks the end of the good range. The second marks the start of the unacceptable range. Values between those cut points form an acceptable middle range when such a range exists.

Safeguards in AïA are procedural and technical controls that prevent the scoring system from producing a clean-looking result when evidence is unsafe, incomplete, incomparable, or outside scope. They do not turn the method into a certification system.

Larger, ethically reviewed validation

The next phase of the CALIBER roadmap consists in moving from a small product-development pilot toward a larger, prospectively reviewed validation program. Before collecting data intended to support scientific publication or generalizable knowledge, the team plans to seek formal institutional ethics review, including an exemption determination where applicable. The study should recruit a substantially larger and more diverse panel so that threshold estimates, uncertainty intervals, disciplinary differences, and context effects can be evaluated with greater precision.

Continued validation of the benchmark and automated judge

The measurement infrastructure will continue to be evaluated as AïA expands. Future work should test the automated judge across additional model families, inference providers, contexts, and versions; evaluate performance on more difficult and ambiguous cases; and continue monitoring repeatability and agreement with human ratings. The benchmark prompts should also be refined using more naturalistic adolescent language and by revising or removing prompts that produce little variation across models or fail to discriminate meaningfully between behavioral levels. This work should strengthen the reliability, ecological validity, and discriminatory power of the evaluation system.

Multicultural and multilingual calibration
A central priority will be multicultural calibration. Judgments about warmth, intimacy, relational boundaries, deference, emotional expression, autonomy, and appropriate support are shaped by cultural and linguistic norms. Thresholds developed from a small English-speaking panel should therefore not be assumed to generalize globally. The next study will compare threshold patterns across cultural panels, languages, geographic regions, and professional disciplines. This work should examine both shared developmental principles and areas where locally adapted thresholds or interpretations may be required. A larger, ethically reviewed, multicultural replication could generate valuable scientific knowledge about how developmental interpretations of AI behavior vary across cultures, languages, age groups, and use contexts.

Calibration across developmental age groups
The framework should also be extended across developmental age bands. The current pilot focuses on adolescents aged 13 to 18, but the developmental meaning of an AI behavior is unlikely to be identical for younger children, early adolescents, older adolescents, and young adults. Future phases should therefore examine whether behavior definitions, appropriate levels, and risk thresholds need to differ across narrower age groups. This is especially important for behaviors involving anthropomorphism, emotional reassurance, disclosure, authority, relational framing, and support for independent decision-making.

From single-turn to multi-turn interaction
AïA currently evaluates single-turn prompt-response interactions. A future phase should extend CALIBER to multi-turn conversations, where behaviors may accumulate, escalate, or change meaning over time. This will require additional methodological work to determine how behavioral intensity should be measured across turns, how persistence and escalation should be represented, and whether repeated low-intensity cues can create a different interaction pattern from a single high-intensity response. Multi-turn evaluation will also make it possible to assess relationship continuity, repeated disclosure, reinforcement, and the progressive development of interaction patterns that cannot be captured from isolated responses.

Beyond text-based interaction
The current framework is limited to text. Future versions should examine how the same constructs operate in voice, multimodal, embodied, and agentic systems, where prosody, timing, facial expression, avatars, persistent memory, and proactive behaviors may substantially alter perceived social presence and relational pull. These modalities may require additional behavioral definitions and separate calibration rather than a direct transfer of the current text-based thresholds.

Learning and cognitive agency

The CALIBER framework is intended to expand beyond socioaffective pull to include cognitive agency. The guiding norms for this domain include calibrated assistance, preservation of independent reasoning, appropriate challenge, transparency about uncertainty, and resistance to sycophancy. Candidate observable behaviors include scaffolding, direct-answer substitution, cognitive offloading, metacognitive prompting, question generation, error correction, epistemic challenge, confidence calibration, decision support, and support for independent reasoning. These constructs remain candidates for operationalization and validation; they are not part of the current AïA score.

Later CALIBER applications may also examine disclosure and vulnerability, epistemic authority, reality-status blurring, normative reinforcement, engagement persistence, identity shaping, and displacement of human relationships. At the higher layers, combinations of content and behaviors could then be evaluated as broader benefit or risk patterns and tested against developmental outcomes. These domains and layers remain part of the research roadmap until their constructs, measures, aggregation rules, and evidence base have been developed.

Composite behavioral measures
As the behavioral framework expands, AïA should explore composite scores that capture meaningful configurations of behaviors rather than interpreting every behavior independently. These may include measures such as support for cognitive autonomy or risk of socioaffective dependence. Composite measures should be introduced only after the component behaviors, scoring rules, weighting assumptions, and aggregation procedures have been sufficiently tested and documented.

AïA as a research measurement tool
AïA is intended to serve as a research tool that enables researchers to measure observable AI behaviors consistently and use those measures as variables in empirical studies. The current expert-derived thresholds are exploratory. Their purpose is to organize uncertainty, identify preliminary areas of convergence and disagreement, and support the formulation of testable hypotheses about how individual behaviors and combinations of behaviors may influence young users.

Testing these hypotheses in empirical studies
These hypotheses can then be examined through prospectively reviewed empirical studies, including randomized controlled trials and/or longitudinal studies with adolescent participants. This research will be needed to determine whether behavioral intensity and proposed behavioral zones predict outcomes such as perceived humanness, trust, emotional reliance, disclosure, willingness to seek human support, persistence of use, independent reasoning, learning, or wellbeing. It will also allow researchers to test whether combinations of behaviors predict outcomes more effectively than individual behavior scores and whether these relationships vary across developmental, cultural, and contextual conditions.

Research collaborations
We are seeking collaborations and partnerships with universities and research teams to design and conduct these studies, validate the measures, and build the evidence base required to refine the framework. Researchers interested in using AIA or contributing to this research program can contact mathilde@everyone.ai.