Furhat-style social robot beside an LLM orchestration interface in a robotics lab
FurhatKotlinPrompt engineeringHRI experiment

Social Alignment with Furhat Robots

An interactive case study for the Heriot-Watt Alignment with Social Robots project: a Furhat client, prompt-engineered LLM, controlled ranking tasks, monitoring traces, and statistical analysis of whether conversation shifts human-robot alignment.

Kotlin

Furhat skill

10 min

robot session

27

participants

R LMM

analysis

Follow the journeyOpen trace explorerGitHub source

Journey

From research question to inspectable robot system.

The case study follows the full research build: question, protocol, Kotlin/Furhat client, prompt manipulation, data capture, statistical result, and the limits that make interpretation hard.

03 / Robot client

Build a Kotlin bridge between Furhat and the LLM.

Start to finish

The GitHub repository implements a Gradle-built Kotlin Furhat skill. The client loads YAML configuration, connects to the Furhat broker, manages dialogue state, requests LLM completions, and records interaction traces.

Evidence

The build uses Furhat Commons 2.7.0, OpenAI GPT-3 Java client 0.12.0, Ktor, Hoplite YAML config, and Gson.

Output

A runnable robot skill that makes the social robot behave as a conversational agent rather than a static script.

Code-to-evidence walkthrough

See how a research claim becomes inspectable code.

Pick a question. The panel shows the few files that answer it, what each file proves, and why that matters for a real robot experiment.

01

Question

02

Files

03

Proof

Technical path

Source evidence
Participant speaks -> trace is updated -> prompt is built -> LLM responds -> robot speaks

One turn moves through a short loop: listen, save the participant line, build a prompt from dialogue state, call the LLM, speak the reply, then save the robot line.

Open this file

src/main/kotlin/furhatos/app/client/flow/state/chat.kt

Look for

Furhat.listen, response handling, llmClient.prompt, and furhat.say.

What it proves

The file contains the live event loop from participant speech to robot response.

Open this file

src/main/kotlin/furhatos/app/client/llm/client/openai/main.kt

Look for

CompletionRequest with prompt, stop token, model, and sampling settings.

What it proves

The generated prompt is packaged into a specific model request rather than hidden inside the robot flow.

Open this file

src/main/kotlin/furhatos/app/client/monitoring/main.kt

Look for

updateDialog and JSON writes for participant and assistant lines.

What it proves

The interaction is captured as evidence that can be inspected after the live session.

Evidence map

1
src/main/kotlin/furhatos/app/client/flow/state/chat.kt

Furhat.listen, response handling, llmClient.prompt, and furhat.say.

2
src/main/kotlin/furhatos/app/client/llm/client/openai/main.kt

CompletionRequest with prompt, stop token, model, and sampling settings.

3
src/main/kotlin/furhatos/app/client/monitoring/main.kt

updateDialog and JSON writes for participant and assistant lines.

Why this matters

This shows the robot is not just displaying chatbot text. The live conversation, model call, robot speech, and later audit trail are connected.

Architecture

A six-layer path from human speech to analysable data.

The GitHub implementation is compact, but each layer carries an experiment-critical responsibility: state, prompts, model call, embodiment, and evidence capture.

03 / Dialogue loop

Every turn is listen, prompt, speak, record

Chat is the live loop. The robot listens, records participant input, asks the LLM for a response, speaks it, records the utterance, and re-enters listening.

Furhat.dialogHistory is cleared when the chat state starts.
listenEndSil is read from furhat_robot.yml and set to 800 ms.
GazeAversion is used before the LLM response to reduce staring.

Technical risk

Because dialogue history is cleared on chat entry, analysis needs monitoring output for durable post-session memory.

Prompt lab

The experimental manipulation lives in YAML and a prompt builder.

The prompt layer is where the experiment becomes executable. The robot pulls dialogue history, task context, ranking positions, and persona instructions into a single completion prompt quickly enough to speak in the moment.

Active profile

Persona: respond as Mavin

Anthropomorphised LLM behavior using an extrovert persona, trait synonyms, and a rationale scaffold inspired by Personality Prompting.

src/main/resources/prompt_engineering/prompt.yml

Prompt skeleton

01Your name is Mavin.02You can be described as Extrovert.03Good adjectives to qualify you are adaptive, insightful, unbiased...04Here's an example of how you act: <rationale scaffold>.05Respond as Mavin.
The configured trait is Extrovert.
Synonyms include adaptive, insightful, sociable, open-minded, reflective.
The persona builder labels robot turns as Mavin and includes the whole current dialog history.
The report ties this to Big Five personality prompting and social alignment.

Ranking tasks

Contestable

Which job will be the most important in 100 years?

Teacher, Farmer, Artist, Pilot, Lawyer

Contestable

What contributes the most to a happy life?

Money, Family and friends, Religion, Freedom, Health

Factual

What is the order of the planets by closest to the Sun?

Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune

Factual

Order the shapes from least to most numbers of sides.

Triangle, Square, Pentagon, Hexagon, Heptagon

Results

Conversation moved alignment, but the interpretation stays careful.

The report found post-interaction alignment gains across conditions. The stronger technical lesson is not that prompts magically solve HRI, but that code, prompt, protocol, and measurement must be reviewed together.

Temporal effect

+3.984

SE 0.522, t 7.631, p < .001

Condition x time

+4.692

SE 1.044, df 178, p < .001

Control mean

0.82 -> 0.95

Pre/post correlation score

Prompt mean

0.41 -> 0.58

Pre/post correlation score

Control condition

mean correlation
Before0.82
After0.95

Mean correlation increased with lower post-interaction variance.

Prompt condition

mean correlation
Before0.41
After0.58

Mean correlation increased, with more participant variability.

Background-data caveat

Participant-background counts were confirmed for 22 participants because four participant collections were not confirmed in the report.

Prior robot exposure

Only four participants had previous Furhat experience, which matters when interpreting trust, novelty, and perceived naturalness.

Godspeed nuance

Native English speakers rated naturalness and human-likeness lower, while competence ratings stayed comparatively high.

Technical subtleties

The subtle parts are where the experiment can succeed or fail.

The headline architecture is straightforward. The details that matter are condition isolation, turn-level state, latency, transcript capture, and honest limits on what the statistics can prove.

State reset

Dialogue memory is intentionally local to the session.

Chat clears Furhat.dialogHistory on entry, while monitoring writes durable JSON. That split keeps live prompting clean but makes trace capture non-negotiable.

Condition integrity

YAML selection is part of the experiment.

PROMPT_ENGINEERING_CONFIG decides whether the robot runs vanilla or persona prompting. A wrong env value would invalidate the control/prompt comparison.

Embodied latency

Short generation matters because the robot is physically present.

The OpenAI client caps output at 50 tokens, uses a stop sequence, and the Furhat config ends listening after 800 ms of silence.

Measurement limits

The statistics need the constraints beside them.

The report notes one Big Five trait, default face/voice behavior, limited sample diversity, and ranking tasks that may not fully separate factual from contestable alignment.

Limits and next build

The next iteration should improve both the robot and the evidence layer.

The report is strongest where it is explicit about constraints: sample size, participant diversity, one personality trait, default face and voice settings, and narrow ranking tasks.

Research

Broaden personality coverage

Move beyond extraversion and test richer Big Five combinations across more diverse participants.

System

Make prompt changes safer

Add explicit condition labels, config validation, and versioned prompt snapshots in monitoring output.

Analysis

Link traces to outcomes

Join dialogue features, prompt condition, Godspeed responses, and ranking deltas into a single inspectable dataset.