When was last you went on date? Was it through one of the dating apps, and how did you find the match, boring, interesting or barely a match?! We kinda took the hint from date-me-docs and inspired from the “Hang the DJ” Black Mirror episode did the following little experiment to simulate date between ai-agents based on real people.
Simulating Human Interaction
So if you are curious about simulating human interactions you must be definitely aware about Generative Agents: Interactive Simulacra of Human Behavior. The paper introduces "generative agents," that simulates human-like behavior, from daily routines to complex social interactions. Its a pretty cool paper, and has a even more fascinating open sourced repo. You should check it out if you haven’t already.
However for a quick experiment with decent frontend we needed something simple and more easily customisable; so we settled upon AI-Town** developed by Convex and a16z.
Date-Me-Docs to Date-Me-Town
It is every engineer’s wet dream to be able to simulate the reality. Because like with code or games, you can’t often restart in real life. So after the Generative Agents paper, I was curious about simulating some basic human interactions and was undecided between simulating a date or simulating a recruiting* process. But then saw this NY times article on date-me-docs, and the directory of docs made a perfect starting point for creating ai-agents to simulate dates.
At that point I roped in my colleague Tinsae to try out few experiments. And here is how he approached the topic in his own words -->
So far, we have a simulation environment that is highly customisable to meet our specific requirements. Additionally, we have dating profiles which will function as the personality templates for our agents. Our next objective is to construct a "Love Island" within which these agents can potentially form romantic relationships.
The initial stage involved designing the island's layout using Tiled software
For agent personalities we picked eight profiles from Date-Me Docs. These “personalities” are essentially predefined descriptions for each agent, which state in plain english who they are and what they’re into. Below is one example of such descriptions
{
name: 'Jose',
character: 'f5',
memories: [
{
type: 'identity' as const,
description: `I am 30 years old. I work as Head of Theory at Retro Biosciences. I like Reading, watching TV shows, attending parties, hiking and working out. My music preferences are Snarky Puppy, Dream Theater, Meute, Diablo Swing I am looking for someone who is Reliable, likes to plan ahead, truth-seeking and enjoys learning new things`,
},
{
type: 'plan' as const,
description: 'You want to find love.',
},
],
position: { x: 10, y: 10 },
}Let’s break down these properties:
name: the character’s first namecharacter: the sprite, or set of graphics, corresponding to this character (each refers to a specific subset of the character sprite-sheet.position: the character’s starting coordinates on the town mapmemories: chunks of information about this character, each with a specifictype:identity: a first- or second-person description of who this character is, what they’re interested in, what kind of attitude they have, etc.plan: a second-person description of the character’s main goal in “life”
Let’s see how our agents do in the island!
So, what happens when we initiate the simulation for our custom-built Love Island? First, the simulation environment and all the participating agents are instantiated with their unique profiles. The central engine kicks in, identifying all agents that are ready to be scheduled for activities—and in this case, that means everyone.
Once they're in the loop, agents start to move, logging their coordinates from point A to point B. Those not on a collision course use this time to update their internal memory states based on past experiences.
When two agents are on track to intersect, the system pairs them for an interaction. It's not just random chatter; the system looks at their past memories and current emotional states to kick off a meaningful dialogue. Each conversation begins with a context-aware greeting and unfolds as the system queries a vector database, incorporating historical interactions to generate the dialogue. Eventually, one of the agents decides it's time to wrap up, and both update their memory embeddings based on the interaction.
After each round of conversation, agents are marked as available for new activities, and the engine's cycle restarts. The result is a dynamic, ever-evolving social landscape within the simulation.
Curious about what the agents think of each other? We can directly query them within the simulation, much like you'd ask a friend about a recent date. For example, we had a chat with an agent named Priya, who just met Colin.
Summary:
This has been an intriguing experiment, but we're still a few steps away from achieving that "Hang-the-DJ" level of realism. Next we plan to work on refining the simulation to allow agents to engage in day-to-day activities and chores to get a more realistic match between people. I am also curious about locally run Llama 2 agent, fine-tuned with a dataset that includes emails, meetings, and social content to capture the full spectrum of human interaction. Let's see what the future simulation holds for us...:)
*AI Recruiting aka Hire-Me-Town
Before diving into the world of Date-Me-Town, we toyed with the idea of simulating the recruitment process. So you would have companies with their own job postings personified as ai-agents on one side, and ai-applicants based on CVs, cover letters, blogs, and social profiles on the other. The match-making wouldn't be random; it would be based on simulated work environments, aligned with the personal goals and ambitions of both the applicants and the companies. Hopping to find the most synergistic match that benefits both parties.
**AI Town
The primary goal of AI-town, beyond just being a lot of fun to work on, is to provide a platform with a strong foundation that is meant to be extended.
Tech Stack for AI-town
- Game engine & Database: Convex
- VectorDB: Pinecone
- Auth: Clerk
- Text model: OpenAI
- Deployment: Fly
- Pixel Art Generation: Replicate, Fal.ai
- Background Music Generation: Replicate using MusicGen
The fundamental components of the AI-town are agents, engine, journal and memory.
Agents are initialised with a default identity which is hard-coded. Agents either have a conversations with each other or walk around by themselves and reflect.
The engine is responsible for walking up the agents when they finish a task and schedule a branch of agents to move around when they need to.
The journal is an append only log that notes down every agent’s path, the time they start walking, the time they end walking and the agents they have spoken to recently. When the agent sets off on a new path they first consult the journal to see if they will collide into any other agents during their walk by inferring a path from the journal. If they haven’t talked to that agent recently that will kick off a conversation. In addition to tracking the path, the journal also keeps track of all the conversation messages and positioning of the characters which is important for recalling chat history and creating memories.
In Ai-town conversations are kicked of every time the engine has grouped two agents together that will run into each other on the map. The agent interaction functions handles most of the logic with setting up parameters and entering a conversation loop of the agents to go back and forth sending messages. The agents will face each other, create the conversation object with the leader and follower to determine who initiates the talking and retrieve the existing relationships to inform the tone of the conversation and finally enter the conversation loop.
Memories are created and stored when agents finish conversations or self-reflection. Memory creation most importantly involves creating an associated embedding in a vector database that can be recalled later in relevant conversation.