What building safer conversational AI for youth mental health taught me

Safe development of conversational AI systems in youth mental health contexts

This past week, I was very humbled to participate in a groundbreaking national hackathon focused on building safer AI for youth mental health. It was a very insightful experience that required some really deep thinking and long sleepless nights.

As an accessibility professional, I’ve dealt with a lot of problem solving, team building, product inclusion and accessibility challenges over the years, but this particular challenge felt different right from the start. Mental health, diversity, equity and inclusion were core to the challenge. We were tasked with this very important problem:

How might we make an AI agent genuinely safe for a young person who might already be having a really hard time?

The people behind the “users”

As we stress tested the mock virtual assistant, I kept envisioning real moments while we worked. Teamwork was vital as we were all dealing with the pressure of tight time-constraints while maintaining our own mental well-being in this context and throughout this process. My team quickly figured out that I'm naturally a bit of a nighthawk. Often late at night as I was working through this challenge, there were moments that brought me back to a younger version of myself. I pictured a teen lying awake at night, not sure who to talk to, or someone staring at their phone, typing and deleting the same message over and over. It brought me back to being that young person trying to make sense of something really heavy, without knowing how to put it into words.

It made everything feel more real, and more human. I quickly understood just how important quality was in this challenge.

Conversational AI, in this space, doesn’t feel like a feature. It feels like a presence. Like something that’s stepping into a moment which already matters a lot, and that feeling has stayed with me throughout this challenge and I'm sure it will stay with me beyond.

Rethinking what “safe” actually means

By creating scenarios to stress test a mock VA, we realized pretty quickly that the usual ways of thinking about safety and quality isn’t enough.

It’s easy to test the happy paths of simple, straight-forward, and carefully crafted "standard" conversations, document system failures and then create default “safe” responses. But what happens when we apply messy, multi-turn, multi-lingual conversations that are nuanced, vague and full of spelling mistakes, slang and cultural references? And then, what if we applied real harm reduction methodology to ask harder, messier questions like:

• Would this actually make someone feel understood?

• Could this response accidentally shut someone down?

• Are we creating space for someone to open up—or quietly pushing them away?

There were moments where I caught myself thinking, this sounds fine … but how would it land for a young girl who was born with short arms, lives in a small town, struggles with body image, is feeling alone and is looking for help? That shift, from checking boxes to really thinking about impact was extremely uncomfortable at times, but deeply important.

The part I kept coming back to...

Accessibility was always in the back of my mind, because this challenge had a deeper, more meaningful impact. If any young person can’t understand what’s being said, or if the experience feels confusing or overwhelming, none of it works.

Safety in this context isn't just about creating simple and straight-forward guardrails, it's about keeping language simple, making things easier to follow when someone’s already stressed, and supporting different ways people communicate. Typical self-expression doesn't exist in this context and the system would need to be able to navigate through the messiness of human behaviour in someone's most vulnerable moments.

For me, it reinforced something I already believe deeply: if it’s not accessible, it’s not safe.

Sitting with the tension

One of the hardest parts of the whole experience was sitting with this question:

When is AI helping—and when is it causing harm?

There’s a real risk in trying to make AI feel too capable in moments that actually need human intervention. Assessing whether something is being helpful or harmful means making sure it’s not only easy to reach a real person when it matters most, but it also means assessing the level of guidance and intervention that is required for a particular situation or circumstance. There isn’t a perfect answer here, but ignoring the question isn’t an option.

What I’m taking with me

I left the hackathon feeling a bit different about my work. I'm much more aware of the level of responsibility that comes with building in this space. More thoughtful about the small details that can change how something feels for someone on the other side.

A few things I know I’ll carry forward:

• Think about people in their hardest moments, not just their easiest ones

• Care about how something feels, not just how it performs

• Make space for youth voices—they understand this better than anyone

• Keep accessibility front and center, always

Final thought

What stuck with me most is this, when a young person reaches out for help, even in a small way, it’s not casual. It takes courage.

Whatever or whomever meets them in that moment, whether it’s a person or a piece of technology, becomes part of that experience. Being part of this hackathon reminded me that this isn’t just about building smarter systems. It's about showing up, in the right way, when it really matters.