From Scans to Stories with Gaussian Splatting

When 3D earns its place

I tend to think about useful 3D in three broad buckets. They overlap, and they’re certainly not the only possibilities, but they help me decide whether an idea deserves to move forward.

The first is taking someone somewhere, or some time, they couldn’t otherwise experience.

One of my early XR journalism projects at Syracuse University explored the history of Interstate 81 and the Black neighborhood it cut through. We scanned a church that had been central to the community and captured 360-degree images around the neighborhood. Historical photographs were aligned with those present-day views, so people could see what had once occupied a place they recognized. We also showed visualizations of possible changes to the road.

The power of that project came from holding the location steady while moving through time. Someone who knew those streets could see a past they hadn’t experienced and consider a different future. A familiar place gave the history a point of reference.

The second bucket is making space legible.

At Yahoo, we worked on basketball shot charts that used height and color to visualize where shots were being taken. Comparing seasons made a change in the game visible: shooting became concentrated around the basket and beyond the three-point line.

The extra dimension carried information. You could look at the court and understand a pattern, then compare how it changed over time. Even a data visualization needs that thread running through it. What are we helping someone see?

The third bucket is letting people act.

We built an experience around The Price Is Right wheel. We could have given people a model to look at. Instead, we let them spin it. They could also place it in their room in AR and see it at scale.

Of course people wanted to spin the wheel. That was the whole appeal. The interaction followed naturally from the object and gave people something they wanted to try.

I return to these examples because the reasons for using 3D are concrete. Explore a place across time. Understand a spatial pattern. Try something you’ve always wanted to try. The technology has a job to do.

The scan is a starting point

Gaussian splatting is exciting because of how convincingly it can bring a real place onto a screen. I spent a long time learning photogrammetry and working out how to get good scans. Seeing capture become more approachable opens up possibilities for people who haven’t spent years learning that process.

But an impressive scan can still become a pretty blob of data that nobody knows how to navigate.

If I hand a free-moving 3D scene to my grandma, she isn’t going to automatically know what to do. Those of us who build these experiences, or grew up playing games, take a lot of navigation habits for granted. Our audience may have none of them.

That’s the problem I’m interested in with StorySplat: what happens after the scan. How do we give someone a way into the space, help them understand what matters, and make exploring it feel worthwhile?

Before adding features, I think through four things: path, place, perspective, and interaction.

Give people a path

What should someone encounter first? Where should they go next? What needs to happen before the next part will make sense?

If the goal is simply to explore a hotel room, open navigation might be enough. If we’re trying to communicate a sequence of events or tell a story through a space, we need to orchestrate more of the experience.

In the hotel tour I showed at VRTO, a camera fly-through introduced the space, while clickable hotspots let visitors move between scenes. Images could appear when someone wanted more detail. Those choices gave people recognizable ways to move forward.

A path can be as simple as an opening camera movement or a clear next stop. It doesn’t have to dictate every second. It just has to give someone enough direction that they can engage with the content without first learning how to operate the scene.

Let the place shape the experience

A hotel room and an artist’s studio call for different choices.

In a hotel, someone may want to understand the layout and decide whether it suits their stay. Giving them freedom to look around can serve that purpose well. In an art studio, the objects and the way they sit together may be the reason to linger.

One of my favorite examples from the talk was a captured art studio. It was a working space, full of things that would naturally move and change. The scan held a version of that space at a particular moment, and visitors could click on artworks to open images and look more closely.

The room provided context for the work, while the images helped people examine individual pieces. Each part added something useful to the other. Thinking about what makes a particular place interesting helps you decide what to add to its scan.

Choose a perspective

Whose story are we telling, and what are we asking the audience to understand?

That question matters even when the scene looks like a straightforward record of a place. A visitor choosing a hotel has a different purpose from someone learning about a neighborhood’s history. The opening view, information, and route through the scene should reflect that purpose.

In the Syracuse project, showing an old photograph in the location where it belonged helped connect the neighborhood people knew with the community that had been there before. The choice to bring those views together was part of the storytelling.

For a scan, the same thinking should guide the material we attach to it. A caption, photograph, or voice can explain why a detail matters. Decide what you want people to understand, then give them the context to see it.

Make the interaction mean something

The wheel works because spinning it is the thing you want to do. That’s a useful standard to bring to other scenes.

One technique I like is comparing captures from two points in time with a slider. It could show a landscape in winter and summer, or a construction site before and after work. Moving between the views gives the visitor a direct way to examine the change.

In another scene, the useful action might be opening a photograph or moving into the next room. We don’t need to make every object clickable. We need to choose actions that help someone explore the thing they came to see.

There’s also a practical limit here: each interaction asks something of the visitor. A person who came to understand a place shouldn’t have to spend all their attention figuring out the controls.

Remember the sound

Sound is easy to forget when everyone is focused on how good a scan looks. I think it deserves much more attention in Gaussian splat experiences.

A voice can tell you why an object is there. Narration can accompany a route through a space. Spatial audio can locate a sound within the scene, giving it a relationship to where you are.

In the questions after my talk, we discussed capturing audio on location. That’s worth thinking about while you’re there. If you’re interviewing someone in a space you’re also scanning, their voice and video can become part of how you tell the story of that place.

I think of this as adding anchored meaning. The information belongs to something in the scene. You can encounter it where it makes sense, with the surrounding space helping you understand it.

Make it easy to enter

Most of my work has happened outside headsets. I care about spatial storytelling wherever someone can actually access it.

A link that opens on a phone gives people a simple way to enter an experience. They can encounter it inside an article or share it with someone else. That accessibility belongs in the design conversation from the beginning, alongside camera paths and interactions.

It also applies to the person making the experience. I could build these things by writing code, but many of the people with a place to document or a story to tell don’t have that background. Making the work more approachable is a big part of why I’m building StorySplat.

If you’re looking at a scan and wondering what to do next, start with what you want someone to get from it. Then choose a path, a perspective, and the few interactions that help them get there. Add the context and sound that the place needs.

And keep asking why. It’s still the most useful question I know.