AI for 3D Special Interest Group
ACM SIGGRAPH
A gathering around the emerging intersection of generative AI and 3D representation, creation, and interaction.
AI3D Foundation / Field Building
Human-Guided Generative Interfaces
HGGI is the study of interfaces through which humans directly shape generative processes.
Generative AI is increasingly capable of producing images, objects, environments, software, and worlds.
But much of its interface has converged on a surprisingly narrow interaction: describe what you want, generate a result, and try again.
This leaves a large space unexplored.
What happens when generation becomes something we can directly manipulate?
Interaction
From prompting to interaction
A prompt communicates intent by translating it into language.
But human intent is not inherently linguistic.
We point. Draw. Move. Stretch. Arrange. Demonstrate. Compare. Navigate. Act in space.
We understand relationships that may be cumbersome—or impossible—to specify completely in words.
HGGI asks how those forms of expression can become first-class inputs to generative systems.
The objective is not simply to give humans more controls around a model. It is to construct representations in which human action and machine generation can operate on the same evolving object.
The interface becomes part of the computation.
Shared representations
Representation translation
Different models understand the world through different representations. Humans have representations of our own.
A central problem of HGGI is therefore representation translation:
Human expression Model representations Editable result
The human should not have to understand the internal representation required by every model in a pipeline.
The interface should translate.
This changes the role of generative AI from a system that interprets instructions and returns artifacts into a medium that can be continuously acted upon.
Experimental ground
Why 3D?
3D makes this problem unusually visible.
A spatial object can simultaneously have geometry, appearance, semantics, parts, views, physical relationships, and latent representations. A person may want to manipulate any of these without knowing which representation a particular model requires.
This makes 3D a useful experimental ground for a broader question:
How can people manipulate meaning across changing computational representations without having to perform the translation themselves?
Our work in generative 3D, spatial interfaces, and AI3D Canvas explores this question through working systems.
But HGGI is not limited to 3D.
The same problem appears wherever generative systems mediate between human intention and computational representation.
An emerging area of inquiry
Building the field
We use Human-Guided Generative Interfaces as a name for this emerging area of inquiry.
It intersects human-computer interaction, computer graphics, artificial intelligence, spatial computing, and creative tools, but is not reducible to any one of them.
We build systems.
We publish and present what we learn.
We create spaces for researchers and practitioners working on related problems to find one another.
And we share the ideas so that the field can develop beyond any single project, laboratory, company, or model.
Gatherings and labs
Field-building in practice
Implementing SOTA Generative AI Pipelines in Your 3D Application
ACM SIGGRAPH 2025 — Hands-on Lab
A practical exploration of connecting contemporary generative models to interactive 3D systems.
A field begins
We dream. We build. We share.
A field does not need to begin with permission.
Sometimes it begins by giving a recurring problem a name, building the systems that make it visible, and sharing enough of the work that other people can build somewhere you did not anticipate.