Experiment shows human designers retain creative edge over AI

LAWRENCE — A new experiment shows that, at least from a human perspective, humans retain an edge over the latest generative artificial intelligence tools when it comes to brainstorming buildings. 

Mohammad Dastmalchi, assistant professor in the University of Kansas School of Architecture & Design, together with Newton D’souza of North Carolina State University, has published a chapter titled “Architectural Creativity Stranded at MidJourney? Evaluating Creative Potential of Prompts and Images in Generative AI” in the new book “Design Computing and Cognition ’24” (Springer).

“Part of what I study is how we utilize different media in the design process, or the design-thinking process, to enhance different stages of design,” Dastmalchi said.

In their experiment, D’souza and Dastmalchi used 6,000 images and prompts created using the MidJourney website, an AI-powered platform specialized in generating images from user prompts. They culled the images down to 306 based on relevance and finally to 20 for in-depth analysis.

Dastmalchi said this included images that were produced for exploration and abstraction, rather than finalized images of buildings and spaces.

For example, they looked into things like “how the concept of dancing could be turned into a building — very conceptual, very abstract.”

The researchers then asked a group of design experts to rate the images for creativity.

“We focused on the early phases of design,” Dastmalchi said, “so we picked images and prompts from people who were experimenting ... and we asked five experts in the field to rate how creative they think each image and its prompt is ... by arriving at a consensus.

“In this paper, we also trained ChatGPT to evaluate the creativity among all these same images and prompts, and we asked both the human experts and ChatGPT to provide their rationale for those ratings.”

The results?

“How people rated the designs versus ChatGPT was almost the opposite from each other,” Dastmalchi said. “Almost, but not exactly. But it was interesting, because the rationale is another data point that we had to analyze.

“We found that ChatGPT primarily focuses on functionality and utility. This occurs despite its training, which includes understanding that the images were selected for purposes of exploration and conceptualization.”

The expert designers, Dastmalchi and D’souza write, judged the best designs as those having the best “fit” between the creators’ stated intention and the end product.

Another result they found was a specific pattern in the prompts of images that experts rated as "highly creative," highlighting the crucial role of human input in AI-generated work.

"It seems like designers’ creative input will have a different level of importance in the age of generative AI," Dastmalchi said. "This suggests that while AI can generate impressive outputs, the quality and creativity of the results still heavily depend on the designer's ability to craft thoughtful and imaginative prompts. By guiding AI tools with well-considered instructions, designers can leverage technology to push the boundaries of innovation while retaining their own creative agency.”

Thus, for now, it seems, human space designers retain a certain creative edge over generative AI – at least when judged from a human perspective. But Dastmalchi intends to keep probing the evolution of the technology and its impact on the design process. The main goal of this study was to contribute to the ongoing conversation about the role of AI and professional designers in the design process.

“This study also triggered a new idea that we are currently working on that focuses on the application of training AI to evaluate novice designers’ work,” he said.