Catalyst-AI

The "Catalyst-AI" application is an experimental collaborative design simulation application developed to facilitate a design fiction test case for participatory urban planning and design, specifically focused on the conceptual development of the "Karlsruhe Library of the Future" within the context of a design fiction workshop held at the Karlsruhe University of Arts and Design in June 2024. By integrating AI driven chat and image generation with an interactive workshop environment, the application provided a unique interface for workshop participants role – playing as stakeholders to collaboratively articulate, explore and visualize design ideas in real-time through a chat interface augmented with the integration of a large language model.

Core Objectives and Implementation:

At its heart, the project simulates a collaborative design ideation environment where participants can engage in the conceptual development of the Karlsruhe Library of the Future. The augmented workshop experience was designed to foster creativity, encourage diverse perspectives, and facilitate rapid ideation among stakeholders aided by the abilities of generative AI to parse and synthesize ideas.

To enhance the collaborative design process, the system incorporates an AI-driven image generation capability. This feature allows participants to quickly visualize their ideas, transforming text-based concepts into visual representations. The implementation utilizes a locally running ComfyUI node for image generation and Anthropic’s Claude 3 Opus large language model connected via http requests to the Anthropic API for the chat interface.

The project was built to enable the use of Godot's UI provisions to display AI-generated images within the workshop interface. This immediate visualization which helps participants iterate on their ideas rapidly, fostering a more dynamic and engaging design process was however carried out in a separate generative AI application called InvokeAI for speed and convenience, given the limited capacity of the ComfyUI integration in Catalyst-AI. The integrated image generation within Catalyst-AI is a potential area for future improvement.

While multiplayer is possible and is intended for future iterations, CatalystAI was developed with a work around that allows a workshop group of multiple participants to use the same instance of the application by providing a pull down from which a different role can be selected by groups of participants for each chat prompt sent to the AI, to indicate the role the chat prompt is issued by.

The AI response was programmed to be guided by metaprompts that set the workshop context and the role the AI persona is playing in that contex.

The design fiction and speculative context underpinning the workshop established four roles of which one was reserved for an AI persona (project managers) to drive. Participants were able to adopt any of the other three different stakeholder perspectives (architects, city planning authority and property developers) to ensure a comprehensive exploration of design requirements and ideas. In playing the project manager role, the AI persona was guided with metaprompts that centered collaboration, community input and mediation in resolving divergent perspectives from the other roles.

A minimalistic user interface was favoured with a chat output window given the greatest prominence, followed by the prompt input window. A side bar is provided with options for changing the role, the day of the workshop and for generating images (partly implemented but not used during the workshop). Optional text-to-speech functionality was also integrated for playing back the AI output and this really increased the level of immersion for the workshop experience when used. Potentially, more immersive and tightly integrated AI voice services such as OpenAI’s Advanced Voice Mode can be integrated into a system like Catalyst-AI, this would make brainstorming sessions managed with Catalyst-AI a lot more immersive as the AI chat interface would be replaced completely with voice, both for prompts and AI responses. For prototyping and testing however, the version of Catalyst-AI described here and used for the workshop was pretty adequate.

Daily chat threads in Catalyst-AI are saved in JSON files, one for each day’s workshop session, and each file is automatically loaded, on application start, making it possible to have Catalyst-AI respond to queries or prompts that refer to instances from any of the previous days with full context awareness. These chat threads can also be retrieved and analysed for research purposes or as part of strategy sessions that follow ideation workshops.

In providing a platform where stakeholders can collectively visualize and iterate on design concepts in a collaborative framework, while in its first iteration, the system is evidently formulaic, somewhat predictable and given to cliches in its output – tendencies which can be dialled down with further fine-tuning – yet the project demonstrates the potential of AI-augmented tools in fostering community engagement and creative problem-solving in collaborative or multi-disciplinary contexts.

With further refinement, this approach can help not only in accelerating the ideation process but also help to improve communication between diverse stakeholders, leading to more inclusive and well-considered design outcomes. The project serves as a prototype for how emerging technologies can be harnessed in creative and technical collaboration.