Hi, I'm Kyle Steinfeld, 
    and today I'll be talking about 
    an undergraduate research studio 
    titled "Drawn Together".
    conducted at UC Berkeley 
    in the Spring of 2020. 

[click]

This studio was motivated 
    by the observation 
    that changes in the media 
        through which design proceeds 
    are often associated with 
    the emergence of novel design practices 
    and new subjectivities.

While the dynamic 
    between design tools and design practices 
    is complex and non-deterministic, 
there are moments 
    when rapid development in one of these areas 
    catalyzes changes in the other. 
    
The nascent integration 
    of machine learning (ML) processes 
    into computer-aided design 
    suggests that we are in 
    just such a moment.

Students of the "Drawn Together" Studio

By introducing undergraduate novice students 
    to a set of experimental tools and processes 
    based on ML techniques, 
    this studio seeks to uncover 
    those original practices or new subjectivities 
    that might thereby arise. 

I'll describe in this talk 
    a series of small design projects 
    that examine the applicability of such tools 
    to early-stage architectural design. 

Specifically, 
    I'll document the integration 
    of several generative ML models 
    into undergraduate design pedagogy, 
    and evaluate their use 
    as "creative provocateurs" 
    at the start of a design. 

After surveying the resulting student work 
    and documenting the studio experience, 
    I'll offer a reflection on the approach,
    and present an argument 
    that this work suggests promising new modalities 
    of design authorship.

It has been observed (Carpo, 2017) that changes in the media through which design proceeds - that is, the means and methods of design - are often associated with the emergence of novel design practices and new subjectivities - the manner in which a designer may act, make decisions, and assert their agency.

I'll begin by re-stating the observation 
    that **changes in the media through which design proceeds** 
        that is, the means and methods of design
    are often associated 
    with the emergence of novel design practices 
    and new subjectivities 
        the manner in which a designer may act, 
            make decisions, 
            and assert their agency. 

While there is not a direct deterministic link between design tools and the culture of design practice - between, for example, design software and design subjectivities - there have been moments at which rapid development in one of these areas catalyzes changes in the other (Loukissas, 2012).

While there is not a direct deterministic link 
between design tools and the culture of design practice
    between, for example, design software and design subjectivities
there have been moments 
    at which rapid development 
    in one of these areas 
    catalyzes changes in the other

Just as the adoption of scripting in the early 2000's facilitated new practices such as rule-based design (Hensel, Menges, Weinstock, 2013), and as the later broad acceptance of parametric modeling tools facilitated new practices such as generative design (Leach, 2009), we may expect to see that a more widespread availability of design tools based on statistical inference will similarly engender novel design practices.

Just as the adoption of scripting 
    facilitated new practices such as rule-based design
just as parametric modeling tools 
    facilitated new practices such as generative design
we may expect to see 
that design tools based on statistical inference 
will similarly engender novel design practices. 

As such, the nascent integration of ML processes into computer-aided design suggests that we may be in the midsts of one such moment.

I, for one, would welcome such a shift
    from computer-aided design 
    to machine-augmented design.

Such a paradigm shift from computer-aided design to machine-augmented design would be most welcome. Existing models for computer-aided design fail to directly support what is arguably the important moment in early-stage design: that point before an idea is fully manifest, when we first recognize the pattern of a compelling solution that lies latent in our problem.

Existing models for computer-aided design 
fail to directly support 
an important moment in early-stage design: 
    that point before an idea is fully manifest, 
    when we first recognize 
    the pattern of a compelling solution
    that lies latent in our problem. 

Insofar as the machine learning is inherently relational, imagistic, historical, and concerned with the recognition of pattern, we posit that this technology may be better suited than the currently-dominant modes of computational design in supporting the abductive nature of early-stage design (Steinfeld, 2017).

As things currently stand,
    this is a moment for a sketchbook
    more than for a mouse and keyboard.
    but it need not be so.

Of particular interest here is one mechanism that has been shown to be important to supporting creative thinking: the "provocation". Speaking of the role that provocations can play in the creative process, Edward de Bono coined the term "po" to describe an intentionally disruptive stimulus that is used to facilitate creative thinking (Bono, 2015).

Insofar as machine learning 
    is inherently relational, 
    imagistic, 
    historical, 
    and concerned with the recognition of pattern, 
I propose that this technology 
may be better suited 
    than the currently-dominant modes of computational design 
in supporting the **abductive nature** 
of early-stage design.

While de Bono clearly articulated the concept, he was far from the first to notice the utility of disruptive points of origin in creative fields. Proceeding concepts and works include Pierre Boulez's aleatorism (Riley, 1966), which describe compositions resulting from actions made by chance; and Brian Eno and Peter Schmidt's "Oblique Strategies" (Eno and Schmidt, 1975), a series of prompts for overcoming creative blocks printed on cards.

Of particular interest 
in the work I show here 
is one mechanism 
that has been shown to be important 
to supporting creative thinking: 
    the **"provocation"**. 

The work of the studio seeks to understand how ML tools might function as tools of creative provocation in particular, and of tools of early stage design more broadly.

    
Speaking of the role 
that provocations can play 
in the creative process, 
Edward de Bono 
coined the term "po" 
to describe an intentionally disruptive stimulus 
that is used to facilitate creative thinking. 

This question comes at an opportune time. Triggered by new advances in machine learning, and the development of methods for making these advances visible and accessible to a wider audience, the past five years has seen a burst of renewed interest in generative practices across the domains of fine art, music, and graphic design.

de Bono was not the first 
to notice the utility 
of disruptive points of origin 
in creative fields. 

In this context, this work seeks to further our understanding of ML tools on architectural design.

"aleatorism" describes compositions resulting from actions made by chance;

     
Brian Eno and Peter Schmidt's "Oblique Strategies" 
    a series of prompts for overcoming creative blocks printed on cards.

the work of the studio 
follows in this tradition.
seeks to understand 
how ML tools 
might function as tools of creative provocation

Text Generation Models

A text generation model (Li et al, 2017) generates synthetic text that completes a given passage based on the corpus on which the model was trained. Among the most successful text generation models at the time of writing is the GPT-2 model (Radford et al, 2019), which serves as the basis of two online tools used by the studio: Talk to Transformer (King, 2019) and AI Dungeon (Walton, 2019).

As an example of a conditional generative model,
the studio employed 
the text generation models shown here,
each of which generates synthetic text 
that completes a given passage 
based on the corpus on which the model was trained.

Runway

The RunwayML platform (Valenzuela, 2018) is a ML model training, hosting, and distribution service. This platform is used in a variety of ways in the studio, and represents a critical link in a central workflow of the class: the connection between ML models generally, and the sketching environment of Adobe Photoshop. This workflow is discussed in a section below. The studio also made use of a particular web-based implementation of Runway, the Runway Generative Engine, which creates synthetic images from textual captions.

An important platform for the studio was RunwayML
    a ML model training, hosting, and distribution service. 
    
This platform is used in a variety of ways in the studio, 
    and represents a critical link 
    in a central workflow of the class: 
        the connection between ML models generally, 
        and the sketching environment of Adobe Photoshop.

Artbreeder

Artbreeder (Simon, 2019) is a web-based creative tool, created and maintained by Joel Simon while at Stochastic Labs in Berkeley, CA, that allows people to collaborate and explore high-complexity spaces of synthetic images generated by various generative adversarial networks (GANs). The studio employs two models hosted on Artbreeder: the "general" model that appears to be an implementation of BigGAN (Brock et al, 2019) and the "landscapes" model that is a custom-trained implementation of the same.

Another important platform 
    that facilitated access to ML models 
is **the Artbreeder platform**.

This web-based creative tool
    exposes high-complexity spaces 
    of synthetic images 
    generated by various generative adversarial networks (GANs). 
    
The studio such models on Artbreeder: 
    the "general" model 
        that appears to be an implementation of BigGAN 
    and the "landscapes" model 
        that is a custom-trained implementation of the same.

Sketch2Pix

Sketch2Pix is an augmented architectural drawing tool developed by the authors (Steinfeld, 2020) that supports architectural sketching augmented by automated image-to-image translation processes (Isola, 2016). This tool enables novice undergraduate designers to conceptualize, train, and apply their own personal AI "drawing partners".

The RunwayML platform forms a link between student-trained image-to-image translation models and the sketching environment of Photoshop. Because students hold agency both over the training and the application of these "bots", and as such are able to configure these tools to meet the perceived demands of a given design problem as well as the subjective dictates of their trainer's tastes, we observe that the approach taken here suggests promising new modalities of design authorship.

This brings us
to the central tool 
employed by the studio:
    **Sketch2Pix**

[click]

Sketch2Pix 
is an augmented architectural drawing tool 
developed by the authors
    that supports architectural sketching 
    augmented by automated image-to-image translation processes 
    
This tool enables novice undergraduate designers 
to conceptualize, train, and apply 
their own personal 
AI drawing partners.     

Augmented Sketching as a Daily Practice

Given the novelty of the augmented sketching tools adopted by the studio, students are asked to engage in a practice of daily sketching, and to post these sketches for public display. This practice served to encourage increased competency with these tools. The primary conduit for the public display of this work is an Instagram hashtag related to the course: #ARCH100D. The graphic material found at this hashtag relates to each of the propositions listed here.

I'd first note that 
to cultivate a basic competency 
the tools of the course, 
students were asked 
to engage in a practice 
of daily sketching, 
and to post these sketches 
for public display

Three Propositions

Given the speculative nature of the course, rather than privilege the development of a singular design project, the studio proceeds through a series of three lightly-connected "propositions" that explore the potential role of ML tools in design. By proceeding in short bursts, the studio values patience in allowing the discovery of small questions to aggregate into larger and more elaborate proposals. The role of each ML tool differs, and each proposition offers a chance to better know the underlying technology and how it might figure in a larger process of design.

While the studio is primarily driven by method, such an investigation benefits from the comprehensive details of an architectural design problem. This begs the question: what is an appropriate test bed for ML tools as technologies of the artificial? The studio responds with a focus on the Northern California Landscape, and on the interface between the built environment and the natural environment.

In the sections that follow, we detail each of the three propositions completed by the studio, illustrated by a small sample of student work. Prior to this, we briefly discuss an ongoing practice of conceptual sketching that permeated the semester.

Speaking very broadly,
the studio proceeded 
through a series 
of three lightly-connected "propositions" 
that explore the potential role 
of ML tools in design.

I'll describe these in summary.

Proposition One: Strange Fruit

This two-week proposition introduces students to the nature of the ML tools employed in the course, and provides a platform for understanding the utility of these tools as design prompts.

Here, students are instructed to select a fruit or vegetable grown in Northern California, and to conduct basic research on the life-cycle of this produce, including the climate in which it was grown, and who may have participated in its production. Next, based on this research, students collaborate with a text generation bot to author a narrative about a person involved in the production of the produce, including an explicit description of the various settings involved. Then, based on this story, students use the Runway Generative Engine to create a storyboard of scenographic images generated from textual captions.

The first proposition
introduces students 
to the nature of the ML tools 
employed in the course

First, students were instructed to select a fruit or vegetable that is grown in Northern California, and to research this produce.

Next, based on this research, students collaborate with a text generation bot to write a story about a person involved in the production of the produce.

Finally, based on this story, students use an image generation tool to create a storyboard of seven (7) captioned images.

Here, students employed a text generation bot
that serves as a provocation
in the authoring of an imaginary user-narrative.

For example, 
a narrative about the actions of a strawberry-picker.

In parallel with this exercise, students utilize the Sketch2Pix tool to train an augmented drawing assistant - in the parlance of the studio, a "brush" - based on their chosen produce.

As is detailed in another scope of work, the technical process of training these brushes combines 3d scanning, calibrated rendering, and a custom implementation of a Pix2Pix model (Isola, 2016). Once trained, this brush allows students to sketch imagined three-dimensional forms and spaces in collaboration with a conditional image generation model that generates the colors, textures, and forms that are related to the produce on which the brush is trained.

In parallel with this 
students trained an augmented drawing assistant
using the Sketch2Pix tool

At the end of this two-week period, students present a graphic interpretation of the images generated from captions, applying their "brushes" in the service of the embedding of a set of three-dimensional forms and spaces.

While this first proposition serves primarily to build familiarity with the requisite tools, we may clearly see the positioning of the ML generative model as a early-design "provocateur", as it serves to provoke a creative response in the tradition of Eno, Schmidt, and Boulez.

..and then deployed this tool 
to compose conceptual sketches 
for an imagined architecture
that serves our imagined user.

Proposition Two: Landscapes of Change

This two-week proposition expands upon some of the tools introduced in the previous proposition, Artbreeder and Sketch2Pix in particular, and seeks to extend these further into the realm of architectural production.

The second proposition expands upon these themes
asks students to design an intervention
for a synthetic landscape.

To begin, students again train an augmented drawing assistant, or "brush", using more familiar subject matter: the architectural model. Here, students construct a series of non-scaled physical models that employ an intentional formal language common to academic design studios.

We again trained 
an augmented drawing assistant, 
but this time we trained on 
somewhat more familiar subject matter: 
    the architectural model. 

Examples include: a series of foam-core massing models; a collection of plaster "blob" models; and a third series of linear basswood matrix models. These models are then 3d scanned such that, employing the process mentioned above, they may serve as the basis of a training set for a "brush". The result is an augmented drawing assistant that transforms hand-drawn sketches into "deepfake" photographs of architectural models. These tools were then deployed in the service of the design of a dwelling for a specific site.

This proceeded through
the construction and scanning 
non-scaled physical models 
that employ an intentional formal language 
common to academic design studios.

To conceptualize an appropriate site for these dwellings expressed as "deepfake" architectural models, students are asked to return again to Artbreeder. First, selecting from a number of sites at the border between urban and rural spaces in the San Francisco Bay Area, students speculate on the changes likely to occur at this site over a span of 100 years. Then, students work with the "gene-splicing" feature of the Artbreeder landscape model to produce images that evoke their chosen site and animations of this imagined transformation. This approach to the generation of images - via a manipulation of "genes", or archetypal features discovered during training - is a mode of authorship unique to the latent space of generative adversarial networks (GANs), and is distinct from other modes of large-space exploration currently practiced in architectural design (Woodbury and Burrow, 2006).

The result is an augmented drawing assistant 
that transforms hand-drawn sketches 
into "deepfake" photographs 
of architectural models.

At the end of this two-week period, students again present a graphic interpretation that functions as an early-stage design proposal. Here, following the steps described above, the proposal is expressed as a photomontage that shows a a "deepfake" architectural model situated in a synthethic landscape.

These augmented drawing assistants 
were then used to compose designs 
for synthethic landscape sites.

Proposition Three: Four Elements of a Synthetic Architecture

The final proposition of the studio extends over nearly four weeks. Continuing the movement of the studio toward the language of architecture, here we focus on developing a more formal language of building systems, expressed through a single drawing type: the axonometric.

The final design exercise
seeks to develop a language of building systems, 
expressed through a single drawing type: 
    the axonometric.

Following Gottfried Semper (Semper, 1851), and using Artbreeder as a provocateur once again, students each make a coordinated proposal for four elemental building systems. Each of these forms the basis of the training of a separate Sketch2Pix brush, which is then employed to produce a number of sketch proposals of dwellings. As above, these dwellings are proposed for a synthetic landscape, but are developed simultaneously through two types of projected drawings: exploded axonometric and perspective.

To develop this collection of four related brushes, students first make use of the Artbreeder general model to create four synthetic images that are suggestive of forms related to each of Semper's four elements. In crafting these images, students again make use of Artbreeder's ability to specify and edit the "genes" of an image, which allows for adjustments to be made in terms of the intensity of influence of imagistic archetypes.

Next, following the four archtypical images generated by Artbreeder, students compose collections of textured 3d models in Rhino and Blender to serve as the basis of corresponding Sketch2Pix "brushes". Similar to the "deepfake" architectural models, these digital models must be authored in a manner distinct from traditional modes of architectural production. Due to the nature of the data extraction and training processes, the specific geometries and overall forms found in these models is holds less impact on the behavior of the resulting brush than imagistic features, such as textures, colors, and small-scale formal relationships.

Here we train four related drawing assistants,
    based on Semper's four elements.

As in previous propositions, at the end of this four-week period, students present an early-stage design proposal. Here, the proposal is expressed in exploded axonometric.

This was accomplished 
by composing collections 
of textured 3d models 
that serve as the basis
of corresponding Sketch2Pix "brushes". 

These models were calibrated to operate in axonometric.

First, as demonstrated by those projects that rely on the gene-mixing and gene-editing capabilities of Artbreeder, we suggest that the epistemic action of latent space exploration may be different from that of known forms of design space exploration (Woodbury and Burrow, 2006). We suggest this difference in that imagistic archetypes that serve as the "genes" of latent space do not result from a user-defined set of parameters, but, in the case of unsupervised learning models, are "discovered" during training. Further research is needed to distinguish the ways in which latent space exploration may differently support design cognition.

First, 
as demonstrated by those projects 
that rely on 
    the gene-mixing 
    and gene-editing  
capabilities of Artbreeder, 
we suggest that the epistemic action 
of latent space exploration 
may be different 
from that of known forms 
of design space exploration. 

We observe that the imagistic archetypes 
    that serve as the "genes" of latent space 
are not the result of a user-defined set of parameters, 
rather, these are "discovered" during training. 

Next, we observe that an ML Generative model functions as a novel and effective form of early-design "provocateur", in the tradition of Eno, Schmidt, and Boulez. While similarly functioning to serve as a quasi-random "po" deliberately used to facilitate creative thinking, it is notable that the forms and images conjured by ML processes are not random, but rather are associations drawn from a specific body of experience (the training dataset) that may or may not be apparent to a user. This feature suggests a novel form of "guided" provocation, in which an author consciously selects the predilections of their drawing assistant to be provoked in an intentional way in relation to the particularities of a design problem.

Next, 
we observe that 
the tools considered here
are able to function as 
a novel and effective form 
of early-design "provocateur".

It is notable that 
the forms and images conjured by ML processes 
are not random, 
but rather are 
**associations drawn from a specific body of experience** 
    (the training dataset) 
that may or may not be apparent to a user. 

This feature suggests 
a novel form of **"guided" provocation**, 
in which an author 
consciously selects the predilections 
of their drawing assistant 
to be provoked in an intentional way 
in relation to the particularities 
of a design problem.

Finally, we highlight a new form of subjectivity offered by machine-augmented design. In the of training of Pix2Pix "brushes", including the critical step of crafting data sets for training, we find a new authorial position that should not be overlooked by designers engaging with this media. In stark contrast to other modes of computational authorship, such as parametric modeling, design action expressed through the defining of a training dataset is curatorial more than logistic or compositional. It is an action that may be regarded as uncomfortably indirect by designers new to the media: suggestive of imagistic traits more than deterministic of formal or geometric ones. This new locus of subjectivity holds broad ramifications for the future of computational design education, and suggests that connections must be strengthened with allied knowledge domains, such as data science and critical data studies.

    

Finally, 
I would highlight a new form of subjectivity 
offered by machine-augmented design. 

In the of training of Pix2Pix "brushes", 
we find a new authorial position 
that should not be overlooked by designers 
engaging with this media. 

In stark contrast 
to other modes of computational authorship, 
    such as parametric modeling, 
design action expressed 
through the defining of a training dataset 
is **curatorial** more than logistic or compositional. 

It is an action 
that may be regarded as uncomfortably indirect 
by designers new to the media: 
    suggestive of imagistic traits 
    more than deterministic 
    of formal 
    or geometric ones. 
    
This new locus of subjectivity 
holds broad ramifications 
for the future of computational design education, 
and, 
    from an institutional perspective,
suggests that connections must be strengthened 
with allied knowledge domains, 
    such as data science 
    and critical data studies.

Conclusion

The approach taken by the undergraduate design studio presented here suggests promising new modalities of design authorship that are significantly different from those supported by existing modes of computational design. More research is required to better understand these modalities, and to support the more widespread adoption of machine-augmented design tools in architectural practice.