Motivation & Related Work

The intersection 
    of recent advances in machine learning 
    and architectural design tools 
remains a largely undefined territory. 

While it is clear that AI-based design tools 
    present fresh challenges 
    to the way we understand and use software 
    in the service of designing and building, 
the nature of these challenges, 
    and the character of the related opportunities, 
are still uncertain. 

This project represents one step 
toward demonstrating an opportunity 
that we have identified in brining ML to bear 
on better supporting modes of reasoning 
central to creative design. 

While, broadly speaking, 
existing paradigms for design tools 
have been shown to effectively support 
    deductive reasoning, 
    
we assert that 
a new paradigm 
based on machine learning 
will be uniquely suited 
to address heretofore unsupported modes 
of reasoning about design, 
    such as imagistic and inductive thinking.

This difference is clear 
from the intrinsic mechanisms 
of the tools involved: 

while 3d modeling proceeds 
    by composing geometric operations in sequence, 
    
and parametric modeling proceeds 
    by assembling elaborate chains of logic, 
    
machine learning models proceed 
    by matching patterns drawn 
    from prior data. 

Just as we would expect an author 
    to reason differently 
    when wielding a tool of induction 
    as opposed to a tool of deduction 
        
        a charcoal pencil as opposed to a calculator
        
we would similarly expect 
    entirely new cultures of design 
    to emerge in response 
    to paradigm shifts 
    in the nature of software. 

Our project anticipates just this.

The tool described in this project is thoroughly inductive. 

This is clear 
insofar as we first invite a designer 
to define an indexical relationship 
between a hand-drawn mark 
and the qualities of an image 
it is mapped to, 

and only then 
to find the "reason" 
for these images 
in the composition of a drawing. 

This is how the artist Scott Eaton works
as we see in this animation 

So, in this way, 
the project seeks to anticpiate 
new cultures of drawing practice 
that will arise in relation 
to this tool 
    and to others that take a similar approach.
    
from this, we hope to uncover 
new modalities of authorship 
latent in this still-emerging paradigm 
of computer-assisted design.

Terms

In the parlance established by the project, 
a trained neural network 
that is ready for use in sketching 
    is termed a "brush", 

    
The activation of a brush 
    to perform an image translation 
    is alternatively termed 
        a "transaction" 
        or an "inference". 

Central Mechanisms

Embedded in a brush 
is the central mechanism 
of the augmented drawing activity. 

Envisioned as a conversation, 
this activity is expressed 
as a graphic "call and response". 

First, 
a human author composes a drawing 
that may or may not adhere 
to some anticipated graphic convention. 

Next, this "call" image is passed 
to a neural net 
trained to offer a "response" image, 
which is passed back to the sketching environment 
to be displayed to the author. 

If the results are not satisfactory, 
this process may be repeated 
by modifying the original drawing, 
or the author may choose 
to move on to a new portion of the sketch. 

Due to certain technical limitations, 
the current system is constrained 
to transacting with images 
smaller than the typical size 
of most architectural sketches. 

For this reason, 
each call-and-response transaction 
typically represents 
only a portion or segment 
of a larger drawing. 

To mitigate this limitation, 
mechanisms are provided 
to allow for the tiling and layering 
of multiple response images 
in a single drawing. 

Thank You to the Students at UC Berkeley

We acknowledge that this research is closely related to an upper-division undergraduate research studio offered at UC Berkeley in the Spring of 2020. This studio was instructed by the author, and attended by an inspiring group of sixteen students. We would like to thank the students of this course for the generous willingness to participate in the case study described here. This resilient group of young designers faced difficult circumstances with grace, and adopted unfamiliar methods with enthusiasm.

• BARRIO Daniel

• CARRASCO Robert

• DEY Sarah

• DOERSCHLAG Nicholas

• DOVE Siri

• GOLESTANI Payam

• HO Garrick

• IMOUKHUEDE Uade

• JAIN Nehal

• LI Can

• NEHORAYAN Gabi

• NGUYEN Tina

• PHAN Andre

• TANG TJ

• WANG Rose

• ZHAO Leo

This research is closely related 
to an upper-division undergraduate research studio 
offered at UC Berkeley 
in the Spring of 2020.

Establishment of Training Data

Here, 
the central operation of a brush is established, 
as expressed by the mapping 
between a graphic "call" provided by a human author 
and the desired image "response" of the neural network. 

Because a data set of substantial size 
is required for training, 
pipelines are here developed 
for the establishing of training data 
using texture-mapped 3d models.

Training and Validation of a Neural Network

Because most novice designers 
lack direct access to computing resources and expertise
necessary to complete this training 
in a reasonable time, 
we provide guidance and resources 
for completing this step 
in a cloud computing environment. 

Deploying a Trained Model

While in the recent past, 
such a step would present considerable difficulty 
for a novice user, 
the recent introduction 
of model hosting and distribution services 
have significantly eased this burden 
and lowered barriers of entry. 

Sketch2Pix is registered with one such service: 
    RunwayML 
    
Given this service, 
hosing a model is a relatively trivial process 
from the user's perspective.

Interactive Sketching

The details of this collaborative drawing activity 
are largely guided 
by the features of the interface developed 
for the given graphic sketching environment. 

A large amount of development time 
was spent noodling 
on how to get Adobe Photoshop 
    to interact with a hosted model
in a way that is supportive of architectural sketching.

I would just note here that 
    how to best structure 
    an successful interaction 
    with an AI assistant 
    is a current topic 
    in HCI research. 


From this perspective, 
    we strongly value the iterative development 
    of small-scale graphics 
    in the service of larger compositions. 

This is because, 
    in our view, 
ease of iteration is essential 
to facilitating the feeling 
of a "conversation" 
with an AI partner. 

- - - - - 

One may see this value reflected in the decision to integrate alpha-channel generation into the Sketch2Pix model itself, as transparency is necessary to effective layering. 

We also see it in the way we've handled the Photoshop layer structure, and in the protocols for returning "response" images. 

We encourage our users to, at the start of a collaborative sketching session, establish two layer groups in Photoshop: one that will contain the hand-drawn "call" images, and a second that will house those "response" images returned by the brush. 

Rather than replacing the user-drawn image with the computer-generated one at each transaction, our application returns generated image to its own Photoshop layer, and links the source layer to the generated layer. 

Then, if the user modifies their hand-drawing, the generated image may be updated rather than duplicated. This pattern of user interaction is intended to encourage the sort of iterative development that is commonly practiced in design drawing. 

In this talk 
I've presented the mechanisms 
of a tool that supports 
**machine-augmented architectural sketching**

an activity that we see 
as a special case 
of **inductive design thinking**.


First is a reminder 
of the central role 
of iteration in design, 
and of the small discoveries we made
on how to instrumentalize iteration 
in the user interface of design tools. 

We expect that ease of iteration 
will become even more essential 
in facilitating the feeling 
of a "conversation" with an AI partner.


Second is an indication 
of the challenges that lie ahead 
int the shift from geometric / logical modeling activities 
to those required by training an ML model. 

We see such a challenge 
in the role of the 3d models 
produced in the service of training. 

Here, users must shift their thinking 
away from models playing a directly representational role, 
    where the model stands in 
    for the building we envision. 
    
Instead, these models play a loosely indexical role, 
where the model defines 
the relationships and patterns 
to be learned by a neural network, 
which prefers information 
such as colors and patterns 
over forms and organizations 

We expect to find many such differences 
between traditional forms of computer-aided design, 
and emerging forms based on ML.

Thank you.