Clarisse Sieckenius de Souza graduated as a Translator and Conference Interpreter from PUC-Rio, where she also obtained an M.A. in Portuguese and a Ph.D. in Computational Linguistics. In 1988 she joined the Informatics Department at PUC-Rio, where she started the Human-Computer Interaction area. Over the years she has been a visiting scholar and visiting professor in different universities and institutes, like Stanford (with Terry Winograd), University of Waterloo (with Tom Carey), and UMBC (with Jenny Preece).

In 1996 she founded SERG (the Semiotic Engineering Research Group), which she headed till 2003. Among the 13 M.Sc. and 16 Ph.D. students that she has (co-)supervised, more than a half are faculty in various Brazilian universities, and nearly one third work in the industry.

In 2005 she published her first book, The Semiotic Engineering of Human-Computer Interaction (The MIT Press). Her second book, Semiotic Engineering Methods for Scientific Research in HCI, co-authored by Carla Leitão, was published in 2009, in Morgan & Claypool's Synthesis Lectures Series.

...................................................................................

Interview Conducted by Shawn De Freitas
Semiotic Engineering was originally proposed as a semiotic approach to designing user interface languages. Over the years, with research done at the Department of Informatics of the Pontifical Catholic University of Rio de Janeiro, it evolved into a semiotic theory of human-computer interaction (HCI). Today we ask Clarisse De Souza to share some insights on her ground breaking work in Semiotic Engineering.
 

Use8> Semiotic thinking has long been featured in the literature of culture, communications and media studies. In fact, others (not comprehensively) have attempted to link a semiotic framework to the HCI discipline. What would you say is the major contribution of “semiotic engineering”?

De Souza> Semiotic Engineering has reaped the benefit of all previous work bringing a semiotic perspective to HCI. In particular, it has been influenced by the pioneering ideas of Peter Bøgh Andersen and Mihai Nadin. In my view the major contribution of our work is to have moved from a semiotic analysis of HCI to a semiotic theory of HCI. The difference is subtle, but important. A semiotic analysis of HCI uses semiotic methods, concepts and ontology to produce new knowledge about certain instances or types of HCI objects and phenomena. The outcome is usually a series of fragmented insights that are valuable per se but cannot be used to produce comprehensive accounts of what human-computer interaction is, or to generate and evaluate new scientific knowledge in HCI  (as opposed to new knowledge in Semiotics, for instance).

A semiotic theory of HCI is founded in Semiotics but has its own system of methods, concepts and ontology. It can be used to analyze, inform and develop HCI as a field. It is specifically constructed to support knowledge discovery as a process, rather than to produce knowledge instances supported by theories and methods borrowed from other disciplines.

Our two books about Semiotic Engineering (Semiotic Engineering of Human-Computer Interaction [2005] and Semiotic Engineering Methods for Scientific Research in HCI [2009]) present a comprehensive theory of HCI that can be used and developed independently of existing semiotic theories. In other words, Semiotic Engineering is not an application or a theory of Semiotics – it’s a theory of HCI.



Use8> Often times, great insight and inspiration come out of necessity. I think semiotic engineering is both insightful and inspiring. Was semiotic engineering born out of any necessity?

De Souza> Yes, possibly from a very peculiar kind of necessity. I am a Linguist by training, and although I have been working in a Computer Science department for more than 20 years I have never taken a single course in Computer Science. The shift from Linguistics to CS was made through Computational Linguistics and Artificial Intelligence, and an important tool in my survival kit was a set of foundational theories shared by CS and Linguistics.

In the early nineties I began to work with colleagues that specialized in Computer Graphics and Multimedia Interfaces, so the Chomskyan theories I was using to work with Natural Language Processing, mostly, were too abstract to help me frame, understand, learn and contribute to develop the graphical user interfaces that my colleagues wanted to develop. So, I added semiotic theories to the survival kit. First came Umberto Eco’s, closer to the philosophy of language, and then Charles S. Peirce’s, closer to logic and mathematics. In its origin Semiotic Engineering is, I guess, the expression of a non CS native’s sense making about computers and computing.


Use8> At the heart of “semiotic engineering” are communication, interpretation and understanding - which ultimately lead to some satisfaction and enjoyment for the user. These are also things central to storytelling cultures. I can envision the designer as a sort of mime artist, before an attentive audience, acting out a story through body motions, without the use of speech. Is the role of the designer in SE storyteller? Is semiotic engineering storytelling?


De Souza > Yes, you are right. The designer is a storyteller. He tells the story of his design vision. However, unlike most storytellers, he cannot use his body or speech to reach the audience. He can only use his mind and his ability to express what he has in mind through interactive computer languages and devices.

          Use8 >Are you yourself a story teller?
          De Souza > At heart, I suppose I am:) 


Use8 > One of the huge claims of SE is that it expands and also compliments existing UCD approaches. UCD and other predictive research methodologies are still prevalent and predominant amount HCI communities; can you comment on the adoption of SE thinking?


De Souza > In our two books, but most especially in our latest one (de Souza and Leitão, 2009), we try to show the limits of theoretical compatibility, so to speak, between Semiotic Engineering and other theories (cognitive or other). The main point is that SE is not a predictive theory (in accordance with its semiotic foundations) and thus cannot be used for predictive purposes. We have been deeply influenced by Don Schön’s perspective on design, the idea that design problems are unique and that what designers therefore need is an epistemology of practice. 


Semiotic Engineering can contribute extensively to an epistemology of practice. Our models and methods are called epistemic tools, that is, tools that help researchers and professionals name and frame (Schön’s terms) the elements of the design problem space and proceed to generate and evaluate alternative unique solutions.  This is possibly the main attraction of Semiotic Engineering, it’s been designed to support innovative thinking.


Use8 > The expanded possibilities of coupling Semiotic Theory and Engineering provides designers with a mental framework that allows them to view their role in the design process differently. However, designers often seek practical approaches to solving problem. Can semiotic engineering really provide designers with practical approaches to enable them to encode their design vision?


De Souza > When I first named our approach I was thinking about the literal meaning of engineering, the act of laying out or constructing something. So, the linguist in me was trying to analyze and explain the activity of laying out or constructing semiotic objects of computational nature. Unlike physical objects, semiotic objects are not (so extensively) predictable, in the sense that the rules governing their structure and behavior are more cultural than natural. And ‘cultural laws’, if they exist at all, are of a completely different nature than ‘natural laws’. So what Semiotic Engineering can offer is a set of methods, models and concepts to generate design knowledge that can be used in professional practice as well as in scientific research. It helps the designers become aware of the cultural resources and processes that they can use (in conventional or innovative ways) to get their message across to users. This is a clear demonstration of how much we have been influenced by Don Schön’s perspective on design.  It’s valuable, I think, for reflective practitioners.


Use8 > Before I ever touched the Apple iPhone I had assimilated some understanding of how its new interaction model might work. I guess I was conditioned by advertising, stories from my friends and media about the device. I constructed a mental model for the iPhone (based on no real experience) which I referenced on my first interaction with the device. This engagement did not happen at the moment of interaction with the system image, it happened before ...  is this communication also the domain of the designer? How does the user mental model factor in SE framework?


De Souza > In all intentional communication processes the speakers build a model of the targeted listeners and try to express themselves in such a way that the listeners will react in accordance with their intent. The common ground for communication is, of course, culture. Your engagement with the iPhone began in culture before it began in practice. And the ‘semiotic engineering’ of interactive systems begins exactly with the exploration of signs that belong to the culture of the users that designers aim to communicate with. In other words, culture, the shared collection of signs pertaining to the targeted audience  mental models, provides the semiotic sources for the semiotic engineering of interactive systems and devices.


Use8 > Design is increasingly becoming a multidisciplinary process that involves many stakeholders, each with competing needs that somehow must be negotiated.  The design vision in this sense isn`t purely the intent of one visionary designer or in fact those that are closest to engineering the solution. The design vision becomes a compromise that absorbs other requirements like marketing and business needs. Does semiotic engineer provide any practical insight for the multidisciplinary nature of design teams?


De Souza > One of the greatest advantages of semiotic engineering compared to other theories in HCI is that designers play a 1st person role in the process. They also have to engage in human-computer interaction because they need to integrate and communicate, in a single interactive artifact (ie. a single speaker that will communicate with users), the competing aims and messages that you are talking about. So, by trying to help designers explore, construct and adapt various kinds of computer-mediated communication strategies we are helping them analyze, negotiate, and eventually resolve the conflicts and complexity of collective discourse that emerges in design teams. Theories that are ‘user-centered’ in a very strict sense cannot help the designers attend to their own needs as they engage in computer-mediated communication with users.



Use8 > Is there any interesting contemporary work in the world of HCI or beyond that you think sheds some light or informs some of the inquiries and problems you were investigating in your own work?


I learn from every work I read. So, all of HCI work, contemporary or not, somehow fits in this category. As a result of circumstances, in the sequel of my latest book, with Carla Leitão, I began to look at programming languages and software development tools more closely. We did an extensive evaluation of Audacity, a free digital audio editor, and had the privilege to interview some of its developers for purposes of triangulating the findings achieved with semiotic engineering methods. During this interview we saw that some of the problems faced by end users may in fact originate in the problems experienced by developers  as they interact with various software development tools and resources. So, I am now trying to learn more about these tools, resources and practices, so that I can try to understand the message that developers are getting from their construction tools. I’d like to see how these relate to the messages that users are getting from the constructed systems. We called this new object of investigation the recursive metacommunication structure of interactive systems. It lies at the intersection of HCI, Software Engineering and Programming Languages.



Use8 > You have published extensively, written a book and founded the centre of Semiotic Engineering Research Group. This has tantamount to several years of dedicated research work ... how has your views changed and what is currently leading your interests for future work?


De Souza > Part of the answer has already been given above.  Over the years I have learned more about computing, and I like the idea of applying Semiotic Engineering to look inside  CS and not just at the interface with its ultimate social destination. On the other hand, I have also been led to explore the cultural rooting of all signs we compute with and compute upon. So, my interest in the cultural aspects of HCI has grown extensively. Together, learning more about software engineering aspects and learning more about culture provide an interesting direction for research – exploring ICT production and use as two sides of a continuous social communication phenomenon.