What is Bret Victor Trying To Do?

Update 2021-04-03: Partially rewritten, as this is a popular post

Introduction

Not every UI enthusiast can reach out to signal processing and circuit design — truly complex systems — for their real world examples. The engineer and designer Bret Victor stands apart among the leading makers and thinkers in the fields of user interface design, software tools, information systems and computational media.

Victor has spent his life building things. He has worked on instruments and tools for a variety of creative workflows. He has built digital gadgets of all kinds, and has pushed the industry forward by showing what modern personal computing could be like. Victor has built award-winning applications and influential demos that continue to inspire software designers and engineers everywhere.

Through his celebrated essays and talks, Victor has influenced a whole generation of makers, guiding them to ask bigger questions about the ways in which we interact with digital tools and complex systems — and other people.

In his research Victor draws on the rich, yet practically forgotten history of Silicon Valley pioneers and other visionaries. We have lost our way, the argument goes. Back then, the ideas were big and the technology was basic. Today, the tech is sophisticated by comparison, but the ideas are really small. Victor invites us to think deeply about the intent behind the designs of past engineers, when we imagine what is possible tomorrow. At the same time we shouldn't settle for emulating old ways of working, when we think about computational tools and a digital future.

Victor is an unusual sort of engineer, one blessed with a romantic mindset and a certain humanistic sentimentality. Victor maintains a softer outlook that allows him to see beyond the immediate concern. He wants technology to be there to lift all of us, together.

In our rush for the future, we let technology run over the human and the humane. We increased our productivity by supercharging the old tools, the old representations and the old paradigms, instead of inventing new ones that are native to the new medium. To entertain ourselves, we settled for the iconic and the facile instead of embracing the dynamic capabilities of our new toys. Today, we have the power to simulate arbitrarily complex dynamic behaviour, but we still think in static symbols and use linear tools.

The mission is to help people through the development of new tools, new artefacts, and new ways of seeing. Victor is interested in making systems intelligible, and complex behaviour visible and useful. In his vision our tools enable us to directly manipulate dynamic representations and to gain an ever deeper understanding. The world is so complex that we must move beyond the limited ways of thinking that the paper and pencil mindset and metaphor enable.

Victor wants to take us into a brighter future where our media and our modes of communication are expressive and dynamic and in service of a better informed discourse. By upgrading our tools and our techniques, perhaps we can bring forth new thoughts, and new solutions to the most pressing problems we face today.

Victor's path has taken him from gadgets and shiny things all the way to a new dynamic medium. On this journey he has held on to a set of ideals and principles about the role of technology in our lives. The whole of human being is sacred, and the tools we use must adapt to us — not the other way around. We must have immediate feedback. We must have direct access to the fruits of our creative process. We need to see, deeply, when we try to understand.

In the end, our technology primarily serves a social function. We share our thoughts with others, either directly or in a mediated fashion. We study and process our thoughts to discover new things to use and share. Bret Victor is working towards a future where our tools and representations are as dynamic as our mercurial minds, and where we can better understand our environment and one another.

This biographical essay is an attempt to trace out Victor's trajectory from his early days in East Bay to his present days in East Bay. On the way we'll encounter college humour, keyboards, shiny apps and beautiful products, tools for creatives and scientists, yearnings and rants, and ultimately a brilliant vision of a future worth fighting for.

Origins

Early years, college, and the cultural milieu of the golden state. Details drawn primarily from Bret's homepage as well as this Postlight: Track Changes interview.

A Silicon Valley kid

Bret Victor (born 1977) grew up in California, in East Bay, in the heartlands of Silicon Valley. The Bay Area and California in general have been his home ever since.

In my view, Victor is part of a transitional tech generation. He was both a little late to have experienced both sides of the personal computing revolution, and a little too early to truly belong in the saturated social media culture we live in today. Victor thinks about the future of technology by drawing on the insights of past generations. Those early dreams were never fully realised, or even explored. In other words Victor is early Web, but more Xanadu at heart.

For context, some visionaries with Victor / Silicon Valley connections (all men, alas):

Not that much is available on Victor's youth, except a throwaway comment on how he started programming at an early age, around seven or eight. No data on his childhood experiences or ambitions or school days, but I have no difficulty imagining a typical Silicon Valley kid, with an interest in electronics and computer systems. Making music was another early Victor passion.

A portrait of an era:

Bret started playing the piano at six, but only got into it more when he discovered fakebooks (aka lead sheets). This material allowed him to focus on playing and enabled improvisation and a "feel" for chords. Rather than perfecting technique in tedious practice, Victor has always been more drawn to making his own music. Somewhat reluctantly, it seems, he left behind the grand piano and picked up the more portable electronic keyboards — a switch that proved professionally meaningful later on.

"Faking allowed me to sit down and play a song right away."

Victor was probably as awkward as any nerd in school. From a rather detailed description of the health-conscious Victor household pantry, I speculate that his childhood home was rather strict and orderly. I suspect that Victor, almost certainly an only child, felt much more free out in the world, doing things with other people. The social dimension has always been essential to his being and doing.

On his teenage years, a later website entry reads: "All I can remember about junior high school is an endless string of tormenters, real and imagined, and a crushingly miserable sense of isolation." All this followed by an appropriately self-aware and self-deprecating note about this being a fairly standard high school experience. Victor had his ups and down: he had a decent track career, but pulled his hamstrings; he found a passion for the piano, but suffered carpal tunnel pains; he had a pretty girlfriend, but he messed it up; and so on.

I think the key takeaway from his youth is that Victor must have been intrigued by the magical properties and possibilities of technology from early on. He first saw this magic as something confined to the display or the screen, but then discovered that through electronics tinkering, he could bring his magic with him for others to see outside his cave. Build stuff and show it to other people: Victor grew up as a maker.

College

The official Bret Victor trail begins in southern California, which Victor has always liked less than the northern end.

Victor began his studies in 1995 studying electrical engineering at Caltech, in Pasadena. He quickly found the right challenges and the right people. Victor thrived in the Caltech undergraduate programme.

"Caltech was the best four years of my life. I was introduced to brilliant, fascinating, inspiring people, people who thought and played as I did, people who made me feel at home."

The 1990s Caltech is more than a generation after Feynman, but in my mind Caltech was and is very much a school for playful geniuses. Victor kind of agrees, describing the Caltech culture being "very much prank oriented", where "impressing people with your crazy, little projects" was how you found your tribe.

"Basically, what I did as an undergrad, was I just built gadgets."

Eventually Victor had some kind of an Enlightenment moment as he learned his way through Caltech's "broad but shallow curriculum" that left him "a jack of all trades, and master of jack". After four years he was confident that he had discovered his true calling: "the profound, soul-completing Joy of creating Boxes With Lights And Buttons."

Victor graduated with a BS in 1999, with honours, and with what is undoubtedly a first-rate engineering education.

"I was taught the Way Of The Electron."

After Caltech, Victor joined the EECS Ph.D. programme at UC Berkeley. Lacking a strong purpose and without a sharp research focus, Victor quickly felt misplaced. He couldn't find any research area that really resonated with him and he wasn't producing anything that he was particularly happy with.

"Grad school basically consisted of me trying various areas of EE and hating each one, and thus was useful primarily as a process of elimination."

Victor wrote a well-received paper, Bus Encoding to Prevent Crosstalk Delay (ICCAD 2001), on "self-shielding codes", a novel way to circumvent a circuit design issue through data encoding. The paper was a great success, but at this point Victor had already realised that the academic path would not be carrying him much further.

Compared to his golden Caltech years, life at Berkeley was not treating him well. After two years in the programme Victor decided to bail out early: he extended his research paper into a full Master's thesis and called it done. "I spent much of the two years directionless, advisorless, adviceless, deskless, friendless, and generally rather miserable," Victor wrote later, "I was the Ronin of the EECS department."

"[In grad school] suddenly EE was not about building gadgets, it was about how do you push Moore's law? How do you take a chip that goes at 1 GHz and make it go 1.1 GHz? And that ... didn't appeal to me at all."

It's a difficult place for a learned person to be in, when you've trained for years and years in a discipline, have found your Path, only to discover that deep down you don't really care about the frontier, the research questions that the establishment is interested in. When all the novelty and fun of student life grows thin, and the beautiful ideas that originally got you excited in your field make way for the stark reality of the post-graduate grind, your hard-earned mastery can leave you empty and drifting.

In the summer of 2001, with his consolation diploma in hand, Victor began a gap year of sorts. Instead of travelling in Europe, Victor appears to have spent his days mostly drifting about the Bay Area, brooding, playing in a band and hating it, recovering from a relationship that didn't work out, and being generally miserable.

Most pressingly his post-graduation job search was not going particularly well. For many months Victor was unable to find the perfect job that would simultaneously A) match his skills, B) match his deeply Californian ideals, and C) allow him to pursue some form of independent research.

Despite his incompatibility with academic EE, Victor recognised his aptitude and indeed love for designing electronic gadgetry, and knew this is something he could do while he was figuring things out. He had imagined doing "Boxes with lights and buttons" as a grown up all throughout his college years, so now it was just a question of finding the right kind of gadget gig.

"WILL DESIGN EMBEDDED SYSTEMS FOR FOOD"
— Bret, with his cardboard at the freeway exit

While Silicon Valley was still reeling from the dotcom crash, Victor's skills and credentials were surely plenty to land him any number of solid jobs. However, Victor had more ambition and a burning desire to work on something more meaningful.

"I just have this feeling that instead of making toys for rich kids, or devising ways to make computers go 5% faster, I could somehow somehow somehow be using my skills to save lives. Or significantly improve the global quality of life. Or something big and noble and hopelessly idealistic like that."

In September of that year, just like everyone else, Victor spent one Tuesday sat in front of a TV helplessly watching and re-watching a plane fly into a building. He wanted to do something, he wanted there to be robots to help the rescuers. And if he cannot be building rescue robots for the disasters of tomorrow, if that's beyond his engineering reach, at least he might be able to make something to help out the people who do the rescuing. Victor wanted to do engineering with a purpose.

"The advancement of technology is a noble and worthwhile goal. But not for me. I don't want to advance technology. I want to help people."

It's difficult to judge the impact of singular events on peoples' lives, but on the other hand, I feel like a pivotal event like 9/11 happening exactly at a time when you are thinking about the nature of meaningful work cannot NOT leave a mark of some kind.

Victor did not get into rescue robotics, and indeed he was compelled to shelve his idealism at least for a few more years. To discover what he really wanted to do with his life, he had to first get into building toys for rich kids.

A man of culture

"There is no medium better suited to individual sharing than the internet."

College is a time of self-discovery and experimentation, of trying different things and in the process figuring out who you really are. Victor's experiences at Caltech and Berkeley — his heaven and hell, respectively — had a great influence on his future trajectory. Many important pieces of the Bret Victor puzzle found their shape during these formative years under the California sun.

(Note that Caltech usually doesn't re-accept undergrads as grad students. I'm certain Victor would have wanted to continue at Caltech if that was at all an option.)

The two personas I want to highlight here are Bret the Maker, and Bret the Romantic. They are of course happily intertwined in the same person, but I think the deconstruction is useful, as the parts help with understanding the whole. It's difficult to appreciate what Victor is working on, and has been working on through all the chapters of his life, without an understanding of the culture and practice that underlies his work. As with all of us, Victor's work is shaped by his experiences, and his experiences are shaped by his principles.

"I am what I create. What else is there?"
— a tagline from Victor's grad school era website

Victor has always been a maker and a tinkerer. The digital artefacts of Victor's college days and subsequent years are all available on his home page. The original college websites of yore are long gone, the URLs do not resolve, but Victor still hosts his old content under "Art" and "Archaeology" on his current site. It would be easy to leave all this behind, but Victor has chosen to leave a somewhat more permanent trail.

This sentiment of history and permanence somehow feels consistent with Victor's drive to not let the history of information technology be forgotten, as evident in his twitter feed and much of his writing. Victor's work and research has a strong notion of time and timeline — perhaps fitting for a circuit designer.

Victor's websites and other things form a personal slice of Internet history: his own Internet story. They provide a fascinating insight into the mind of a young man, and at the same time give a taste for the early days of the Web and what it might have been like to be a part of the first generation of authors in this new medium.

Styles change, but some things stay the same. And the passion to share with others is a constant. Some of Victor's material can only be described as "college humour", with all the variety and that singular aesthetic of the early Web era, but equally there are more thoughtful pieces scattered in the mix.

Some treasures from the archaeological dig:

Together these artefacts paint a picture of an engineering infused social scene with all its ups and downs, a life shared with fellow nerds, artists and makers. It's a veritable voyage full of expression, feeling, and little projects of all kinds — a tale told through photos, games, songs, software, poems, gadgets, websites, flyers, T-shirt designs, and more. And the making did not stop at graduation, far from it: all this making was the foundation for all of Victor's future work. Variety helps with finding a direction.

"Art — The management assumes no responsibility."
— section head from Victor's current website

In college Victor encountered a whole range of people and ideas, and came to fully embrace the grad student lifestyle — perhaps he has never really given it up. During his Berkeley years, Victor lived in a big house with many friends, a commune of sorts, which seems to have suited him well. On balance he was a rather lonesome fellow, particularly when not in a relationship.

In my view the Californian community ethos, the hacker culture, the Whole Earth Catalogue crowd and its seedlings, all this history enters the picture at Caltech (if not earlier). As a Silicon Valley native, Victor is very much part of that tradition. He surely learned about and immersed himself in that tradition, and this shows both in the principled stance he took in his early professional career, as well as in what he is doing today. Victor is driven to share his research, work in the public domain, because that is fundamentally who he is.

(Recently, sadly, Victor has been more quiet on the publication front. Granted, given his current long-term mission, this is not that surprising. At same time, something has changed with the web itself. Sharing has changed.)

This brings us to Bret the Romantic. Many a troubled young man finds his social situation desperately lacking, but I feel in his extraordinary extracurricular output Victor shows that he is perhaps slightly more sentimental and heartfelt and feeling than your average engineer. This is important, because I believe this romantic view of the world is the true source of his insightful views about technology. For all his formal training and rigorous scientific and technical ability, Victor sees the world fundamentally as a human endeavour, a humane playground.

In college Victor treasured giving gifts that he had made himself. He created diverse media to mark occasions. He played and composed and improvised music to express the nonverbal, at four in the morning or after a break-up. He wrote funny things and serious things, clever things. He created visual "art" from his lonesome moments, impressed his dates with gadgets of his own making. He joined a band "just for the experience". He took long walks, for brooding or just airing his thoughts. He admired the greats of decades past. He was a young engineer who felt deeply.

Bret Victor cared about the people around him and of strangers beyond his sphere — at least as much as he cared for technology and Boxes with Lights and Buttons. After college and the post-graduate grind he desperately wanted to work on something meaningful, do engineering with a higher purpose, but it turned out that he first had to gain some professional experience.

Alesis, Apple, and information design

"Pete thinks I should go down to Alesis and make audio electronics."

After grad school, Victor's first job was with a company called Alesis who make pro audio equipment. Victor joined the Los Angeles based company at a particularly turbulent time, possibly caught in the wake of the dotcom bubble.

"Alesis was an interesting company in that they had just gone through a bankruptcy and they had gone from a hundred people to eight," Victor described the situation in an interview. "The engineers were just kind of running the show."

As a result, Victor's job description was remarkably holistic. He was trusted with all aspects of product development from hardware and software design to box art and marketing copy. At Alesis, he worked on three keyboard products: the Ion, the Micron, and the Fusion.

For the Alesis Ion, a "virtual analog" keyboard, Victor created the sound synthesis engine. The product got great reviews and "sold well for a niche product". For the Fusion, he again created the sound engine, which received praise, but ultimately the product did not sell. Victor's view was that this was due to it being unpleasant to use, clunky, unwieldy.

The third product, The Micron, was Victor's baby. He was inspired to focus not on sound, which is what keyboards typically compete on, but on the user experience — still a fairly novel concept in early 2000s. Victor designed the compact Micron specifically for music making. He set out to make the keyboard fun to create with. And people liked it — it got stellar reviews and sold very well.

This was the genesis of Bret the Creative Tools Maker and Bret the UI Inventor. Victor saw that the keyboard product space was stuck in a stale tradition and he wanted to reinvent it. Fortunately, he happened to be in a sufficiently chaotic environment where he eventually was able to pursue his ideas. He took what people liked about the existing product, the Ion, and focused on the human side of the technology. He made the Micron extremely portable and added a bunch of conveniences, instead of obsessing over sound engineering.

With this experience under his belt, Victor headed back north and started a garage company of sorts with a friend. They started developing a new kind of instrument, envisioned a whole new radical approach to making music.

Around this time, maybe late 2004, someone pointed him at Alan Cooper's The Inmates Are Running the Asylum, a classic diatribe on people and software. Today, Victor doesn't think too highly of the book, but notes that it was "kind of exactly what I needed at the moment". He realised that the thing he was missing in grad school had a name: user interface design.

"Interface matters to me more than anything else, and it always has. [..] People change the world by using things. The focus must be on the 'using', not the 'thing'."

Victor dived in and read everything and anything related to UI and UX. He came across Tufte and got hugely into information design. He read about HCI, and even ventured far out to check out all kinds of visual design resources — blogs and magazines. He found his key texts, and in the process learned about various related fields that further gave him focus.

"The sorts of things I saw the capital-D Designers concerned with left me really cold."

In 2005, Victor made a Bay Area public transit widget, a trip planner for the desktop Mac. Originally, he built the widget as a joke for his fantasy keyboard company friend, but it ended up getting a lot of attention. The widget went on to win an Apple Design Award and got Victor in touch with people at Apple.

For the BART widget, Victor had created a UI prototyping playground where he was able to explore different user interface techniques. He developed this tooling and the theory behind it further, and then began to write a paper on interface design and implementation. This then ballooned into a full-on manifesto, Magic Ink: Information software and the graphical interface (2006). Magic Ink proved influential: the Internet started to take notice.

In August 2007, Victor took a job at Apple. He worked on experimental hardware platforms and input technologies, with the charge of playing around with new user interfaces and application concepts. He would build things, see what was possible with experimental devices, and then demo apps internally to inspire people who were building the end user products.

This research-product split is more or less the research model Douglas Engelbart advocates for in his 1962 Augmenting Human Intellect report: the A team plays around, dogfoods tech, stretches the realm of possibility, and then ultimately feeds technology to the B team, who build products. Indeed this is how every successful research lab has ever functioned.

"Steve wants a tablet." — response to Victor's query about a proto-iPad

Victor made many things at Apple. He developed what he calls "pervasively direct-manipulation interfaces", where "users move and gesture with meaningful objects, instead of verb buttons and other indirect controls". He developed new creative tools and new ways of learning information. His prototyping influenced products such as the Apple MacBook, the iPad, and the Apple Watch.

Victor stayed with Apple until 2010, having contributed directly to at least the iPad, the iMac, and the Apple Watch. These were productive, important years in the Bret Victor story, but our protagonist wasn't happy. Over time Victor grew frustrated with the realities of corporate research.

He was dissatisfied with the secrecy, with the fact that his inventions were locked down in an environment where only a dozen people could learn from what he was doing. The dilution of his vision started to get to him as well, and he began assembling ideas for his post-Apple future. He developed new interests that were not aligned with Apple's direction.

"The [apps, UI concepts] I cared deeply about mostly didn't catch on."

In Victor's eyes, Apple is in the consumer content device business, providing entertaining digital experiences. His own visions are somewhat more elevated. Victor wants to help people understand things more deeply, to enable people to create things they could not create before.

To regain his freedom, Victor had to leave Apple. And as the first order of business, he jumped on a train and wrote down all the things he had been bottling up inside him.

"Apple has a depository where you drop off your soul for the duration of your employment. I went to retrieve it, and they gave it back to me scuffed, bruised, deflated, but hopefully still in working order."

Research Hobo

Towards the end of his Apple tenure, Victor got involved in the design and implementation of an interactive e-book version of Al Gore's Our Choice, a call for climate change action. He later wrote a more personal piece on the subject, a kind of a "engineer's approach to climate change". In other words, Victor was slowly getting back around to his idealism.

It seems clear that at this point in his career Victor had mastered what he calls the Way of the Electron, the Way of the Algorithm, and the Way of the Interface — "the Yang of technology" and "the Yin of Design" — but had found all of these to be false paths. Victor needed some time off for reflection, for finding his own path.

"I had built up a set of things I wanted to think about that could not be thought at Apple."

Victor jumped on a train and travelled the US by rail, visiting research labs and universities around the country. He ended up spending many months of 2011 on the road, following a slow-burning desire to build powerful tools for scientists. He was on an anthropological quest of sorts, studying the practice of science.

He took three piles of ideas with him, resolved to figure out what each one was about. He spread out his clippings and memos and notes on big desks at "random public libraries" and tried to understand what he had assembled. What was the common denominator? What was the abstraction? What do these things add up to? What are the categories?

The three piles became three interconnected research streams: Dynamic Pictures, Explorable Explanations, and Kill Math. Victor assembled some of his old works and wrote new essays around these themes, and put all his thinking out on the web for others to study. These, much like his many excellent talks, each "paused the Internet" on release, and remain influential works to this day.

"Tools and techniques for reducing abstraction and indirection. Encouraging people to understand and create via exploration and intuition."

Dynamic Pictures

"Dynamic pictures will some day be the primary medium for visual art and visual explanations."

The Dynamic Pictures stream builds on the ideas explored in Magic Ink (2006), arguing for better tools for authoring dynamic pictures, interactive visual media elements.

On its surface DP is an argument for new tooling to enable artists to author dynamic artefacts. In his motivational essay (2011) Victor distinguishes dynamic pictures from static ones by the fact that dynamic pictures are input parameterisable: "A dynamic picture looks different in different scenarios."

Dynamic pictures, including user interfaces of all kinds, enable understanding and visual explanations. In order to create dynamic pictures, designers today are not self-reliant, as they cannot realise their own creations. This is because the tooling to create dynamic media isn't there yet.

The only way to create dynamic behaviour today is to use code, programming languages. And this canvas, while powerful, distances the artist from the making of art, the designer from the design. And this is not good enough.

The deeper, more general concern here is with indirection in the creative process. Artists, makers of all kinds, need immediate feedback, or risk missing out on creative insight and other opportunities and possibilities. And if the artists miss out, then so does all of culture and society.

To limit people to inappropriate, awkward representations and the indirect manipulation of their chosen medium is fundamentally flawed and broadly unacceptable — an inhumane use of technology.

Victor has been at this for a good while. Back in 2007, Victor released research language Substroke, exploring a functional transform metaphor in describing parameterisable dynamic behaviour.

Victor revisited these ideas later with tool-oriented talks such as Stop Drawing Dead Fish (2013) and Drawing Dynamic Visualizations (2013, see also the addendum).

Explorable Explanations

Explorable Explanations (2011) is Victor's take on active reading, and how active readership could be encouraged.

"People currently think of text as information to be consumed. I want text to be used as an environment to think in."

Fundamentally, Victor feels that in some sense the age old reading experience does not serve the modern reader. The complexity of the information, the material content, the reading context, and indeed the readers themselves have all changed over the years and decades and centuries. Regardless of what has changed the most, Victor feels that the static text form leaves something to be desired. Surely there is something to be done about the medium?

"Active readers ask questions."

With EE, Victor wants to encourage an active take on reading. The reader should be empowered to find answers to questions they form as they read, to explore alternatives and examples, to challenge the author's assumptions and trustworthiness. All this in pursuit of a deeper understanding and perhaps a deeper appreciation.

Victor has worked on several examples and prototypes of active reading. Ten Brighter Ideas (2010) is a prototype for a reactive document, where the reader is empowered to "play with the premise and assumptions of various claims". Victor took the statically stated information from a publication and gave it context and dynamics through basic interactive elements. The primary mechanism here is enhancing text with interactivity, dynamic behaviour.

"A spreadsheet without the spreadsheet."

The augmented text in Victor's Explorables examples makes use of Tangle (2011), his stand-alone JavaScript library for adding reactivity and dynamics to static web documents. (More details on the API reference page.)

Interactive things are inherently pleasing, we love to play with knobs and things, but Victor has a deeper point here with Explorable Explanations. In a sense, he is looking to change the conversation around a given text, by exposing not just the author's view, but the whole model they used for their argument. Would you have chosen different parameters? This adds to the transparency and honesty around reporting, and makes way for a more informed debate, beyond emotional appeal.

With Explorable Explanations, Victor shows us how an interactive feature in an example can help with understanding, discovery, and intuition about the behaviour of a system. Victor believes strongly in the power of multiple representations of the same thing, and wants to build support for seeing all of them at once.

"Interactivity itself is not the point."

In a great Explorable, the explorable part is tightly integrated with the explanation. The text still works as a commonplace static text, but for those readers who want to go deeper, there is a low-barrier way to take a closer look. It's not a huge context switch, nothing is loudly competing for your attention. The augmented text is behind a natural interaction. The dynamics just is there, available if you want it.

All of this dynamic behaviour is of course more work for the author, but with the right tooling, perhaps the burden is not too much to bear.

"[With explorables] the author holds up their end of the conversation."

But how about existing documents then? How could normal text be made more explorable? Victor proposes that contextual information would go a long way to help out a reader. He has considered a kind of a plugin model, where extra tooling around the static text in a browser could provide convenient access to online searching and its results, all without removing the reader from the reading context.

Another way to augment existing documents is to embed some light interactivity or extra visual depth into them. With Scientific Communication As Sequential Art (2011) Victor takes a few ideas from the comics playbook and reorganises a classic paper on network theory into an Explorable. The unobtrusive interaction and Victor's dynamic representations, an example serial, make a world of difference to an already fairly readable paper, highlighting the power of Victor's method.

The comics influence Victor wants to draw on here is not about superheroes or speech bubbles, but about powerful exposition. In Tom Oreb's Portrait of Ward Kimball (2007) Victor presents a "comicification" of an expert's analysis, in effect applying the comic mindset in a different context, where dynamics perhaps would not add that much.

"The comic form is about sequences of tightly-integrated words and pictures, together conveying a message more powerfully than the sum of their parts."

Overall, with Explorables, Victor invites us to move from a text-as-information mindset to a more humane way of thinking about information digestion. The emphasis is on treating text as an engaging environment, with room for both questioning and play.

(For a more recent example of the kind of thing Victor is talking about here, a modern Explorable of sorts, see for example this wonderful Gears demo by Bartosz Ciechanowski (2020).)

Kill Math

Note: I prefer maths, with the 's' at the end, but I'll stick to Victor's lead here.

In Kill Math (2011), Victor takes issue with the commonly used tools available for the practitioner of mathematics.

Why do we (still) model the world using abstractions invented for pencil and paper? Can we solve quantitative problems without manipulating symbols? Can we do better than blind symbol shunting, be liberated from the constraints of pencil and paper?

"Math needs a new interface."

Victor is NOT talking about reforming math education. In Victor's view the problem with math is much more fundamental: current mathematical notation and the methods around it are frankly unusable. We need to rethink the whole tool chain. Helping people more efficiently pick up the esoteric art of symbol manipulation is another concern altogether — trimming the branches versus striking at the √root.

Victor is yearning for more "concrete, visual, tangible, direct" representations that allow efficient manipulation of concepts and constructs, more humane quantitative problem solving. In his writing he gives many examples of paradigm shifts, where new inventions, like the line graph for data visualisation, completely revolutionised our ability to make sense of numbers.

"A person should not have to imagine the interpretation of abstract symbols."

As part of this research effort, Victor has generated a number of documents that outline his vision. All of them feature new ways of working with numbers and data, using human-friendly representations that leverage people's perception and intuition.

Many of these ideas make an appearance in his excellent talk Media for Thinking the Unthinkable (2013), which serves as a kind of manifesto for building new tools, for a whole new medium for expression and thought. After all, our representations are how we understand a system. We can push the representations of an stale medium only so far. For new thinking, a deeper understanding, we need to upgrade our tooling.

"Media are our thinking tools."

One example of a new kind of numeric tool is Victor's Scrubbing Calculator (2011), a small calculating utility in the spirit of Soulver. The idea is that a user can play around with the values of a free variable in the context of the application. Instead of abstracting things into a formula and solving it through symbol shunting, the user can "play around" with values, estimate and try something close enough and find the answer they need through experimentation. The math is embedded in the sentence.

In Interactive Exploration of a Dynamical System (2011), Victor gives us an example of a tool for studying a dynamic system, a prototype designed for a scientist or engineer. The key thing here is ubiquitous visualisation: everything is on display. The interactivity gives the user the power to study the example system, to see what effect its parameters have. All done intuitively through in-context manipulation. This is the opposite of looking at code and keying in "plot(..)" to see what changed.

Victor gives a comprehensive treatment of what working in a richer medium could look like in Ladder of Abstraction (2011). In this document, Victor presents the ladder as a technique for thinking explicitly at and about multiple levels of representation. Through working with this system in this medium, the designer can gain a better understanding of the problem, which then can lead to its satisfactory solution.

In the example, Victor walks us through systems of increasing complexity, building the problem toolset as we go, developing ideas and an intuition about the system. Iteration on a workable solution would then follow. More abstractly, Victor argues that the same pieces can be found in most real-world engineering problems: an independent variable, structural rules, and data that parameterises the rules.

In other words, for all the specifics of the example he is using, Victor is arguing for a general approach to building tooling for understanding dynamic systems. The ladder of abstraction is built on taking control of the parameters of the system, and then on building up and down in terms abstractions. A good systems tool allows its user to highlight the system's behaviour, to drill down to specific instances, and to generalise and discover high-level patterns.

"Our visualizations are not plots on paper. They are worlds that we move within, live within, learn from the inside out."

Finally, Simulation as a Practical Tool (2009) is an early precursor — a veritable tour de force — that offers another look at working with dynamical systems. Starting off from what Victor considers a weak example, he first solves the original problem using simulation, then extends the solution (and problem) into a more insightful direction and gives a contextual solution. From this, Victor identifies a follow-up question, and a delightful extension that yields a surprising result. And then turns the whole thing into a generative piece, bordering on art.

"A truly revolutionary simulation tool can beat symbolic methods at their own game."

Victor's idea with Kill Math was to assemble a collection of meaningful problems from different application areas and design means for solving each one, in line with his philosophy of humane, dynamic representations. With these, through comparative analysis with the methods available, he envisions that the techniques and design patterns of a more general framework will eventually emerge.

"The pocket calculator and the spreadsheet both made huge contributions in allowing people to do mathematical explorations without dealing with the crap. I'm hoping to take the next step." Fast Company piece on Kill Math (2011)

Towards a humane medium

After getting his main position papers out over 2011, Victor started to think about the next chapter. He took a hard look at all of his prior work and research, and began a process of synthesis that over the next couple of years culminated in his vision for "a humane medium for seeing, understanding, and creating systems".

Victor kept on writing essays, and his output turned somewhat more personal, perhaps. His call for better tools went from a banner and a statement to more of a heartfelt plea. This is the return of Bret the Idealist, the blossoming of the humane engineer here to help people.

Having completed his post-Apple, post-hoboing soul searching, Victor released what became one his most celebrated and influential talks, Inventing on Principle (2012). In this talk, Victor outlines the principle behind his research (so far), the thing that has driven him to create a whole library of tools and techniques for helping people bring their ideas into the world. His principle is something like the notion that creators, authors, readers and users all need immediate feedback. That bad representations and indirect ways of manipulation, manifest in our misuse of media, hinder understanding and creativity.

"I followed this guiding principle, and it guided me to what I had to do."

Victor gives a few examples of this principle of immediacy and powerful tools enabling new thoughts, but then shifts to talk about principles in general. Violations of his personal principle are not opportunities per se, they are fundamentally wrong things that need righting. Injustice, responsibility, moral wrong — these are all concerns Victor would like to see engineers and techies discuss more. He advocates that everyone should build a repository of experiences and take time to evaluate them, to help them figure out their own principle through reflection.

Inventing on Principle features prominently Larry Tesler and his powerful, concise principle: "No modes." Tesler died last week, 28 Feb 2020. Guardian had a decent obituary. Engelbart, also featured, died about a year after the talk, back in July 2013. Victor wrote about his towering influence in this remarkably sober piece.

Victor's principle is perhaps best demonstrated in his talk Stop Drawing Dead Fish (2013). Aimed at an audience of creatives, he gives a demonstration of dynamic behaviour and the use of simulation as an expressive tool. Victor presents his ideas using prototype tools, with which one can do animation without the explicit use of programming languages, through more direct visual manipulation. The computer as an art medium.

Another take on the principle can be found in Victor's talk Drawing Dynamic Visualisations (2013), together with the companion piece. The thesis here is that spreadsheets, drawing (programs), and code are all useful for some things, but being limited to each one individually is restricting the thoughts data analysts, scientists and other data workers can think. Victor then floats the idea of Dynamic Drawing, and presents his latest tool prototype that allows the user to draw dynamic visualisations that are parameterisable by data and can be interactively explored.

Many of these demos from Victor's hobo years are brought together in his talk Media for Thinking the Unthinkable (2013). The key thing with all of these demos is the immediacy of the manipulation and the powerful representations that allow the user to see deep inside the system, whatever the domain.

By the end of 2013, Victor had completed his demos and was lined up to give a keynote to a programming community at the annual Dropbox conference. Victor surprised the audience with his cheeky overhead projector presentation the Future of Programming (see also the notes), which proved to be a scathing history lesson. Victor relished the opportunity and in no uncertain terms invited the audience to check the software development orthodoxy, to check the programming dogma.

On perhaps a more constructive note, Victor offered some classical programming environment guidelines in his "immune response" essay Learnable Programming (2012). LP is all about helping programmers understand the behaviour of their programs. Again, it's all about seeing the behaviour of a complex system. Also, people should apparently read Seymour Papert's Mindstorms (1980).

Forming a research group

After his many well received demos, essays and talks, Victor again found himself at a career junction point. It was time to take stock and find new direction.

Towards the end of his first hobo year, back in 2011, Microsoft's released what Victor dubbed an "After Effects" vision for the future. This struck a nerve, and Victor responded with A Brief Rant on the Future of Interaction Design (2011). This more or less marked the beginning of Victor's research programme into a new, more humane dynamic medium. In Victor's view we should get rid of the touch screen — the very beast he once was involved in bringing to our lives. Liberating us from this gilded cage of glowing rectangles became the mission.

"Pictures Under Glass is old news. Let's start using our hands."

In 2013, Alan Kay began a new research chapter, a spinoff of sorts of his Viewpoints Research Institute, with support from German software giant SAP. The group was to be called CDG, Communications Design Group, a comically vague name that would allow the group to look into whatever they found interesting. Not much of that organisation or its projects survives today, as far as I can tell.

In any case, Victor joined in, and together with Kay they attracted a whole bunch of people to work with them. The group officially started around 2014 or so, eventually establishing, among other things, a vision and prototypes for what later became Dynamicland. Building on his previous work on interface design and tools for thought, Victor focused on software that could allow citizens and scientists to model and understand systems, further expanding the reach of his methods. For this, CDG set up a research lab in Oakland, California.

Unfortunately, the CDG champion at SAP suddenly left, and the promised long term funding quickly started to look much more unstable. In 2016, as the SAP deal was dying, mostly the same CDG crowd signed up with Y Combinator Research to form HARC, Human Advancement Research Community, only to have the funding rug pulled from under them again, in 2017. It's not clear what the funding model is today.

Despite the research group turbulence, Victor took a hard look at everything he had worked on and synthesised a vision for a future in which we have left the confines of the touch screen behind, and have regained the full use of our bodies, our presence. In a nutshell, the idea was to take the tool concepts Victor had been exploring for years, and to pull them out of the computer into the real world. With this, the room, the spatial environment, becomes the tool.

In Seeing Spaces (2014), Victor gives a taste of this remarkable vision. A Seeing Space is a new kind of environment that helps engineers and designers, creatives, see into a system, through ubiquitous sensing and data visualisation capabilities, dynamic controls and powerful ways of recording and playing back. All this to facilitate a deeper understanding about the system at hand. Crucially, this model is emphatically social, in the spirit of maker spaces.

"The challenge is not building it, but understanding it."

Finally, perhaps the best and most complete articulation of Victor's vision for the future can be found in his brilliant talk The Humane Representation of Thought: a trail map for the 21st century (2014), which begins with a retrospective. Indeed, this presentation is probably the only thing you need, if you want to understand what Bret Victor is working on. (For extra info and a summary, including a CDG stamped research poster, see the companion piece).

In the HRT talk Victor gives a highly personal account of his path so far, and outlines his long-term research agenda, starting, not unlike Engelbart back in the day, from human faculties. It's about representations and tools, about thinking the unthinkable, and then the inhumane medium we find ourselves caged in.

"We now have the opportunity to design a new medium of thought that undoes some of the damage [caused by our inhumane tools]."

The talk is Victor's attempt to put into words and in context the work he has done up to the end of 2014, or as he puts it, "to unearth the demons that have driven my work over the last decade, and to draw a map of the destination they're trying to get to". Victor outlines an illustrated research programme, a future in which people work with data and systems and one another in a more perfect union.

In short, the dynamic medium is now available, but we are missing the right dynamic representations. We must reinvent the way we represent thought. The task at hand is to bring each mode of human communication into the dynamic medium. Let people use their whole body to think, to understand. New tools for new thoughts.

Victor's is a research vision for decades to come, but the work to realise it is already underway in what journalist Carl Tashian describes as "a beautifully renovated building in old Oakland". That is, in the East Bay, the neighbourhood Victor has called home ever since his childhood.

Dynamicland

The future of computing

Dynamicland (DL, 2018-) is a new kind of research space for inventing and exploring a shared computational medium. Dynamicland is a communal computer, a human scale computationally enriched physical environment that facilitates intrinsically humane ways of working within a dynamic medium. Dynamicland is fundamentally accessible, collaborative and social in a deep, comprehensive way.

"The whole building is the computer."

Operationally Dynamicland, Victor's research lab, follows the spirit of Douglas Engelbart and Xerox PARC, who decades ago invented the technologies and principles that became the foundation of the computer revolution and the information age. This first DL instance in Oakland is being realised by a small non-profit research group with a long-term orientation, something Victor is highly passionate about.

The Dynamicland mission:
Incubate a humane dynamic medium, whose full power is accessible to all people.

The Dynamicland mission, as shaped by Victor and Alan Kay, together with their collaborators, is to liberate us from the confines of the touch screen and the display rectangle. In the new dynamic medium, the emphasis is on human interaction and humane interaction with the physical world. DL is about tangible items and artefacts, about building — crafting — dynamic controls and whole systems out of physical stuff. DL is built on real world objects you can hold and push and stack and shelve and pass.

The technology is still there, but the classical computing hardware is hidden away from sight to let dynamic behaviour and human interaction take centre stage. In Dynamicland, the objects are as if magically enchanted, enhanced beyond their common function. At the same time the "spells" that power the dynamic behaviour are plainly visible for all to see, ready for remixing and re-purposing. In a sense the hardware is implementation detail, while the software is an intrinsic part of the dynamic environment.

"Every scrap of paper has the capabilities of a full computer, while remaining a fully-functional scrap of paper."

Principles

It is not difficult to see that Bret Victor has been working towards Dynamicland his entire life. His research, his projects, his talks before Dynamicland all seem to point in this direction. DL is a physical manifestation of the ideas he has been exploring with his work on tools, on media for understanding systems, and on human-centred design.

Victor is all-in on Dynamicland: this is very much his personal mission. The vision is not at all his alone, but Victor is certainly the prime mover. Dynamicland itself is all about sharing and community, and Victor believes in doing his research in the public domain for the common good. Dynamicland is simply Victor's attempt to bring alive his vision for a better future.

"We need a new dynamic medium to understand and address the world's critical problems".

In Dynamicland, the computational media is not "hidden away in isolated virtual worlds", but rather exists in the common meatspace for all to see and get their hands on. This enables powerful ways of collaboration and sharing, working together with many hands and in eye contact with others around you. Dynamicland is "as multiplayer as the real world".

Another Dynamicland principle is that everyone, not just specialist technicians, can make apps, these dynamic documents of the new medium. And everyone can modify existing artefacts to make them do what they want, just as anyone can craft artful or useful things from paper and clay and office supplies. In Dynamicland one walks the path from playing to crafting to remixing all the way to building useful things. This is programming liberated from the confines of the iridescent rectangle.

"Programs are small, because the real world does most of the work."

In Dynamicland, programs are real things, which completely changes "the social dynamics of programming". The language of Dynamicland, Realtalk, enables this by supporting easy composability, operational flexibility. Everyone can remix and reuse the documents they come across in the dynamic medium. The interaction is dynamic as much as the behaviour of the document.

Finally, the Dynamicland model is fundamentally oriented around the idea that when dealing with a dynamic medium, we should be able to make use of the full spectrum of physical human faculties. In creative work, we should be free to spread out and reach out and walk around our document — to work at room and environment scale instead of being fixed to a display, imprisoned by the classical computer interface.

The dynamic medium invites improvisation and experimentation, intuitive comparison of differences, and facilitates the "countless ways in which human beings use their minds and bodies". This is computing with the whole body.

"Dynamicland is a communal computer,
designed for agency, not apps,
where people can think like whole humans."

Communal computing

The Dynamicland manifesto states that the mission is to have the dynamic medium serve as the foundation for new modes of thought and communication. DL is not about creating a product for the consumer market, but rather about establishing by demonstration "a new kind of civic institution", a community space where people shape the dynamic medium together. A public library for the 21st century.

"[Dynamicland] must lift all people, not just those traditionally advantaged by technology"

A communal computer could feature workspaces and galleries, where a maker space meets a science museum and residents and visitors alike can create and remix dynamic media. This dynamic space could serve as the "town hall of the future", facilitating discussions and talks on local and why not global issues, all built around dynamic, interactive resources.

Victor is thinking decades and centuries forward, but at the same time is taking concrete steps to bring his vision to reality. Dynamicland Oakland was launched in 2018 as the first full-scale realisation of the vision Kay and Victor had been working on at CDG. Since then, staff and guests at Dynamicland have created hundreds of small projects and prototypes, playfully exploring the possibilities of the new medium. Now, two years later, Victor is ready to take the next step.

In 2020, the Dynamicland folks are focused on creating the next version of the Realtalk system that powers the dynamic medium, Realtalk-2020. In 2022, Dynamicland will be shared with the world in the form of "new kinds of libraries, museums, classrooms, science labs, arts venues, and businesses".

"Realtalk-2020 will form the foundation for the next decade of research and applications."

Fundamental research for a humane future

Today, Dynamicland operates on a model Victor likens to the genesis of the Internet, with an overarching focus on openness, decentralisation, and equal access for all. These principles were "embedded in the core [Internet] protocols", and are still "worth fighting for".

Victor is acutely aware of the financial reality of running a research lab to pursue his vision. As a non-profit organisation focused on fundamental research, Dynamicland is particularly cash strapped, but also not a great fit for government funding schemes, which in Victor's view are typically incrementally oriented. As a result, Victor has become quite the vocal advocate for renewed interest in and funding for public fundamental research.

"Fundamental research is now in a time of extreme scarcity."
"A humane future must be funded."

Victor points out that the Internet age was built on a non-commercial research culture that incubated the underlying technologies for decades. The personal computing and Internet industry exploited this culture, generating massive global wealth, but then failed to meaningfully contribute back. The dominant players today are "not planting seeds for a humane future".

And so we find ourselves at a crossroads: now is the time do decide what the future of computing will look like. Will Victor have more success than Engelbart in winning funding to fully realise his vision? Will we curl ever deeper inward into our shiny rectangles, or are we witnessing the dawn of a new era in social, humane computing?

Conclusion

Bret Victor is trying to:

• Share the magic of computers and dynamic media with everyone
• Help people; do engineering for a cause, for a higher purpose
• Build creative tools for human expression
• Empower people; help everyone see and understand through insightful representations and humane interfaces
• Provide direct manipulation tools and immediate feedback
• Liberate us from the constraints imposed by poor tools and stale ways of thinking
• Encourage active reading and informed discourse
• Reinvent the way we represent thought
• Build a career around a guiding principle

Epilogue

"In 1968 — three years before the invention of the microprocessor — Alan Kay stumbled across Don Bitzer's early flat-panel display.

Its resolution was 16 pixels by 16 pixels — an impressive improvement over their earlier 4 pixel by 4 pixel display.

Alan saw those 256 glowing orange squares, and he went home, and he picked up a pen, and he drew a picture of a goddamn iPad.” (from A Brief Rant)

References