Reverse Engineering Immersive Media Systems

I

What do we mean by reverse engineering?

The traditional meaning starts with an object, a system, or a piece of software. You examine how it works, infer the structure, and reconstruct the logic or the implementation.

In this class the goal is different. We are not trying to recover the original code or the original hardware. We are trying to make a reasonable model of how an experience behaves as a system.

That model can describe an existing professional project, a proposal, a classmate’s idea, or your own capstone.

The question to keep asking

If we had to build this experience, what would need to happen for it to work?

Immersive projects often depend on parts that are expensive or easy to get wrong:

• sensors
• projection
• VR
• motion tracking
• networking
• spatial audio
• fabrication
• custom software

A problem is much cheaper to find in a diagram than in a finished installation.

II

Moving from an idea to a system

Start with a vague concept:

An interactive projection responds to visitors’ movements and gradually transforms the space.

The sentence is a concept. A system has to answer what is missing:

• What kind of movement?
• How is movement detected?
• What changes, and what causes the transformation?
• Does it respond to one person or many?
• What happens when no one is there, or when someone leaves?
• Does the system reset? Is there a beginning and an end?

Reverse engineering is how we move from idea to system, before prototype and implementation.

1. Idea
2. System
3. Prototype
4. Implementation

III

Pseudo-code

Pseudo-code is a plain-language description of system logic. It looks a little like code. It does not need to compile. Syntax does not matter. Logic does.

• IF
• ELSE
• WHEN
• WHILE
• WAIT
• REPEAT
• DETECT
• CHANGE
• RESET

While you write it, keep asking:

1. What does the system know?
2. What does the system check?
3. What decisions does it make?
4. What happens as a result?
5. What repeats?

“The participant is interested” cannot be checked. “The participant stays in the area for 20 seconds” can. The same is true of direction: left and right are observable, so the particles can follow them.

Turn “something happens” into “WHEN X happens, DO Y.”

IV

Flow charts

A flow chart maps how information moves through a system.

Input, then process, then output.

Inputs

Camera, microphone, controller, keyboard, motion or proximity sensor, participant position, audio level, a network message, or time.

Processes

Detect, measure, calculate, compare, classify, transform, generate, store, or randomize.

Outputs

Projection, image, sound, lighting, animation, physical movement, haptic feedback, or a network message.

The diagram shows dependencies, technical complexity, bottlenecks, and the places where systems connect. Those are often the risky parts.

What enters the system, what happens to it, and what comes out?

V

State machines

A state is a condition the system can remain in for a while: idle, active, waiting, calibration, interaction, success, failure, or reset.

The boxes are states. The arrows are events that cause change: someone enters, a timer expires, an object is selected, a score is reached, a threshold is crossed, someone leaves.

Participants do unpredictable things. They enter halfway through, leave early, press something over and over, skip the instructions, act at the same time, or do nothing. The diagram has to say what happens anyway.

What can the system be doing, and what causes it to change?

VI

Interaction diagrams

Pseudo-code, flow charts, and state machines describe the system. An interaction diagram describes the relationship between the participant and the system.

Participant ↔ system

The participant acts. The system responds. The participant perceives that response and acts again.

1. Action
2. Response
3. Perception
4. New action

What does the participant do, and what does the system do back?

VII

Core mechanics

A core mechanic is a fundamental action-response relationship that makes the experience work.

• Participant moves, and a virtual object moves.
• Participant speaks, and the environment changes.
• Participant approaches, and the sound gets louder.
• Two people cooperate, and hidden content appears.
• A player selects an object, and information is revealed.

A finished project might include projection mapping, motion capture, shaders, spatial audio, AI, custom controllers, networking, and fabricated objects. The mechanic underneath can still be simple: moving your body changes the position of something on screen.

That is what to prototype first. The prototype does not need the final technology. It only needs to test the important interaction.

Body tracking → Unity → projection
can be tested as
mouse → Unity cube.

VIII

Reverse engineering as risk reduction

The diagrams are a way to see trouble before a build starts. Four kinds of risk show up in the same system:

• Technical. Can we make the technology work?
• Interaction. Will participants understand what to do?
• Experience. Is the interaction actually interesting?
• Production. Can we build this within our time and resources?

Reverse engineering helps expose each of these before major implementation begins.

IX

Applying the method to a peer project

On Wednesday, you will take another student’s project idea and model it as a system.

You are not being asked to redesign it. You are trying to answer one question:

How would this project need to work?

Work from the idea as it was described. Where the description is still open, write down the assumption instead of inventing a new project.

X

Closing principle

Before you build it, can you explain how it works?

• Pseudo-code. What are the rules?
• Flow. Where does the information go?
• State. How does the system change?
• Interaction. What happens between participant and system?
• Core mechanic. What is the smallest thing we need to test?

1. Reverse engineer
2. Prototype
3. Learn
4. Build