CS 352: Computer Graphics

Input
and

Interaction

Chapter 3 - 2

Interactive Computer Graphics

What good is Computer Graphics?

JS1k canvas examples: 1 2 3 4 5 Games, visual demosof what value? Is there a Christian perspective?

Communication of information to the user

Data visualization, simulation, GUI Even a word processor is an "interactive graphics program"

Communication between users

"Collaborative environments" are hot (multi-player games?)

Social networking is transforming the world

Interaction is an essential component

Chapter 3 - 3

Interactive Computer Graphics

Interaction

Much of the fun and utility of graphics is in interaction: accepting user input that affects the display Paint programs, modeling, games, word processors, spreadsheets, powerpoint

User initiates input events such as clicking on a menu or drawing a circle Computer response by changing graphical display

Chapter 3 - 4

Interactive Computer Graphics

Chapter 3 - 5

Interactive Computer Graphics

Projects 3

Projects 3 will be a paint program

we'll learn 2-D graphics, interaction, event-loop programming, double-buffering simple animation, basic image processing Pen, line, and rectangle tools Color, pen size selectors Image processing (sharpen, blur, detect edges) Open, save images Toolbar and menu for controlling application

Features:

Chapter 3 - 6

Interactive Computer Graphics

How to set up an application UI?

How How How How

to to to to

make make make make

a menu? a color picker? a toolbar? toolbar buttons pop in and out?

Chapter 3 - 7

Interactive Computer Graphics

How to paint?

How do I make a colored line follow the mouse or fingertip?

Interactive prog:

Input devices Event handlers Event loop

Chapter 3 - 8

Interactive Computer Graphics

Input devices

Interaction requires handling input devices

Physical: mouse, fingertip, keyboard, joystick,

digitizer, accelerometer, head tracker

Logical:

Text Locator Valuator (slider) Stroke Color picker

How to read?

Chapter 3 - 9

Interactive Computer Graphics

Input devices

Sample mode

Program reads the current state of input device (frequently) Each click or motion generates an event that goes on a queue Program can process events from queue

Event mode

HTML?

Chapter 3 - 10

Interactive Computer Graphics

Event-loop programming

Events: button press, slider value change,

menu selection, mouse move, keypress

Event loop:

Poll (or wait) for events Process events, update state Update display (ideally: wait for rest of frame time to elapse)

Chapter 3 - 11

Interactive Computer Graphics

State-driven

Typically, the event loop is driven by a big state table

e.g. depressing the paintbrush tool releases other tools, puts me in "paint mode"

Good libraries will handle some of the bookkeeping for you You still typically have to handle some state

Chapter 3 - 12

Interactive Computer Graphics

State diagram

Chapter 3 - 13

Interactive Computer Graphics

Event loop in HTML/JS

HTML/JS provides event queue, support for many basic events (mousedown, mouseup
mouseover, mousemove, keypress, key release, value change, button click, etc.)

You are on your own for higher-level events, e.g. clicking on a toolbar tool It's also possible to set a function to run every 15 ms, sample input devices

Chapter 3 - 14

Interactive Computer Graphics

Events for painting?

mousedown:

go into paint mode store mouse position draw initial dot exit paint mode if in paint mode

mouseup:

mousemove:

draw a line from old mouse position to current set old mouse position to current position

Chapter 3 - 15

Interactive Computer Graphics

Event handling with jQuery

Binding events to functions

$(cpaint.canvas).bind('mousedown', cpaint.drawStart)

Processing events
cpaint.drawStart = function(ev) { var x, y; x = ev.pageX - $(cpaint.canvas).offset().left; y = ev.pageY - $(cpaint.canvas).offset().top; ev.preventDefault(); cpaint.paintmode = true; cpaint.oldX = x; cpaint.oldY = y;

Chapter 3 - 16

Interactive Computer Graphics

Looking neat and spiffy

How to avoid crinkles in your paint strokes?

Chapter 3 - 17

Interactive Computer Graphics

Looking neat and spiffy

How to avoid crinkles in your paint strokes?

Draw connected paths

Or just use line caps

Chapter 3 - 18

Interactive Computer Graphics

Menus

There are many jQuery menu plug-ins

<ul id="mainmenu"> <li>File <ul> <li id="menuNew">New</li> <li id="menuOpen">Open</li> <li id="menuSave">Save</li> ------------------------------------$('#menuNew').bind('click', cpaint.clear); $('#menuOpen').bind('click',cpaint.open); $('#menuSave').bind('click',cpaint.save);

Chapter 3 - 19

Interactive Computer Graphics

Toolbar

How to make a toolbar?

How should buttons behave?

State diagram?

Chapter 3 - 20

Interactive Computer Graphics

Buttons

Widgets may have several states State should be evident in visual feedback E.g. how exactly does a button work? States?

State transition diagram

Most buttons: six states, with six different appearances

neutral neutral-highlighted neutral-depressed selected selected-highlighted selected-depressed

Events: mousedown, mouseup, enter, exit Transitions: what happens in each state under each event?

Chapter 3 - 21

Interactive Computer Graphics

Button state diagram

(buttons selectable, not unselectable)
N S

NH

SH N: neutral H: highlighted NHD SHD

(usually mouse over)

Press Move

S: selected D: mouse down

Chapter 3 - 22

Interactive Computer Graphics

Other button considerations

Could also consider

Tooltips Whether button merely clicks and releases or can be selected Whether button can be unselected (e.g. B/I/U vs. Left, Center, Right)

Want consistent appearance, behavior over whole program or whole computer Really need a library implementation and a strict set of UI guidelines

Chapter 3 - 23

Interactive Computer Graphics

4-State toolbar buttons: CSS

.toolbarCell { background-color:#ddd; width:20pt; height:20pt; border: solid #eee 2px; -webkit-box-shadow: 2px 2px 2px #666; }

#markerButton { background: url(img/paintbrush.png) no-repeat center center; }

.selected { -webkit-box-shadow: inset 2px 2px 2px #666; }

.toolbarCell:hover { border:solid #555 2px; }

Chapter 3 - 24

Interactive Computer Graphics

4-State toolbar buttons: JS

$('#markerButton').bind('click', {tool:"marker"}, cpaint.selectTool); -------------------

cpaint.selectTool = function(ev) { cpaint.tool = ev.data.tool;

// get tool name

$('.toolbarCell').each(function(index) { // unselect $(this).removeClass('selected'); // others }); var tool = '#' + cpaint.tool + 'Button'; // get ID $(tool).addClass('selected'); // select

Chapter 3 - 25

Interactive Computer Graphics

Paintbrush size selector

How?

Chapter 3 - 26

Interactive Computer Graphics

Color picker

Google "jQuery color picker"

Chapter 3 - 27

Interactive Computer Graphics

How draw a line?

What kind of feedback is normal?

Rubber Banding

Events and actions?

Chapter 3 - 28

Interactive Computer Graphics

Save & restore

Create your own off-screen canvas Copy it back on each mouse movement Events and actions?

Chapter 3 - 29

Interactive Computer Graphics

Save & Restore event handling

Mousedown

enter line mode remember mouse position as startx, starty save screen draw initial dot paste saved screen draw line from startx, starty to current mouse pos exit line mode

Mousemove

Mouseup

Chapter 3 - 30

Interactive Computer Graphics

Analysis

Drawbacks?

slow eats memory bandwidth for breakfast

copy smallest possible rectangle? Only points from line?

possible flickering

use double buffering?

Chapter 3 - 31

Interactive Computer Graphics

Double buffering

Have two frame buffers, "front" and "back" Draw into back buffer At vertical retrace time, swap buffers
This is a fundamental graphics technique not built into canvas though, some drawing aggregation seems to happen automatically, behind the scenes; not usually necessary in canvas, but browser dependent

Chapter 3 - 32

Interactive Computer Graphics

Fake Double Buffering in Canvas

Create off-screen canvas

canvasBuffer = document.createElement('canvas'); canvasBuffer.width = canvas.width; canvasBuffer.height= canvas.height; canvasBufferContext= canvasBuffer.getContext('2d');

Draw into off-screen canvas

canvasBufferContext.[drawSomeStuff]

Copy onto display canvas

context.drawImage(canvasBuffer, 0, 0);

Chapter 3 - 33

Interactive Computer Graphics

How to erase?

Draw background color

Chapter 3 - 34

Interactive Computer Graphics

Save, Load?

Save

Copy pixels to an image so users can right-click, save-as

Load

What are the security risks? Can only load files from the same server Can use a file chooser if it's a local app Security policies are not entirely in place

Chapter 3 - 35

Interactive Computer Graphics

Resizing

How would you allow the user to shrink, expand image? What would happen to image resolution?

Chapter 3 - 36

Interactive Computer Graphics

Store drawing commands

Realistically, to redraw effectively, you have to save all drawing commands

pen color red fill color blue line width 5 circle 5 10 12 textfont Times 12 text 10 10 "hello world"

Replay commands to redraw scene Could store commands in a file (e.g. Mac PICT file) For some kinds of drawings, files are smaller and more accurate than bitmaps

Chapter 3 - 37

Interactive Computer Graphics

Other toolkits

HTML offers basic UI capabilities in a crossplatform context Writing your own UI extensions is tedious jQuery plugins extend capabilities Ideally, all UI elements ought to be built in In the real world, they come in platformspecific 'toolkits' or 'windowing libraries' E.g. MFC, QT, OpenLook, Cocoa

Chapter 3 - 38

Interactive Computer Graphics

Image processing

Image processing examples:

blur, sharpen detect edges enhance contrast noise reduction posterize fisheye redeye reduction find faces

Chapter 3 - 39

Interactive Computer Graphics

How to fade an image?

Could average pixel colors with white Or just decrease alpha

Chapter 3 - 40

Interactive Computer Graphics

How to blur an image?

Blurring is an averaging process Convolution: apply a convolution kernel, e.g.

1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

Center on pixel of interest Multiply each value by color val underneath Add results Gaussian smoothing

Chapter 3 - 41

Interactive Computer Graphics

Gaussian smoothing

Chapter 3 - 42

Interactive Computer Graphics

Chapter 3 - 43

Interactive Computer Graphics

How to sharpen an image?

It's a differentiation process What's Unsharp Masking?

Chapter 3 - 44

Interactive Computer Graphics

How to sharpen an image?

It's a differentiation process What's Unsharp Masking?

Subtract blurred version from original Add back to original

Chapter 3 - 45

Interactive Computer Graphics

Sharpening

Subtract neighbors Like subtracting a blurred version of the image Unsharp Masking E.g. convolve with a kernel like one of these:

Chapter 3 - 46

Interactive Computer Graphics

Edge detection?

Wikipedia

Chapter 3 - 47

Interactive Computer Graphics

Sobel Edge Operators

Look for horizontal and vertical variation

1 0 -1 2 0 -2 1 0 -1 -1
-2 -1

0
0 0

1
2 1

Could do this at different sales or resolutions Note: results maybe positive or negative numbers; must normalize

Chapter 3 - 48

Interactive Computer Graphics

Image Compression

What if you created say six different lowerresolution images and only stored the difference at each resolution?
Note: most of the data is in the highestfrequency component An early image compression technique

Chapter 3 - 49

Interactive Computer Graphics

The alg that changed imagery

Discrete cosine transform (DCT)

(Wikipedia)

Chapter 3 - 50

Interactive Computer Graphics

JPEG image compression

DCT to convert to frequency domain Perceptual modeling Coefficient quantization Entropy coding

Chapter 3 - 51

Interactive Computer Graphics

Research in image processing and computer vision

How would you clear up a blurry photograph?

How would you recognize a face in a photograph?

Chapter 3 - 52

Interactive Computer Graphics

Summary

Event loop programming Interaction and event handling in HTML State diagrams Painting Rubber banding Double buffering Basic image processing and convolutions "Photoshop Nano"