TOMMUNO DI UN IN DE KILIMO MAN US010055064B2

(12) United States Patent (10) Patent No.: US 10 ,055 , 064 B2

Tokutake (45) Date of Patent: Aug. 21, 2018
TL
(54 ) CONTROLLING MULTIPLE DEVICES WITH (56 ) References Cited
A WEARABLE INPUT DEVICE U .S . PATENT DOCUMENTS
(71) Applicant: Sony Corporation , Tokyo ( JP ) 9 ,069, 385 B1 * 6 /2015 Lynch .................... GO6F 3 /017
2001/0002211 A1 * 5 /2001 Lee ..................... H04M 1 /6066
(72 ) Inventor: Kenji Tokutake, Kanagawa (JP ) 379 /414
(73 ) Assignees: Sony Corporation , Tokyo ( JP ); Sony 2003/0234823 A1 * 12 /2003 Sato ......... . G06F 3 /013
715 /848
Mobile Communications Inc ., Tokyo 2008/0036647 A1 * 2/2008 Jung GO1S 11/06
342 / 118
(JP )
(Continued )
( * ) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 FOREIGN PATENT DOCUMENTS
U . S . C . 154 (b ) by 98 days. JP 2010 -231736 10 /2010
(21) Appl. No.: 14 /526 ,625 Primary Examiner — Benjamin C Lee
Assistant Examiner — Dong Hui Liang
(22 ) Filed : Oct . 29, 2014 (74 ) Attorney, Agent, or Firm — Xsensus, LLP
(65) Prior Publication Data (57) ABSTRACT
US 2016 /0124579 A1 May 5 , 2016 Embodiments include an electronic input system , which has
(51) Int . Ci. a wearable input device having an accelerometer and cir
cuitry configured for an operation detection mechanism and
G06F 3 /042 (2006 .01) an input communication mechanism . The operation detec
G06F 3 /01 ( 2006 .01) tion mechanism acquires acceleration data from the accel
GO6F 3 /0481 ( 2013.01) erometer to detect an input event. The electronic input
G06F 3 /0486 (2013 .01) system also has a first electronic device having circuitry for
G06F 3 /0346 ( 2013 . 01) a first communication mechanism , a first event analyzer , a
(52 ) U .S . CI. screen display, and a first data communication mechanism .
CPC ............ G06F 3 /0425 ( 2013 .01) ; G06F 37017 The electronic input system also has a second electronic
(2013 .01 ); GO6F 3/0346 (2013.01); G06F device having circuitry configured for a second communi
3 /0481 ( 2013 .01); G06F 3 /0486 ( 2013 .01) ; cation mechanism , a second event analyzer, a second data
GO6F 2203/04803 (2013 .01 ) communication mechanism , a screen combining mecha
(58 ) Field of Classification Search nism , and a screen display . Embodiments also include a
CPC ........ GO6F 3 /014 ; G06F 3 /017 ; G06F 3 / 1454 ; method of communicating an event between an input device
GO6F 2203 /04803; GO6F 3/0346 ; GO6F and one ormore electronic portable devices, such as the first
3 /04815 ; G06F 3 / 04842 ; G06F electronic device and the second electronic device.
2200/ 1637; G09G 5/14; G096 2370/20
See application file for complete search history. 23 Claims, 35 Drawing Sheets
32 22 592
i 100
. 40
-

I t t E74ETT - T . 22 XX !
LORS - .

22 528
* S . . . . . .

..
.

TRANSFER FILE
 US 10 ,Page
055,2064 B2

(56) References Cited

U . S . PATENT DOCUMENTS
2008 /0152263 A1 * 6 /2008 Harrison ............... G06F 1/ 1626
382 /313
2009/ 0153475 A1 * 6 / 2009 Kerr ..................... H04N 5 /4403
345 / 157
2010 /0138780 A1 6 /2010 Marano et al.
2010 /0188579 A1 * 7 /2010 Tileannan
Friedman .. ... . . . . . . . . . . . . . . H04N 5 /45
348 /565
2011/0175822 A1 * 7 /2011 Poon GO6F 3 /04842
345 / 173
2011/0199389 AL 8 /2011 Lu et al.
2011/0283334 A1 11 /2011 Choi et al.
2012 /0169482 A1 * 7 / 2012 Chen GO8C 17/ 02
340 / 12 .52
2013 /0278484 Al 10 /2013 Hwang et al.
2014 /0009394 A1 * 1/ 2014 Lee .... H04N 5 /4403
345 / 157
2014 /0176809 A1 * 6 / 2014 Chen ............... H04N 21/42222
348 /734
2014/0267024 A1 9/2014 Keller et al.
* cited by examiner
en man woww .
U.S. Patent
atent Aug. 21 , 2018

*
**

*
**
**
*
Sheet 1 of 35

hi
US 10 ,055,064 B2

*
*
**
**
*
*
*

*
*
**
*
**
*

*
**
**
***
*
*
*
**
*
*
**

*
is

*
*
**
*
**
**
*

*
*

*
****
***
**

Gesture Blueto h .
3030umODpueon

ya
+

ww es ari
w
* *

1A
.
FIG *
*

*
ave *
**

wwwwwwwww .
 atent Aug. 21 , 2018 Sheet 2 of 35 US 10 ,055,064 B2

FIG . 1B - 100

CONTROL LINE DATA LINE

110V
CONTROLLER

WIRELESS
VOICE PROCESSING
SEC } } { {
Bhai
COMMUNICATION ji

1024 PROCESSING SECTION

120 DISPLAY
SHORT -DISTANCE
WIRELESS
COMMUNICATION
TOUCH PANEL PROCESSING SECTION 107

140
SENSOR SECTION 108
OPERATION KEY

MEMORY
CAMERÁ SECTION 10
U . S . Patent Aug. 21, 2018 Sheet 3 of 35 US 10 ,055,064 B2

FIG . 2

+
42079****
**
******
***
**
**

tar5

ya

V
I 21part

var.***.

heit 77734

ostamine ***varitime
17:40
**
**
*
*
. .-.-.- . - .
U . S . Patent Aug. 21, 2018 Sheet 4 of 35 US 10 ,055,064 B2

*
*
*
**
**

*
*

*
*
**
*

B2332
20 Ka
*
03
9
O

303
VSO ervum
w

derin
www

||
wwwwwwwwwwwwwwwwwwwwww
w
.

as
Woococccccccccccccccccc0000ococco

3
.
FIG
U . S . Patent Aug. 21, 2018 Sheet 5 of 35 US 10 ,055,064 B2

*
***
*
***
*
***
**

**** *
* *** ** ** ,449 '"

*
* sowie
. *
.**
*
.
.
.
.

.
.

.
*
**
.
.
*
*

*
*
**
*
**
**

. .
.

.
.

.
.

. .
.
..

.
. .

.
.
.
. .

*** * ** **

41
22
1

*
ka

CDOO& *

PAN

4
.
FIG
U . S . Patent Aug. 21, 2018 Sheet 6 of 35 US 10 ,055,064 B2

FIG . 5
sporaque

**
*

* *
*

tiap **
**
7+
4+
4+
44444*

**
*
**
*

ve

FIG . 6
12
b e75bhanghe ottaisi
369319941 56 **
***

**

*
**
**
****
U . S . Patent Aug. 21, 2018 Sheet 7 of 35 US 10 ,055,064 B2

FIG . 7
pY
you

777156
*

"
P
*
*
*
**

o 4

46# *
**3333346

474
**** * **
*
**

*
**

******** *

48
*

***
20000:07
U . S . Patent Aug. 21, 2018 Sheet 8 of 35 US 10 ,055 ,064 B2

19

BT
iz
US
SOS
efonfraespectrospecti*ve*

WstrFeamDing
31
22 w .

verv

.w w w
20
*
w
o .

er
*

8
.
FIG
-

-
r

369 OG * .

ww
U . S . Patent Aug. 21, 2018 Sheet 9 of 35 US 10 ,055,064 B2

. * " **
************ *
****

.
.

.
-

'
Eivind
niin,

Wi
.
19
*

*
* ***
*
***
*
i
**
**
**
Sri

100A -
.
*
'*'
,-
-

.
*
*..*
**
*
.* -
--
--
-
*
-
-
-

-
-
-
*

9
.
FIG
'real

8
Ki
*
3
19DUBO Om*DO
.
O
N

ODY 22

32
52
U . S . Patent Aug. 21, 2018 Sheet 10 of 35 US 10 ,055, 064 B2

13 14
- 17
- 18
-
* * ** * * * * **

AWDIENAVRPIUBCLTE CELROMTE GYRO DISTANCESTIMA NSGECTION STORAGE

100
12
,
11

** *
KEY OPERATIODNE CTINSGECTION BLUETOCHOMUNIATSECTION
3

70B
.
704
-
|
ISNEPNUTDEV NTS
tyto sto tohoto repeaters

tut i

GESTUROAG COMUNIAT
el en
i
EIVNEPNUT PARSER BLUETO H SECTION
s*e e w
?
???

APLICTON VMAINEGWR
COMUNIATO GRAPHICS DRAWING SECTION DISPLAY
+
*
.

* **

WIFT
79 76

FIG
10
.
U . S . Patent Aug. 21, 2018 Sheet 11 of 35 US 10 ,055 ,064 B2

FIG . 11
1 Key long press
(Ring Power ONS
6 ready as
input device
BT:ON
Acc :ON 52
Mode :Gesture

Pooverkey5minowania
mout YES *ffingPower OFF
Nomination Key is
pressed ?
YES
Monitoring
accelerometer S6
memaine Rolling YES
gesture
deiectec ?
NO Switch the moderne S8
99
YES gesture mode wing NO , drag mode
Sven Gesture one

Monitoring Seddown
Svent via AT sig
accelerometer . .......
Handls down
event with previous sig
coordinates
detected ? ir target device
. . . . ... .

YES $ 12
.. ..
S20
- Soil kay is NO
direction ? pressed ?
www .
YES $ 13
Ser:d target Send gesture Monitoring
Switch event to sveit 10 target
target via 67 .. via B1. accelerometer
$ 24
Handle target Send the vector Send up
switch
target event
device 16 of ring 's move
.
*
*

$ 14 ** * * *
? ? ??
.

*
.

St key is Handle gesturs Handle drag Handle un

event ir ? fverit in ? event in
voressed ? target device target device target device
NO $ 25
U . S . Patent Aug. 21, 2018 Sheet 12 of 35 US 10 ,055 ,064 B2

FIG . 12A
ins

www

** *
miformes

FIG . 12B

744479 f
7 16-

+16

*
**
**
***
**
and

FIG . 120
With

WA
U . S . Patent Aug. 21, 2018 Sheet 13 of 35 US 10 ,055 , 064 B2

FIG . 13
f

' * ** * * *
*

... . . .
terena
*
bert
-- +
-
-- .
** * * ********
* **
** .
* .
** * * .
.
**
***
****
***
**

**.*
-
-
. .

flick
U . S . Patent Aug. 21, 2018 Sheet 14 of 35 US 10 ,055 ,064 B2

FIG . 14

. .. -- --
.... ' -
M '. .

tan
--
-- --
- ---
*** - -
. -*
. --
.
*
* -**.
*
-

-
-
*
*

. .. ***
'A
.

flick
U . S . Patent Aug. 21, 2018 Sheet 15 of 35 US 10 , 055 ,064 B2

FIG . 15

#
#
- . - . - • -* * . . .- . ... . "
- . . . . . *.

.
* ** *
. -. *
- 21
me -
*
**
*
*-* -
- *
* ..
.
. .***
.
1*.
*
**
* --
--

.
.
2t

drag
U . S . Patent Aug. 21, 2018 Sheet 16 of 35 US 10,055 ,064 B2

FIG . 16
venantsins
casting
c www
www o ntent
*

wwwy
-
+
*

.
birn X 823
XI
.

www
se * www . weblio .

Sv
.. . . . .
Sociacicecooooooo
. .V s e
Ni
.
minsreau
.nitmin * * * *
Wonpmrevina
WED streaming e
*
*
***
*
*
**

. .. . . . .. .. .. . !
.
i
.
yinmenina . .
.
.. MOVE IN DEPTH
DIRECTION OF SCREEN
inwiam ento o
SWITCH
SCREEN - TO - BE - CONTROLLED 191
cwmoin wi Bo ts LV1

ce2000
wi over

{ F ] str?????
20
U . S . Patent Aug. 21 , 2018 Sheet 17 of 35 US 10 ,055 ,064 B2

WDIENAVRPIUBCLTE ACELROMT 15
GYRO
16

100 OPERATION OPERATIONDETCINGSECTION

KEY WSTEAR BLE SECTION

21
21
SCRE NDISPLAY
24
TERMINAL HWOBIMLE
24 23

MOBILETRMINAL APLICTON MOBILEOPERATIO AIN LPYUZETR BTMOBILE SECTION

25 MWEDOBILEOUTPUTSECTION
92
STORAGE .

|
36
WW.WMV
TUNER BT
TV
SECTION
35
OTVPERATIONINPUT ANLYZER
FIG
17
.
TERMINAL TV
HOME
APLICTON WONIG
SCRE N SOFVECTION
WFDTTVINPUTS|ECTION

STORAGE SCRE NDISPLAYUNIT

U . S . Patent Aug. 21, 2018 Sheet 18 of 35 US 10 ,055 ,064 B2

FIG . 18

. . . : :: : : . . .. ...

Funk 2)
DOS OST
Logan
SARE! TV 9
U . S . Patent Aug. 21, 2018 Sheet 19 of 35 US 10 ,055 ,064 B2

••••••••••••••••••••••ervv . .

miner

EAROCMETHR *
*
GYRO
13DI12WENAVRPIUBCLTE wwwwwwwww .

erin wan 16
+

+
-

+
*

.
NOLIVado OPERATIONDETCINGSECTION
w

in er BTWEAR BLESECTION
* * *** * * * * * * * * *** *** *** . . . . . . . . . .

SCRE NDISPLAY ein -

MOBILE TERMINAL Prevai HMOBMILEAPP10TION MOBILEOPERATOIANMPLYUZLER

34
2
,
e EZ vere

.
ni *
. . . velit

.
1837180N NOT1035
001
22
-
0
25 Ww
STORAGE Ww NdioNOL.1089
35
|
36 * * *

mi
UN Iminae n 24 w ee

N O T I S M
18

aNO LY09 0INPUTAN LYZER

19
.
FIG TERMINAL HOME
IV
APLICTON W

N3 0 SONI OA0Monas
1
OHM
INONISECTION
SCRE NDISPLAYUNIT
,

La.

3978015
U . S . Patent Aug. 21, 2018 Sheet 20 of 35 US 10 , 055 ,064 B2

FIG . 20

.
w ww .
*
*

*** ** metown
atten www
*

80 FNLBL
m

mm
.
vet e

ew wwwww ***
4 . W
www . wwwwww ..
wwwwwwwwwww

winnermini
wwwwwwwwww
Z wwwwwwwwwinswwwwwwwwwwwwwwwwwwwww
wwwwwwwwwwwwwwwwwwwwww w wwwwww
U . S . Patent Aug. 21, 2018 Sheet 21 of 35 US 10 ,055 , 064 B2

S38
MWOOBILEOUTPUT 34
$ 37
$
UONRECEPTION
.
.

TM20EORBMINLAE LSAEVCTLIOENR DBTETVEICÍE SMUTLIAFORNTI SITNAPRUT MSOIBDLE 40

$

KOLE INPUT
MHOXEBRIMALTEON
APLIGATON 36
$ 39
$
* ** * *
STOMKULITCAH TERMINALCMONECDTIEON STRANFERITNMSHGOCBRIMELNBYWEDUSING 48
$

326(39)3133824 327 INPUT

1
WED
TV 1960
49
$
S43

30IVTERMINAL
TODISPLAY SCOEMBTINOG *

*
DCIOSMPBILNAEYSDCREN
JVBTTOPERATIONSSECTION 33
$
NTHATBTISDAOECTVIFCVYE TVHSCORMEEN
789

S44 45
$

35 HOME
IV

S32
FIG
.
21
1
S E A R BL E IADPOFUENLSVIPCEAUTCIREON SDETVAIRCTE
OFERATION OTNURSNPOGOWERVICE
U . S . Patent Aug. 21, 2018 Sheet 22 of 35 US 10 ,055 , 064 B2

MWOOBILESOUTPUTSECTION
muunnitelless an . + + + + + + + + +

20TMEORBMINLAE 232726 58

WOL OEPERATIMAOTON AN LYZER IANPLUYZTEV NI

55
$ 56
$ NIOTPIUFTYCONT UPDATD{EISPLAY
24 ILFROME 57
$

38
TV
WED INPUTSECTION with those

861
50
$
)
39
(
31

TERMINAL
TV
30
OSEATN KCSOERTION KS DISPLAYCOMBINEDSCRE N
V81 TVHOMESCRE N
EA
OPERATION $53SENDEVENT
ITOATECNRTPMIUNVTAEL S60
HOWE
Y
APLICTON
S52
FIG
22
.
Sco to
WOEPARTBILOEN }
_
{
}
R
{
}

PKEYRES DRAG
/
GESTURE
U .S . Patent Aug. 21, 2018 Sheet 23 of 35 US 10,055,064 B2

..

W3: NOIDOS
31 801
0 ** mi sinin

huvitatur . . . . . . . . . . . . .
80
$ 87
$
31TRANSF TORECPTIOAC3015N
T20MEORBMINLAE
hii
SECTION +
-
-
+
*
*
* ** ** * ** * Wo w ww
ISNTPOUPT NOL103 8
329 INPUT
hii
LAZER Sontenti oniewawwwwwwwwww
+

IAN PLYUZTWE3A3 W
tohoto montanhangw

90
$
a . .. .. . .

3H1OV0EHOWorlndy inty ILS 085

.

$
78
NOTTEY
INPUT
ÉVENT
và
.

.
Wawawi
.
Sen t imientowwwwwwwwwwww
::
2OL19I0K3SNOT1 2 1

TRAENQSFURIN1EG9308
W

S88
ht ' . ' . ' . ' .

. -
.

ww :

BS
UNTNOT1038
.

. .. . . insan i i in t o wwwwwwwwwwwwwwwwwwwwww
*
- -
:

,
,

.
.
,

.
81 .

16S
.

12
6£
(

DISPLAYSCOMRBEIN D
.

DSIORPGLEANY SHINI98.00ROIDO35 tewwwwwwwwwwwwww

301VTERMINAL 74
$

atSECTION ISNTPAURTONNREC PTION SCREEN

HOME
TV
|
82
$
989
NOTE830 83
$ 886
1
HOME
TV
NO IR
73
$

WEAR BLE8Add91 30 125 YVORES WHOISPERATIONS Wi

FIG.231363 OPERATION8381 80kWIK OPERATION

w ithin thi r rurowerowvee - - - - -

KB

30
:
SESN
0 www 340
$
A9X
.qi
U . S . Patent Aug. 21, 2018 Sheet 24 of 35 US 10 ,055 ,064 B2

DE

MOBILEOUTPUTSECTION
20MTEORBMINLAE MOBILE
109
$
MOBILEOPERATIONMUWALYZER
HOME S108 110
$

AP LICAT ON TRANSFEIG
?

REQUIRE SBYUCSRIENGWAD
26339)(8234 217 SECTION
SDCITVWFDSRPE NT 111
$ $
112

TERMINAL
TV
30
UNIT
SCRE N
(
A

SN100EICTNIOGN DISPLAYCOMBINEDSCRE N
B11
TV S107UD.IPSDPALTAEY
104
$
JATION IANPLUYZTEVENT
TERMINAL
SENDITOEANCVTPEIUNVTE S105 NIOTPIUFTYEVENT
HOVE
I
APLICTON
FIG
24
.
WEARBLEDIENVPIUCTE
1015 PRESS
OPERATION USER
OF KEY DRAGA GESTURE
U . S . Patent Aug. 21, 2018 Sheet 25 of 35 US 10,055 ,064 B2

FIG. 25
$ $ 122
TERMINAL AROUND ?
YES $ 123
REGISTER DETECTED
TERMINAL ON UST

S124
ERMINALS NO
REGISTERED ON LISTO
YVES $
ACQUIRE POSITION
126 S125
EACH TERMINAL

AGGREGATE AND
SYNTHESIZE VI OF $ 27
BAON IRROINALS
wwwwwwwwwww

CONTINUE OPERATION

DUE TO A008LERATION
DAIGICO N PIN
TORVONATE
OPERATION ?

VISCONTIHOLT S130
U . S . Patent Aug. 21, 2018 Sheet 26 of 35 US 10 ,055 , 064 B2

of ++ +++ * * **

OOT IC

iz
9

:
3

-
29
ezrwa 089 +

+
* *

'
*

+
*

.
.
..
..
.
..
..
.
...
....
.
..
..
..
.
...
.
..
....
. ,

www . HOCOOOOOO

*
**
*
**
**
*
**
**
.. . ...... W
-
.

*
. -

-
.

.
'

tututututu n aww w www ' ' ' ' ' --- -

-
937 *

ART *3279 .
-

-
.
-

*
-

-
.

.
-

30 * .
- *
*

-
-

SE *
.

*
+
* . .

.
.
-

31
32 .

*
*
+
-

AS
-
U . S . Patent Aug. 21, 2018 Sheet 27 of 35 US 10,055 ,064 B2

--- ..--
------ .
-

-- -
-- -
-
----
---
-- ----
-- -
--
-
-
--

--
--
- -----
-----
---
-
-
. .. - 31
30 super
US
! 19
*
*
*
*

*
**
*
**
*
*
W

32
+

09 -
-
-
-
-

-
-
-
--
-

--
-
-
-
-
-- --
-
- -

-
-

30
W

31 *
*
*

22 YWari .w w ww
*

* 117 .

27
.
FIG
32
52

inwww.
v.winwe me w

ave
*

wy +
+
*

*
'

-
U . S . Patent Aug. 21, 2018 Sheet 28 of 35 US 10 , 055 ,064 B2

-
-
-

-
--
-
-
-
-

*
**
*-

mer
-..
.
--.

.
..

*
*
Sariters
*
*
--
--*
*
*
**
**
**
*
***
*
**
*
--
-.
--
*
*
*
*
*
**

*
*
**
*
*

*
*

30

31

52 *

28
.
FIG enden
w

ser

32
Scccccccccccccccccca
U . S . Patent Aug. 21, 2018 Sheet 29 of 35 US 10 ,055 , 064 B2

100
100 *
* -

.
-
-

*
*
*
*

.
corepaonsoito
rowane
:

a wesome 50

40

*
-

w wwwwwwwww Y
FTRIANLSFER
22
52a .

•
.

22
528
wing 9

wetn
*

*
.

ww +

+
.
part
*
+

na
.
.

ww.wwwna
revim .
MAN *

32 32
*
.
*

.
*

*
*

-
re L

14
.
A
w
U .S . Patent Aug. 21, 2018 Sheet 30 of 35 US 10 ,055 ,064 B2

ACELROMT 14

inicii ,
GYRO
DIY12ENAVPRIUCBTLE
VO T OPERATION KEY SOPERATIONDETCINGECTION WBTEARBLE SECTION
84
. . . . . .. . . 444 4

.
KL2
NETWORK MODEN
/
Fi
-
Wi N82
.
64
OPERATION C TROLE 08 WBTEAR BLE SECTION
twie CLOUD CSELROVUEDR SECTION COMUNIAT Powev
.

w
WDOEUAVTRIPBCULET
ww.

w
-
1

- - .. . . . . . . ..
W
... ..
4 - 4 4 - 4- 4
SCRE N
.. .... S y
CONSTIUON Ww
HURU
SECTIOwNant ww

-
CLOUD APLICTON SCRE NOUTPUTSECTION

NETWORK T RA
MODEM
/
-
Wi
N SF E R .

NETWORK-WineMODEM
92
NETWORK Fi
-
Wi
WMODEM
(
1
-
90 2
-
90 3
-
90

MFOIBRSLTE RMINAL SMEOCBINLDET RMINAL -

MTOHBIRLDET RMINAL
.
FIG
30 REMOTED SKTOP REMOTE DESKTOP -

REMOTED SKTOP
U . S . Patent Aug. 21, 2018 Sheet 31 of 35 US 10 ,055 ,064 B2

FIG . 31
110
104 BUS
DISPLAY 105
101 CPU
106
OPERATION SECTION

} } NON -VOLATILES -107

STORAGE

108

10 mouth RAM NETWORK

INTERFACE
NCAA
TO NETWORK
wowowowwwwwwwwwwwwwwwww
U . S . Patent
atent Aug. 21, 2018
Aug Sheet 32 of 35 US 10, 055 , 064 B2

3
-
90

TMOHBIRLDETERMINAL
151
$
2
-
90

SMEOCBINLDETERMINAL
150
$

1
-
90

FMOIBRSLTE ERMINAL 5149

$
148

CLOUDSERVER CSWOTNIEACTRIOHDNPEVRAIBCLE
146
$

S145 TRANSFER DMULTI-EVICE

SMCTLOADRUETTDM-OFEURMLITNAIL SRETMAORTE OMPERADTIEON
CONETION {
O
SPLAY SCRE N
WOPEAURTBPILUOENT
IDOFUENSVPEIUCRTE
142
$ 143
$
MOTIFY
BI
THAT
SDETVAIRCTE S14
COMUNIATO
DISAECVTICVE START
153
$

141
$

There
FIG
.
32 DEVICE
TURNONPOFOWER
U . S . Patent Aug. 21, 2018 Sheet 33 of 35 US 10 ,055 ,064 B2

90
MTOHBIRLDETERMINAL

MSOEBCIOLNEO TERMINAL
2
-
90

.
-

FILE
À -

90
-
1

MFOIBRSLTE ERMINAL ,
-
,
.

AFTRIANLSFERFIROSTU
T0ER8M1INLAETOS CONT308)LEOERMINAL
+
y

"
y

"
-

165
$

CLOUD SERVER ODOFPETRAECIOTNFILEAFRONDRAG INGTMFEOIRBSNLATE AREAOFTODISPLAY.TMSEORBMIOLNEOA 167

$
164
$ 168
$

OWUEATRPUBLTE F
{
}
[

169
$
UDIPSALTEYSCRE N
EISVNEPNUTOSCTOELROVUEDR 163
$

WEARBLEDIENVPIUCTE 161
$ S162
FIG OPERATION
33
.
KEYSTO
PRESS
KEY
P R E S M
FOIBR S
L T
E TOTERMINALMSOEBCILNEDT RMINAL
USEROF FRONAFILEDRAG
 i
U . S . Patent Aug. 21, 2018 Sheet 34 of 35 US 10 ,055 ,064 B2

3
~
90
9
3
-
0 {
?

TERMINAL
90
-
2
90
-
2
SMOEBCILNED NALTERNI
9176
wane FMOIBRSLTE TERMINAL
90
1
-
90 PIRNOPCEUST
175
$
177
$
L
EVENT

61 CLOUD SERVER
5178
minn WOEUATRPBULTED VICE
174
$

TERMINAL
ISNEPNUDTETOACVTEINVTE 173
$
M * * * * *

179
$
UDIPSALIEYSCRE N

5 WEARBLE
DIENVPIUCTE
172
$

FIG
34
.
OPERATION OF
USER PKEYRES DRAG GESTURE
U . S . Patent Aug. 21, 2018 Sheet 35 of 35 US 10 ,055 ,064 B2

116 w w
KEY.FHRERALEONASMDE
ORAGEND
w ISTDERAMGINTEGD
ANDOSOTHATUPECVEUNRT FIG
35F
.

113
W KEYTOMPREOSVINEG room
MDORDAEG WMIAOTHNVUDET 1

DRAGIN
T

vo
35E
.
FIG
-
-
PWHMREAOSINVLDEG DTOCKROANEGTINYUGE
*

112
114
35B
.
FIG VDORDAEG ASUIDHIOSWPILNAGY PSTOARINTG
SDTRARGT SUIDHIOWPILNAGYSEDTVAORETWNITG POINT FIG.35D
MODESWITCH
*

+134466
*

35A
.
FIG MGEOSTDURE *
*

TSÀUIDHIOSWPAILNTGY
CANBEDONEFLICK
 US 10 ,055 ,064 B2
CONTROLLING MULTIPLE DEVICES WITH FIG . 1A is an overview of an input system according to
A WEARABLE INPUT DEVICE one embodiment ;
FIG . 1B illustrates an exemplary electronic device
BACKGROUND OF THE INVENTION according to one embodiment;
5 FIG . 2 illustrates a user with a portable terminal device
Field of the Invention according to one embodiment;
Systems and methods for operating a plurality of devices FIGS. 3 -4 each illustrate a user with a wearable input
are described . In particular, a wearable input device used to device according to one embodiment;
FIGS. 5 -6 each illustrate a portable device according to
control multiple devices is described . 10 one embodiment ;
Description of the Related Art
FIG . 7 illustrates control keys of a ring portable device
Recent developments have provided systems andmethods according
in which an input gesture can beused to direct or control one to one embodiment;
or more computing devices with a camera device . However, embodiment9 ;each illustrate an input system according to one
FIGS . 8 -
there are problems associated with the use of a camera 15 FIG . 10 is a block diagram of an exemplary input system
device , such as power consumption , a size of the camera according to one embodiment;
device , and input precision . In addition, input gestures FIG . 11 is a flow diagram of an exemplary input process
directed to a mobile device cannotbe made full- scale . Also , according to one embodiment :
a camera input gesture system cannot utilize a triaxial type FIGS. 12A - 12C each illustrate a mode ofusing an opera
of accelerometer. 20 tion key of a portable input device according to one embodi
ment ;
SUMMARY OF THE INVENTION FIGS. 13 - 14 each illustrate a flick gesture of a portable
input device according to one embodiment;
Embodiments include an electronic input system , which FIG . 15 illustrates a drag gesture of a portable input
includes a wearable input device having an accelerometer 25 device according to one embodiment;
and circuitry configured for an operation detection mecha - FIG . 16 illustrates switching a control object terminal
nism and an input communication mechanism , wherein the according to one embodiment;
ion data
operation detection mechanism acquires acceleration FIG . 17 is a block diagram of an exemplary system of an
from the accelerometer to detect an input event. The elec - input device before switching a control object terminal
tronic input system also includes a first electronic device 30 according to one embodiment;
having circuitry for a first communication mechanism , a first FIG . 18 illustrates an exemplary screen display prior to
event analyzer , a screen display, and a first data communi- switching according to one embodiment;
cation mechanism . The electronic input system also includes FIG . 19 is a block diagram of an exemplary system after
a second electronic device having circuitry configured for a switching a control object terminal according to one
second communication mechanism , a second event analyzer, 35 embodiment;
a second data communication mechanism , a screen combin . FIG . 20 illustrates a display screen after switching a
ing mechanism , and a screen display . The input communi control object terminal according to one embodiment ;
cation mechanism , the first communication mechanism , and FIG . 21 illustrates a sequence of events to create a
the second communication mechanism have circuitry con - synthetic screen display according to one embodiment;
figured to send notification of the input event of the wearable 40 FIG . 22 illustrates a sequence of events in which a
input device. The first event analyzer and the second event portable home screen is an active screen according to one
analyzer have circuitry configured to analyze the input event embodiment;
detected by the wearable input device . The first data com - FIGS. 23 - 24 each illustrate a sequence of events of
munication mechanism and the second data communication switching a control object terminal from a portable terminal
mechanism have circuitry configured to send and receive 45 to a TV terminal according to one embodiment;
screen data to and from one another. Circuitry of the FIG . 25 is a flow diagram of a user interface configuration
wearable input device is configured to wirelessly send based on a signal strength of Bluetooth according to one
operation content from the wearable input device to a embodiment;
screen -to -be- controlled of the first electronic device or the FIGS. 26A - 26B each illustrate a multiple terminal
second electronic device as a switching event, via a depth 50 arrangement and a user interface configuration according to
direction movement of the wearable input device . Circuitry one embodiment ;
of the screen combining mechanism is configured to com - FIGS. 27 - 28 each illustrate a multiple terminal and user
bine a screen of the second electronic device with a screen interface configuration according to one embodiment;
obtained from screen data received from the first electronic FIG . 29 illustrates transferring a file by utilizing a wear
device . 55 able input device according to one embodiment;
The foregoing general description of the illustrative FIG . 30 is a block diagram of an exemplary input system
embodiments and the following detailed description thereof for performing a file transfer via a cloud mechanism accord
are merely exemplary aspects of the teachings of this ing to one embodiment;
disclosure , and are not restrictive . FIG . 31 is a block diagram of an exemplary computing
60 system according to one embodiment;
BRIEF DESCRIPTION OF THE DRAWINGS FIGS. 32 - 34 each illustrate a sequence of displaying a
synthetic screen according to one embodiment ; and
A more complete appreciation of the invention and many FIGS. 35A - 35F illustrate exemplary user interface dis
of the attendant advantages thereof will be readily obtained plays for different modes according to one embodiment.
as the same becomes better understood by reference to the 65 Referring now to the drawings, wherein like reference
following detailed description when considered in connec - numerals designate identical or corresponding parts
tion with the accompanying drawings, wherein : throughout the several views.
 US 10 ,055 , 064 B2
DETAILED DESCRIPTION OF THE The speaker 104 emits an audio signal corresponding to
PREFERRED EMBODIMENTS audio data supplied from the voice processor 103 . The
microphone 105 detects surrounding audio , and converts the
Embodiments herein describe input systems and methods detected audio into an audio signal. The audio signal may
which control object terminals by using an input device, 5 then be output to the voice processor 103 for further pro
such as a ring -type input device or other wearable type of cessing . The voice processor 103 demodulates and /or
structure . The input device is small and is equipped with 3 - D decodes the audio data read from the memory 150 , or audio
inputting functions. data received by the wireless communication processor 102
and /or a short distance wireless communication processor
FIG . 1A illustrates an overview of an exemplary input 107 . Additionally , the voice processor 103 may decode
system . A user 2 operates a wearable input device 1, such as 10 audio signals obtained by the microphone 105 .
a ring input device to direct and control terminal devices. The exemplary electronic device of FIG . 1B may also
Another embodiment of an input device 1 is a small appa include a display 120 , a touch panel 130 , an operation key
ratus held in the palm of a user 's hand . The terminal devices 140 , and a short-distance communication processor 107
include , but are not limited to a portable terminal 20 , a 15 connected to an antenna 106 . The display 120 may be a
non - portable TV terminal 30 , and portable spectacles or Liquid Crystal Display (LCD ), an organic electrolumines
glasses 40 . Motion information , such as acceleration data of cence display panel, or another display screen technology . In
the wearable input device 1 is transmitted to any one of addition to displaying still and moving image data , the
terminals 20 , 30 , or 40 via Bluetooth 3 . Input operations display 120 may display operational inputs , such as numbers
include, but are not limited to a gesture and a drag and drop 20 or icons, which may be used for control of the electronic
input. The input operation is received by the target side of device 100 . The display 120 may additionally display a GUI
the terminals 20 , 30 , and 40. such that a user may control aspects of the electronic device
FIG . 1B is a block diagram illustrating an exemplary 100 and /or other devices. Further , the display 120 may
electronic device according to certain embodiments of the display characters and images received by the electronic
present disclosure . In certain embodiments , electronic 25 device 100 and / or stored in the memory 150 or accessed
device 100 may be a smartphone . However, the skilled from an external device on a network . For example , the
artisan will appreciate that the features described herein may electronic device 100 may access a network such as the
be adapted to be implemented on other devices (e . g ., a Internet, and display text and /or images transmitted from a
laptop , a tablet, a server, an e-reader, a camera , a navigation Web server.
device , etc .). The exemplary electronic device 100 of FIG . 30 The touch panel 130 may include a physical touch panel
1B includes a controller 110, a wireless communication display screen and a touch panel driver. The touch panel 130
processor 102 connected to an antenna 101 , a speaker 104 , may include one or more touch sensors for detecting an input
a microphone 105 , and a voice processor 103 . operation on an operation surface of the touch panel display
The controller 110 may include one or more Central screen . The touch panel 130 also detects a touch shape and
Processing Units (CPUs), and may control each element in 35 a touch area . Used herein , the phrase " touch operation ”
the electronic device 100 to perform functions related to refers to an input operation performed by touching an
communication control, audio signal processing, control for operation surface of the touch panel display with an instruc
the audio signal processing, still and moving image process - tion object, such as a finger, thumb, or stylus- type instru
ing and control, and other kinds of signal processing . The ment. In the case where a stylus, or the like , is used in a
controller 110 may perform these functions by executing 40 touch operation , the stylus may include a conductive mate
instructions stored in a memory 150 . Alternatively or in rial at least at the tip of the stylus such that the sensors
addition to the local storage of the memory 150 , the func - included in the touch panel 130 may detect when the stylus
tions may be executed using instructions stored on an approaches /contacts the operation surface of the touch panel
external device accessed on a network , or on a non- transi display ( similar to the case in which a finger is used for the
tory computer readable medium . 45 touch operation ).
The memory 150 may include, e .g., Read Only Memory In certain aspects of the present disclosure, the touch
(ROM ), Random Access Memory (RAM ), or a memory panel 130 may be disposed adjacent to the display 120 (e. g.,
array including a combination of volatile and non -volatile laminated ), or may be formed integrally with the display
memory units . The memory 150 may be utilized as working 120 . For simplicity , the present disclosure assumes the touch
memory by the controller 110 while executing the processes 50 panel 130 is formed integrally with the display 120 and
and algorithms of the present disclosure . Additionally , thetherefore , examples discussed herein may describe touch
memory 150 may be used for long -term storage , e .g ., of operations being performed on the surface of the display 120
image data and information related thereto . rather than the touch panel 130 . However, the skilled artisan
The electronic device 100 includes a control line CL and will appreciate that this is not limiting.
data line DL as internal communication bus lines . Control 55 for simplicity, the present disclosure assumes the touch
data to / from the controller 110 may be transmitted through panel 130 is a capacitance -type touch panel technology ;
the control line CL . The data line DL may be used for however, it should be appreciated that aspects of the present
transmission of voice data , display data , etc . disclosure may easily be applied to other touch panel types
The antenna 101 transmits /receives electromagnetic wave (e. g ., resistance type touch panels) with alternate structures.
signals between base stations for performing radio -based 60 In certain aspects of the present disclosure, the touch panel
communication , such as the various forms of cellular tele - 130 may include transparent electrode touch sensors
phone communication . The wireless communication proces arranged in the X - Y direction on the surface of transparent
sor 102 controls the communication performed between the sensor glass .
electronic device 100 and other external devices via the The touch panel driver may be included in the touch panel
antenna 101. For example , the wireless communication 65 130 for control processing related to the touch panel 130 ,
processor 102 may control communication between base such as scanning control. For example , the touch panel
stations for cellular phone communication . driver may scan each sensor in an electrostatic capacitance
 US 10 ,055 , 064 B2
transparent electrode pattern in the X - direction and Y -direc - inter-device communication via the short -distance wireless
tion and detect the electrostatic capacitance value of each communication processor 107 .
sensor to determine when a touch operation is performed . The electronic device 100 may include a motion sensor
The touch panel driver may output a coordinate and corre - 108 . The motion sensor 108 may detect features ofmotion
sponding electrostatic capacitance value for each sensor. 5 (i . e ., one or more movements ) of the electronic device 100 .
The touch panel driver may also output a sensor identifier For example , the motion sensor 108 may include an accel
that may be mapped to a coordinate on the touch panel erometer, a gyroscope , a geomagnetic sensor, a geo -location
display screen . Additionally , the touch panel driver and sensor, etc ., or a combination thereof, to detect motion of the
touch panel sensors may detect when an instruction object, electronic device 100 . In certain embodiments, the motion
such as a finger , is within a predetermined distance from an 10 sensor 108 may generate a detection signal that includes data
operation surface of the touch panel display screen . That is, representing the detected motion . For example , the motion
the instruction object does not necessarily need to directly sensor 108 may determine a number of distinct movements
contact the operation surface of the touch panel display in a motion ( e. g ., from start of the series of movements to the
screen for touch sensors to detect the instruction object and stop , within a predetermined time interval, etc .), a number of
perform processing described herein . For example , in certain 15 physical shocks on the electronic device 100 ( e . g ., a jarring ,
embodiments, the touch panel 130 may detect a position of hitting, etc ., of the electronic device ), a speed and/ or accel
a user ' s finger around an edge of the display panel 120 ( e . g ., eration of the motion (instantaneous and / or temporal), or
gripping a protective case that surrounds the display/touch other motion features. The detected motion features may be
panel) . Signals may be transmitted by the touch panel driver , included in the generated detection signal. The detection
e . g ., in response to a detection of a touch operation , in 20 signal may be transmitted , e . g ., to the controller 110 ,
response to a query from another element, based on timed whereby further processing may be performed based on data
data exchange , etc . included in the detection signal .
The touch panel 130 and the display 120 may be sur- The electronic device 100 may include a camera section
rounded by a protective casing , which may also enclose the 109 , which includes a lens and shutter for capturing photo
other elements included in the electronic device 100 . In 25 graphs of the surroundings around the electronic device 100 .
certain embodiments , a position of the user 's fingers on the The images of the captured photographs can be displayed on
protective casing (but not directly on the surface of the the display panel 120 . A memory section saves the captured
display 120 ) may be detected by the touch panel 130 photographs . The memory section may reside within the
sensors . Accordingly , the controller 110 may perform dis - camera section 109 , or it may be part of the memory 150 .
play control processing described herein based on the 30 FIG . 2 illustrates a first embodiment of utilizing an input
detected position of the user 's fingers gripping the casing . system , such as the input system illustrated in FIG . 1A . The
For example , an element in an interface may be moved to a user 2 has a portable terminal 20 in a left hand , and inputs
new location within the interface ( e . g ., closer to one or more instructions by operating a ring -type wearable input device
of the fingers ) based on the detected finger position . 1, which is on the right hand of the user 2 . The portable
Further, in certain embodiments , the controller 110 may 35 terminal 20 is an example of the electronic device 100
be configured to detect which hand is holding the electronic described above with reference to FIG . 1B . Even though a
device 100, based on the detected finger position . For right finger with the input device 1 does not touch the
example , the touch panel 130 sensors may detect a plurality portable terminal 20 , the portable terminal 20 can still be
of fingers on the left side of the electronic device 100 ( e . g ., operated . When the user 2 performs an input operation ,
on an edge of the display 120 or on the protective casing), 40 results are displayed on a screen display 21 of the portable
and detect a single finger on the right side of the electronic terminal 20 .
device 100 . In this exemplary scenario , the controller 110 FIG . 3 illustrates a second embodiment of utilizing an
may determine that the user is holding the electronic device input system , such as the input system illustrated in FIG . 1A .
100 with his /her right hand because the detected grip pattern The user 2 operates the wearable input device 1 as a cursor
corresponds to an expected pattern when the electronic 45 pointer to drag and drop the displayed image of the portable
device 100 is held only with the right hand . terminal 20 to the screen of TV terminal 30 . The displayed
The operation key 140 may include one or more buttons screen 31 of the TV terminal 30 displays the original screen
or similar external control elements, which may generate an 32 of the TV terminal 30 and also displays the screen 22
operation signal based on a detected input by the user. In from the portable terminal 20 . This was accomplished
addition to outputs from the touch panel 130 , these operation 50 without touching either the portable terminal 20 or the TV
signals may be supplied to the controller 110 for performing terminal 30 , and without the aid of a camera device .
related processing and control . In certain aspects of the FIG . 4 illustrates a third embodiment of utilizing an input
present disclosure, the processing and/ or functions associ- system , such as the input system illustrated in FIG . 1A . FIG .
ated with external buttons and the like may be performed by 4 illustrates a displayed screen 41 of the spectacles terminal
the controller 110 in response to an input operation on the 55 40 . The displayed screen 41 displays screen 32 of the TV
touch panel 130 display screen rather than the external terminal 30 in the background and screen 22 of the portable
button , key , etc . In this way , external buttons on the elec - terminal 20 in the foreground . The screens 22 and 32 are
tronic device 100 may be eliminated in lieu of performing manipulated by the input device 1 on the hand or finger of
inputs via touch operations, thereby improving water-tight- the user 2 .
ness . 60 FIG . 5 illustrates an input device 1 in the form of a
The antenna 106 may transmit/receive electromagnetic ring -type input device , which is annular-ring shaped and has
wave signals to /from other external apparatuses , and the a through -hole in the center. The user 2 wears the ring - type
short-distance wireless communication processor 107 may input device 1 on an arbitrary finger of the left hand or the
control the wireless communication performed between the right hand . The input device 1 has a power supply key 11 and
other external apparatuses. Bluetooth , IEEE 802 . 11 , and 65 an operation key 12 at a main body of the ring -type input
near- field communication (NFC ) are non - limiting examples device 1 . FIG . 5 illustrates two separate keys . However, a
of wireless communication protocols that may be used for single key could be used for both functions of a power
 US 10 ,055 , 064 B2
supply and operations. The user 2 can execute various FIG . 9 illustrates an input system 100A in which input
inputs , such as a gesture , a flick , or a drag and drop actions are remotely controlled . In FIG . 9 , the screens
maneuver by performing a linear motion or a rotary motion displayed on the portable terminal 20 , a table -type terminal
while pressing the operation key 12 . In an embodiment, 50, and the TV terminal 30 are transmitted to an input
when the power supply key 11 is pressed for a predetermined 5 apparatus worn on glasses 40 of a user 2 , via a cloud
amount of time, such as 5 seconds ormore , the power supply structure 60 . Screens 22, 32, and 52 are displayed on a
can be switched on and off. screen display 41 of the glasses 40 . Input system 100A
displays screen content from all three terminals as one
FIG . 6 illustrates an input device 1A , which is a small screen
apparatus configured to be held in the palm of a hand of the 10 tion . As, which is controlled by 3 - D input and cloud utiliza
user 2 . It includes a power supply key 11 and an operation screens displayed on, the
an example given for illustrative purposes only, the
portable terminal 20 , the tablet -type
key 12 , with functions as described above for the ring -type
terminal 50 , and the
input device 1 . FIG . 6 illustrates a rectangular-shaped input glasses 40 worn by the user TV terminal 30 are transmitted to the
device 1A . However, other shapes are contemplated by 2 , via a cloud structure 60 . The
three screens are displayed on a single screen display 41. In
embodiments described herein , such as a round or ovalal 15 addition , an active screen is selected from the three screens
shaped input device . In addition , a shape specific to the left by supplying an input signal in the z - axis direction of the
hand and specific to the righthand can be used . A wristband input device 1 to the glasses 40 , via the cloud 60. As a result,
type of input device could also be used with embodiments a control object terminal is designated and operated by the
described herein . input device 1 .
FIG . 7 illustrates the axes of rotation in which input from 20 FIG . 10 is a block diagram of an exemplary structure of
a ring-type input device 1 can be forwarded to one or more an input system 100 . Bluetooth communication can be used
receiving terminals. A y -axis is illustrated in the vertical between a wearable input device 1, a first terminal device
plane of the page , and two transverse planes perpendicular target 70A , and a second terminal device target 70B . A target
to the vertical plane are illustrated along an x -axis and a is a terminal that can become a control object of the
Z -axis. The z -axis is along an axial direction of the finger 25 wearable input device 1 . With reference back to FIG . 9 , the
when the finger with the input device 1 is extended and target could be one of the portable terminal 20, the glasses
pointed towards one of the receiving terminals. A power 40, the tablet-type terminal 50 , or the TV terminal 30 .
supply key 11 and an operation key 12 are also illustrated . The wearable input device 1 could be a ring-type input
The input device 1 also includes a triaxial - type acceler device or a handheld device . The wearable input device 1
ometer and a triaxial- type gyroscope within the input device 30 has a power key 11 and a controlkey 12 , which could be the
1 . The input device 1 can detect a directional action to any same structural key or two separately -controlled keys . The
of the three axes with the built-in accelerometer. In addition , wearable input device 1 also includes an accelerometer 13 ,
the input device 1 can detect rotation about each of the three a gyroscope 14 , an operation detection mechanism 15 , and
axes with a built- in gyroscope. This provides motion detec - a Bluetooth communication mechanism 16 as hardware
tion in a total of six axes , i. e . three directional or linear axes 35 components . The operation detection mechanism 15
and three rotational axes. In addition to a ring- type input acquires data , which is outputted by the accelerometer 13
device 1 illustrated in FIG . 7 , a triaxial-type accelerometer and the gyroscope 14 , and detects an operation performed by
and a triaxial -type gyroscope can be configured with a a user as an input event.
handheld input device , such as the input devices described The wearable input device 1 changes an input mode by
above with reference to FIG . 6 . 40 detection of a motion from the user . Two types of input are
FIG . 8 illustrates an input system 100 according to one a gesture and a drag . When the user presses the operation
embodiment. Multiple terminal screens are combined into key 12 , the hand having the wearable input device 1 is
one screen by combining a screen from each terminal onto moved . This action sends motion information , such as
a destination terminal. The screens are forwarded or trans - acceleration data to target 70A and /or 70B . When the user
ferred to a destination terminal using a multipoint function , 45 does not press the operation key 12 , the input device 1 does
in conjunction with Bluetooth communications. FIG . 8 not detect input from the user.
illustrates an input device 1 on the finger of a user. A portable For Bluetooth communication , the relationship between a
terminal 20 receives an operation input from the input signal strength or field intensity of Bluetooth and distance
device 1, via Bluetooth , wherein the screen 22 of portable between a signal source and receptor is utilized . A distance
terminal 20 is forwarded from the screen display 21 to the 50 estimation between terminals is performed based on the
screen display 31 of TV terminal 30 . In addition , file sharing signal strength . After generating a communication path
and an output of a media stream can occur from the portable between terminals, a signal strength is corrected to stabilize
terminal 20 to the TV terminal 30 , via Wi-Fi. The user the communication . The wearable input device 1 includes a
operates the input device 1 by utilizing a combination of distance estimation mechanism 17 and a storage mechanism
directional and rotational input gestures. As an example , 55 18 . The distance estimation mechanism 17 estimates a
given for illustrative purposes only , the user can point the distance from the input device 1 to a connecting point
finger containing the input device 1 towards the portable terminal based on the strength of a received signal. The
terminal 20 while holding down the operation key 12 . This distance estimation mechanism 17 can change the strength
may be followed by a flicking gesture away from the of a radio signal transmitted from the Bluetooth communi
portable terminal 20 , and a second pointing gesture towards 60 cation mechanism 16 . The storage mechanism 18 can
the TV terminal 30. The operation key 12 may be continu include a non - volatile storage component, which has data
ously held down for this series of actions, or the operation for a relationship between a signal strength of Bluetooth and
key 12 may be pressed multiple times for each individual a distance between the input device 1 and a connecting
action . As a result, screen 32 of the TV terminal 30 is terminal contained in a table .
displayed on the screen display 31 in the background , and 65 Target 70A in FIG . 10 is equipped with a Bluetooth
screen 22 of the portable terminal 20 is displayed in the communication mechanism 71 , a gesture /drag analysis
foreground . mechanism 72 , an input event transmission mechanism 73 ,
 US 10 ,055 , 064 B2
10
a view manager mechanism 74 , an application mechanism not pushed , i.e . the index finger 2f and thumb 2t are
75 , a graphics mechanism 76 , a drawing mechanism 77 , a separated . FIG . 12B illustrates a state in which the operation
display mechanism 78 , and a Wi-Fi communication mecha- key 12 of the input device 1 is pushed , wherein the thumb
nism 79 as hardware components . Execution of the appli - 2t contacts the operation key 12 of the input device 1.
cation mechanism 75 is achieved when a central processing 5 When the operation key 12 is pressed ( a “ yes ” decision in
unit or a microprocessor unit reads and performs program step S5 ), the operation detection mechanism 15 acquires
code of software stored in a storage component . The content acceleration
ay data from the accelerometer 13 in step S6 . In an
received by the Bluetooth communication mechanism 71 as embodiment, when the operation key 12 is pressed twice at
input from the wearable input device 1 is analyzed in the the time of input, a vibration occurs to indicate detection of
gesture /drag analysis mechanism 72 . The analyzed input 10 a drag gesture .
event is forwarded to the view manager mechanism 74 , via In step S7, the operation detection mechanism 15 deter
the input event transmission mechanism 73 .
The view manager mechanism 74 controls changing a mines from acceleration data whether a rotation gesture was
component of a screen according to content of an input detected . FIG . 12C illustrates one embodiment in which
event, such as a moving pointer. An example of a view 15 acceleration data would be generated , wherein the index
manager mechanism 74 is a window manager. The applica finger af rotates , which is detected by the accelerometer
tion mechanism 75 can receive the view manager 's 74 and /or gyroscope within the input device 1. When a rotation
process results , perform the process according to the input gesture is detected ( a “ yes ” decision in step S7), the opera
event, and control each component in target 70A . The tion detection mechanism 15 switches an input mode to drag
execution result of the input event by the application mecha - 20 mode in step S8. The rotation gesture is utilized for switch
nism 75 is received , the view manager mechanism 74 ing an input mode . However , switching the mode is not
changes the component of a screen , and supplies change restricted to a rotation gesture .
content to the graphics mechanism 76 . When a rotation gesture is not detected in the determina
Graphics mechanism 76 comprises a screen which gen - tion processing ( a “ no ” decision in step S7) , the operation
erates image data of a screen and supplies it to the drawing 25 detection mechanism 15 determines whether the current
mechanism 77 . An example of a graphics mechanism 76 is input mode is a gesture mode. When the current input mode
a graphical user interface (GUI). The drawing mechanism 77 is a gesture mode ( a “ yes ” decision in step S9 ), the operation
obtains image data of the screen from the graphics mecha detection mechanism 15 acquires acceleration data from the
nism 76 onto the display mechanism 78 using a framebuffer , accelerometer 13 in step S10 . The operation detection
as an example . A liquid crystal display can be used as a 30 mechanism 15 determines whether the flick gesture was
display panel with the display mechanism 78 . The Wi-Fi detected in step S11 . When a flick gesture is not detected ( a
communication mechanism 79 communicates according to " no " decision in step S11), the operation detection mecha
telecommunications standards between other targets , such as nism 15 transfers to the determination processing of step
target 70B . A structure oftarget 70B is similar to or the same S17 . When a flick gesture is detected (a “ yes ” decision in
as target 70A . 35 step S11 ), it is determined whether the operation detection
The wearable input device 1 considers a rotation opera - mechanism 15 was a flick gesture in the x - axis direction or
tion first as a motion at the time of switching of an input the y -axis direction in step S12 . When it is a flick gesture in
mode. However, it may classify an input mode into a the X - axis direction or the y -axis direction ( a “ yes” decision
switching mode by carrying out a two - stage push -key uti - in step S12 ), the operation detection mechanism 15 trans
lization . As an example , the structure of a thin -shaped 40 mits a gesture event to target 70A through the Bluetooth
two-stage operation tactile push switch , such as a two -stage communication mechanism 16 in step S13 . The application
push -key can be used . mechanism 75 of target 70A processes the gesture event
FIG . 11 is an exemplary flowchart for a process of an received from the input device 1 in step S14 .
input system , such as input system 100 . An example is given FIG . 13 illustrates an exemplary flick gesture. A user
for a ring - type input device . However, other input devices 45 quickly moves the index finger 2f with the input device 1
can be used , such as a handheld input device . The process is mounted thereon in the upwards y -axis direction . The user' s
given from the perspective of the input device 1 and the thumb 2t depresses the operation key 12 of the input device
target 70A as illustrated in FIG . 10 . In step S1 , the user 2 1 while going through the flicking motion . FIG . 14 illus
holds down the power supply key 11 for a period of time, trates another exemplary flick gesture towards a side z -axis
such as 3 - 5 seconds to power on the input device 1 . In step 50 direction while depressing the operation key 12 of the input
S2, the operation detection mechanism 15 of the input device 1 with the thumb 2t.
device 1 turns on the accelerometer 13 and the gyroscope 14 , With reference back to FIG . 11 , when the flick gesture is
and turns on a Bluetooth function . The input mode of an not in the x -axis or y -axis direction ( a “ no ” decision in step
initial stage is set to a gesture . S12 ), the operation detection mechanism 15 transmits a
In step S3 , the operation detection mechanism 15 deter - 55 target switching event to target 70A through the Bluetooth
mines whether a key operation has occurred within a pre - communication mechanism 16 in step S15 . The application
determined time, such as five minutes . When there is no key mechanism 75 of target 70A processes the target switching
operation in the predetermined time ( a “ yes” decision in step event received from the wearable input device 1 in step $ 16 ,
S3 ), the operation detection mechanism 15 turns off the in which target 70A switches a control object terminal
power supply of the input device 1 in step S4 . When a key 60 between target 70A and target 70B . In step S17 , the opera
operation does occur in the predetermined time (a " no" tion detection mechanism 15 of the input device 1 deter
decision in step S3) , the operation detection mechanism 15 mines whether the operation key 12 is still pressed or
determines whether the operation key 12 is pressed in step pressed again . When the operation key 12 is not pressed (a
S5 . When the operation key 12 is not pressed ( a “ no” “ no ” decision in step S17 ), the operation detection mecha
decision in step S5 ), the operation detection mechanism 15 65 nism 15 transfers to step S3 . When the operation key 12 is
returns to the process flow at step S3 . FIG . 12A illustrates a pressed (a “ yes” decision in step S17 ), the operation detec
state in which the operation key 12 of the input device 1 is t ion mechanism 15 transfers to step S10 .
 US 10 ,055 , 064 B2
12
When the decision of step S9 for a gesture mode is no, i.e . direction or z -axis direction of the screen display 31 to
when it is a drag mode, the operation detection mechanism generate the control object switching event.
15 transmits the down event by the user 2 to target 70A The lower drawing of FIG . 16 exemplifies a mode 100 in
through the Bluetooth communication mechanism 16 in step which the user operates the input device 1 to input into the
S18 . A down event is the downward direction motion of the 5 TV terminal 30 after switching a control object terminal
index finger 2f, which is equipped with the input device 1 . from the portable terminal 20 to the TV terminal 30 . The
This motion is used to contact the detector plane of a screen display 31 of the TV terminal 30 illustrates screen 32
touchscreen . In FIG . 7 , this motion is in the y - axis direction in the forefront of the screen display 31 of TV terminal 30 ,
The application mechanism 75 of target 70A processes the which is a control object terminal after switching the dis
down event received from the input device 1 in step S19 . 10 played screen 22 . The user operates the input device 1 and
Target 70A processes the down event using the coordinates performs input operations to the TV terminal 30 through
designated by the last operation in the target 70A . Bluetooth communications.
In step S20 , the operation detection mechanism 15 of the FIG . 17 illustrates an exemplary system 100 of an input
input device 1 determines whether the operation key 12 is device 1, a portable terminal 20 , and a TV terminal 30 before
still depressed . When the operation key 12 is still depressed 15 switching a control object terminal according to embodi
( a “ yes ” decision in step S20 ), the operation detection ments described herein . The input device 1 includes a
mechanism 15 acquires acceleration data from the acceler - wearable input device , such as a ring , or a handheld device .
ometer 13 in step S21 . The operation detection mechanism The input device 1 includes a Bluetooth communication
15 transmits vector information of a motion of the input mechanism 16 , which was previously illustrated in FIG . 10 .
device 1 to target 70 A through the Bluetooth communication 20 The portable terminal 20 is a first terminal device that is
mechanism 16 in step S22 . The application mechanism 75 of equipped with a portable screen display 21 , a portable
target 70A processes the drag event received from the input operation input analysis mechanism 23 , a portable home
device 1 through the Bluetooth communication mechanism application mechanism 24 , a storage mechanism 25 , a
71 in step S23 . In an exemplary embodiment, the operation portable Wi-Fi output mechanism 26 ( a first data commu
detection mechanism 15 could move the cursor or other 25 nication component), and a portable Bluetooth communica
markers displayed on the display mechanism 78 of the target tion mechanism 27 . The portable screen display mechanism
70A according to a motion of the input device 1 . FIG . 15 21 corresponds to component 77 and the display 78 illus
illustrates an exemplary drag gesture in which a user moves trated in FIG . 10 .
the index finger 2fwith the mounted input device 1 in a side In FIG . 17 , the portable operation input analysis mecha
direction or z -axis direction while depressing the control key 30 nism 23 corresponds to a gesture drag analysis mechanism
12 with the thumb 2t. 72 and the input event transmission mechanism 73 in FIG .
With reference back to FIG . 11 , the operation detection 10 . The portable home application mechanism 24 corre
mechanism 15 transfers to step S20 at the conclusion of step sponds to the application mechanism 75 which controls each
S23 . In step S24 , when the operation key 12 is no longer block in the portable terminal 20 . The storagemechanism 25
depressed a “ no ” decision in step S20 ), the operation 35 stores parameters used by the portable home application
detection mechanism 15 transmits an up event to target 70A mechanism 24 and the generated data , as an example . The
through the Bluetooth communication mechanism 16 . An up storage mechanism 25 has a function of a frame buffer. The
event is a motion in which a user detaches from the detector portable Wi- Fi output mechanism 26 corresponds to the
plane of a touchscreen of the index finger 2f in which the Wi- Fi communication mechanism 79 . The portable Blu
input device 1 is mounted . With reference to FIG . 7 , an up 40 etooth mechanism 27 corresponds to the Bluetooth commu
event motion is in the positive y - direction . nication mechanism 71 .
With reference back to FIG . 11 , the application mecha - In FIG . 17 , the TV terminal 30 is a second terminal device
nism 75 of target 70A processes the up event received from equipped with a TV screen display 31, a TV Bluetooth
the input device 1 in step S25 . As an example, when the user mechanism 33 , a TV operation inputanalysis mechanism 34 ,
2 designates and drags a file icon displayed on the screen , 45 a TV home application mechanism 35 , a tuner 36 , a storage
movementof an applicable file is transferred to a place or a mechanism 37 , a TV Wi- Fi inputmechanism 38 , and a TV
folder icon to which the up event was performed . The screen synthesizing section 39 . The TV screen display 31
operation detection mechanism 15 transfers to step S3 at the corresponds to the drawing component 77 and the display 78
conclusion of step S25 . illustrated in FIG . 10 . The TV Bluetooth mechanism 33
FIG . 16 illustrates an embodiment of switching a control 50 corresponds to the Bluetooth communication mechanism 71 .
object terminal. The upper drawing of FIG . 16 exemplifies The TV operation input analysis mechanism 34 corresponds
a mode 100 in which a user operates the input device 1 to the gesture drag analysis mechanism 72 and the input
mounted upon the user ' s finger. Through one or more event transmission mechanism 73 .
gestures, the user designates the portable terminal 20 as the The TV home application mechanism 35 corresponds to
control object terminal through Bluetooth communication . 55 application 75 and controls each block in the TV terminal
The portable terminal 20 has transmitted image data of 30 . The tuner 36 receives a television signal and supplies the
screen 22 from the screen display 21 to the screen display 31 signal to the TV home application mechanism 35 . The
of TV terminal 30 by a Wi-Fi transfer function . The screen storage mechanism 37 stores the parameter used by the TV
display 31 of the TV terminal 30 illustrates screen 32 of the home application mechanism 35 and the generated data , as
TV terminal 30 and screen 22 of the portable terminal 20 . 60 an example . The storage mechanism 37 includes the func
Since the portable terminal 20 is the control object terminal, tion of a frame buffer.
screen 22 is displayed in front of screen 32 of the TV The TV Wi- Fi input mechanism 38 corresponds to the
terminal 30 . Wi-Fi communication component 79 in FIG . 10 . The TV
The middle drawing of FIG . 16 exemplifies a mode in screen synthesizing mechanism 39 corresponds to the view
which the user operates the input device 1 to switch a control 65 manager 74 and the graphics component 76 . As illustrated in
object terminal from the portable terminal 20 to the TV FIG . 17 , when the portable terminal 20 is a control object
terminal 30 . The user moves the input device 1 in the depth terminal, the communication based on the portable terminal
 US 10 ,055 , 064 B2
13 14
20 and Bluetooth is possible for the input device 1. How - mechanism 33 that the portable terminal 20 has switched to
ever, it is in a state in which Bluetooth communication with a multi -terminal connection mode. In step S44 , the TV
the TV terminal 30 is not possible . The portable terminal 20 Bluetooth mechanism 33 transfers the notification from the
and the TV terminal 30 are in a state in which communi- portable Bluetooth mechanism 27 of the portable terminal
cation is based on Wi-Fi communication . 5 20 to the TV home application mechanism 35 . In step S45 ,
FIG . 18 illustrates an exemplary screen display prior to the TV home application mechanism 35 generates a TV
switching a control object terminal. In the screen display 31 home screen , such as the screen 32 illustrated in FIG . 16 , and
of the TV terminal 30 , the TV home screen 32 and the outputs it to the TV screen synthesizing mechanism 39 .
portable home screen 22 of the portable terminal 20 are In step S46 , the portable home application mechanism 24
displayed . 10 generates a portable home screen , such as screen 22 illus
FIG . 19 illustrates an exemplary system 100 of an input trated in FIG . 16 , and starts a Wi-Fi transfer. The Wi-Fi
device 1 , a portable terminal 20 , and a TV terminal 30 after connection has established the portable terminal 20 and the
switching a control object terminal according to embodi- TV terminal 30 in advance in one embodiment. However,
ments described herein . When a control object terminal is when it is not connected , the Wi- Fi connection may be
switched from the portable terminal 20 to the TV terminal 15 established at the present time in another embodiment. In
30 , the communication based on Bluetooth is possible step S47 , the portable home application mechanism 24
between the TV terminal 30 and the input device 1 . How - outputs a portable home screen to the portable Wi-Fi output
ever, they cannot communication with the portable terminal mechanism 26 and orders a Wi-Fi transfer. In step S48, the
20 , via Bluetooth since communication between the portable portable Wi- Fi output mechanism 26 outputs a portable
terminal 20 and the TV terminal 30 is based on Wi- Fi 20 home screen to the TV Wi- Fi inputmechanism 38 of the TV
communication. terminal 30 . In step S49, the TV Wi-Fi input mechanism 38
FIG . 20 is an exemplary illustration of a display screen of the TV terminal 30 transfers the portable home screen
after switching a control object terminal. The screen display received from the portable Wi-Fi outputmechanism 26 of
31 of the TV terminal 30 contains screen 22 of the portable the portable terminal 20 to the TV screen synthesizing
terminal 20 , as well as screen 32 of the TV terminal 30 . TV 25 mechanism 39.
terminal 30 is the control object terminal after switching the In step S50 , the TV screen synthesizing mechanism 39
control object terminal. generates a combined TV home screen and the portable
FIG . 21 illustrates a sequence of events to create a home screen to output a synthetic screen to TV screen
synthetic screen display . The input system includes content display mechanism 31. The TV screen display mechanism
from the input device 1 , the portable terminal 20 , and the TV 30 31 displays a synthetic screen .
terminal 30 . FIG . 21 illustrates a sequence in which the At this time, the portable terminal 20 is set to the control
synthetic screen of screen 32 ( a TV home screen ) and screen object terminal as a default configuration for example , and
22 of the portable terminal 20 are displayed in the TV screen a portable home screen is displayed on the TV home screen .
display 31 of the TV terminal 30 . The user 2 presses down In another example , a flag is set in an active screen which
the power supply key 11 of the input device 1 in step S31 . 35 enables it to identify the TV home application mechanism 35
The operation detection mechanism 15 of the input device 1 of the TV terminal 30 from another screen . In this example ,
detects that the power supply key 11 has been depressed and a flag is set to a portable home screen .
turns on a power supply in step S32 . FIG . 22 illustrates an input operation sequence in which
In step S33 , the wearable Bluetooth mechanism 16 of the a portable home screen is an active screen when the syn
input device 1 notifies the portable Bluetooth mechanism 27 40 thetic screen of the TV home screen and a portable home
of an active Bluetooth device . In step S34 , the portable screen are displayed on TV terminal 30 . In step S51, the user
Bluetooth mechanism 27 of the portable terminal 20 2 presses down the power supply key 11 of the input device
receives notification from the input device 1 and searches for 1. In step S52, the user 2 presses the operation key 12 of the
the Bluetooth device on a periphery . In steps S35 and S36 , input device and performs a drag gesture . In this embodi
the portable Bluetooth mechanism 27 searches the TV 45 ment, the acceleration data output from the accelerometer 13
terminal 30 and the input device 1 , as an example . The TV can be analyzed more correctly and exact input information
terminal 30 and the wearable input device 1 respond to the can be obtained .
portable Bluetooth mechanism 27 when it exists in a state In step S53, the operation detection mechanism 15 of the
which can be discovered as a Bluetooth device . In step S37 , input device 1 transmits an input event to the portable
the portable Bluetooth mechanism 27 starts a multipoint 50 Bluetooth mechanism 27 of the portable terminal 20 , which
connection with the portable terminal 20 and the TV termi- is an active terminal through Bluetooth mechanism 16 . In
nal 30 when the response from the portable terminal 20 and step S54, the portable Bluetooth mechanism 27 of the
the TV terminal 30 is detected . portable terminal 20 sends notification of the input event
In step S38 , the portable Bluetooth mechanism 27 starts received from the input device 1 to the portable operation
an input registration at the side of the portable terminal 20 . 55 input analysis mechanism 23 . In step S55 , the portable
In step S39, the portable Bluetooth mechanism 27 sends operation input analysis mechanism 23 analyzes the notified
notification to the input device 1 that input registration has input event.
started on the portable terminal 20 side . In step S40, the In step S56 , the portable operation input analysis mecha
portable Bluetooth mechanism 27 sends notification to the nism 23 sends notification of the analyzed result of the input
portable home application mechanism 24 that the portable 60 event to the portable home application mechanism 24 . In
home application mechanism 24 has started input registra - step S57 , the portable home application mechanism 24
tion . In step S41 , the portable home application mechanism displays the updated process according to the analysis result
24 switches to a multi-terminal connection mode . of the input event. The portable home application mecha
In step S42 , the portable home application mechanism 24 nism 24 outputs the portable home screen updated according
notifies the portable Bluetooth mechanism 27 it is switched 65 to the input event to the portable Wi-Fi output mechanism
to a multi -terminal connection mode . In step S43 , the 26 , and orders a Wi-Fi transfer . In step S58 , the portable
portable Bluetooth mechanism 27 notifies the TV Bluetooth Wi- Fi output mechanism 26 outputs the portable home
 US 10 ,055 , 064 B2
15
screen after an update to the TV Wi-Fi inputmechanism 38 Wi- Fi input mechanism 38 of the TV terminal 30 transfers
for the TV terminal 30 . In step S59, the TV Wi- Fi input the portable home screen received from the portable Wi- Fi
mechanism 38 of the TV terminal 30 transfers the portable output mechanism 26 of the portable terminal 20 to the TV
home screen after the update received from the portable screen synthesizing mechanism 39 . In step S92 , the TV
Wi- Fi output mechanism 26 of the portable terminal 20 to 5 screen synthesizing mechanism 39 generates a combined TV
the TV screen synthesizing mechanism 39. home screen and the portable home screen to output a
In step S60 , the TV home application mechanism 35 synthetic screen to TV screen display 31 . The TV screen
generates a TV home screen and outputs it to the TV screen display 31 displays a synthetic screen .
synthesizing mechanism 39 . In step S61, the TV screen FIG . 24 illustrates an input operation sequence of a
synthesizing mechanism 39 generates the screen which 10 control object terminal switching from the portable terminal
combines the TV home screen and the portable home screen 20 to the TV terminal 30 . In step S101 , the user 2 presses
after an update , and outputs a synthetic screen to the TV down the power supply key 11 of the input device 1 . In step
screen display mechanism 31. The TV screen display S102 , the user 2 presses the operation key 12 of the input
mechanism 31 displays the synthetic screen after the update . device and performs a drag gesture operation . In this
FIG . 23 illustrates a sequence at the time of switching a 15 embodiment, the acceleration data output from the acceler
control object terminal from the portable terminal 20 to the ometer 13 can be analyzed more correctly and exact input
TV terminal 30 . The user 2 presses down the power supply information can be obtained . In step S103 , the operation
key 11 of the input device 1 in step S71 . The user 2 performs detection mechanism 15 of the input device 1 transmits an
an operation to switch a control object terminal by pressing input event to the TV Bluetooth mechanism 33 of the TV
the operation key 12 in step S72 . The operation detection 20 terminal 30 through the wearable Bluetooth mechanism 16 .
mechanism 15 of the input device 1 transmits the data of the TV terminal 30 is an active terminal, i. e . the control object
input event to switch a control object terminal in step S73 . terminal. In step S104, the TV Bluetooth mechanism 33 of
The Bluetooth mechanism 16 of the input device 1 transmits the TV terminal 30 notifies the TV operation input analysis
an input event to the portable Bluetooth mechanism 27 of the mechanism 34 of the input event received from the input
portable terminal 20 in step S74 . 25 device 1 .
In step S75 , the portable Bluetooth mechanism 27 of the In step S105 , the TV operation input analysis mechanism
portable terminal 20 sends notification that the input event 34 analyzes the input event. In step S106 , the TV operation
has been received to the portable operation input analysis input analysis mechanism 34 notifies the TV home applica
mechanism 23 . In step S76 , the portable operation input tion mechanism 35 of the analyzed results of the input event.
analysis mechanism 23 analyzes the input event. In step 577 , 30 In step S107, the TV home application mechanism 35
the portable operation input analysis mechanism 23 notifies performs a display update process according to the analyzed
the portable home application mechanism 24 of the analyzed result of the input event. The TV home application mecha
results of the input event. In step S78 , the portable home nism 35 updates the TV home screen and outputs it to the TV
application mechanism 24 recognizes the input event is an screen synthesizing section 39 .
event of control object terminal switching . 35 In step S108, the portable home application mechanism
In step S79 , the portable home application mechanism 24 24 of the portable terminal 20 generates a portable home
notifies the portable Bluetooth mechanism 27 of the control screen and requests a Wi-Fi transfer. In step S109, the
object terminal switching event. In step S80 , the portable portable home application mechanism 24 outputs a portable
Bluetooth mechanism 27 transfers the input reception of home screen to the portable Wi-Fi outputmechanism 26 and
Bluetooth to the TV terminal 30 side . In step S81, the 40 orders a Wi-Fi transfer. In step S110 , the portable Wi-Fi
portable Bluetooth mechanism 27 notifies the TV Bluetooth output mechanism 26 outputs a portable home screen to the
mechanism 33 of the TV terminal 30 that the input reception TV Wi-Fi inputmechanism 38 of the TV terminal 30 . In step
of Bluetooth was transferred to the TV terminal 30 side . In S111 , the TV Wi- Fi inputmechanism 38 of the TV terminal
step 882 , the TV Bluetooth mechanism 33 of the TV 30 transfers the portable home screen received from the
terminal 30 starts the input registration at the side of the TV 45 portable Wi- Fi output mechanism 26 to the TV screen
terminal 30. synthesizing mechanism 39 . In step S112 , the TV screen
In step S83, the TV Bluetooth mechanism 33 notifies the synthesizing mechanism 39 generates a combined TV home
input device 1 that input registration has started at the TV screen and the portable home screen to output a synthetic
terminal 30 side. In step S84, the portable Bluetooth mecha screen to TV screen display mechanism 31 . The TV screen
nism 27 of the portable terminal 20 notifies the input device 50 display mechanism 31 displays a synthetic screen .
1 that an input reception has stopped . In step 585 , the TV FIG . 25 is a flow diagram illustrating a processing flow of
Bluetooth mechanism 33 of the TV terminal 30 transfers the user interface configuration based on a signal strength of
initiation of the input registration by the TV terminal 30 side Bluetooth . Bluetooth communicates by changing a signal
to the TV home application mechanism 35 . In step 886 , the strength with respect to a distance between a signal source
TV home application mechanism 35 generates a TV home 55 and a receptor. The input system estimates a distance of the
screen and outputs it to the TV screen synthesizing mecha wearable input device 1 and each terminal, based on the
nism 39 . signal strength or field intensity of Bluetooth . A presumed
In step S87 , the portable Bluetooth mechanism 27 of the result is reflected in the user interface configuration of the
portable terminal 20 stops reception of the input. In step 888 , synthetic screen . When an input system is displayed on a
the portable home application mechanism 24 generates a 60 screen display mechanism , the depth direction of the screen
portable home screen and requests a Wi-Fi transfer. In step of each terminal is determined according to the distance
S89 , the portable home application mechanism 24 outputs a between the input device 1 and the associated terminal.
portable home screen to the portable Wi- Fi output mecha - FIG . 25 illustrates a flow of the user configuration based
nism 26 and orders a Wi-Fi transfer. In step S90 , the portable on the Bluetooth signal strength . The input system includes
Wi-Fi output mechanism 26 of the portable terminal 20 65 the wearable input device 1 , the portable terminal 20 , and the
outputs a portable home screen to the TV Wi- Fi input TV terminal 30 . In step S121 , the user 2 turns on the power
mechanism 38 of the TV terminal 30 . In step S91, the TV supply of the input device 1 . In step S122 , the Bluetooth
 US 10 ,055 , 064 B2
18
communication mechanism 16 of the input device 1 attempts mounted on a finger of a user and each terminal occurs in the
to communicate with a surrounding terminal using the order of the portable terminal 20 being the closest, the
largest signal strength or field intensity for Bluetooth tablet- type terminal 50 at a mid - distance, and the TV ter
according to a command of the distance estimation mecha minal 30 being the most distant terminal from the input
nism 17 . The distance estimation mechanism 17 determines 5 device 1 . In FIG . 26B , the screen display 31 of the TV
whether there is a terminal on a periphery from the result of terminal 30 displays screen 22 of the portable terminal 20 in
the Bluetooth communication mechanism 16 and a sur front view ( the closest terminal to the input device in FIG .
rounding terminal. When a terminal does not exist on a 26A ), screen 52 of the tablet-type terminal 50 (the next
periphery, the input device 1 transfers to step S124 . closest terminal to the input device in FIG . 26A ) , and screen
In step S123, when there is a surrounding terminal ( a 10 32 of the TV terminal in the most rear view (the most distant
" yes" decision in step S122 ), the distance estimation mecha terminal to the input device in FIG . 26A ). The active screen ,
nism 17 registers the information , which identifies the screen 22 is displayed on the front view of the screen display
terminal associated with the response into a list stored in the 31. FIGS. 26A - 26B illustrate a correspondence between the
storage mechanism 18 . The input device 1 returns to step distance of the input device 1 to each terminal and the
S122 at the conclusion of step S123 . 15 arrangement of the depth direction on the screen display 31 .
In step S124 , when a terminal does not exist on a As a result, it is possible to replace an active screen by
periphery as determined in step S122 , the distance estima- operation in a depth direction , i.e . the z - axis direction of the
tion mechanism 17 determines whether a list has registration input device 1.
of a terminal. In step S125 , when there is no registration of FIG . 27 illustrates a second embodiment of a multiple
a terminal in the list (a “ no ” decision in step S124 ), the input 20 terminal and a user interface configuration . In FIG . 27 , the
device 1 ends the operation . horizontal distance of each terminal from the input device 1
In step S126 , when a terminal is listed in the terminal is the same, although the total distance from the input device
registration table (a “ yes” decision in step S124 ), the dis - 1 differs for each terminal. The portable terminal 20 , the
tance estimation mechanism 17 of the input device 1 esti - tablet - type terminal 50 , and the TV terminal 30 are all
mates the distance of the terminal from the input device 1 . 25 located in a straight line in which the depth direction , i. e .
The distance estimation mechanism 17 acquires the posi- z -axis direction with respect to the input device 1 is sub
tional information of the input device 1 for each terminal stantially the same. Even though the portable terminal 20 is
registered in the registration table . The registration table the most distant terminal from the input device 1 , it is set as
stores the relationship of the Bluetooth signal strength and the control object terminal. As a result , the screen display 31
the distance of each terminal in the storage mechanism 18 . 30 of the TV terminal 30 is displayed in the order of screen 22
The distance estimation mechanism 17 matches the infor- of the portable terminal 20 in front view , screen 52 of the
mation with the positional information which identifies each tablet -type terminal 50 in second view , and screen 32 of the
terminal, and stores the information temporarily in the TV terminal 30 in the most rear view .
storage mechanism 18 . When the tablet -type terminal 50 is operated more pref
In step S127 , the Bluetooth communication mechanism 35 erentially than the portable terminal 20 , the screen display
16 of the input device 1 transmits the information which 31 of the TV terminal 30 can display screen 52 of the
identifies each terminal stored in the storage mechanism 18 tablet -type terminal 50 in the front view ahead of screens 22
with positional information to a control object terminal, such and 32 of the portable terminal 20 and the TV terminal 30 ,
as portable terminal 20 . The portable terminal 20 transfers respectively.
the terminal identification information and positional infor - 40 FIG . 28 illustrates a third embodiment of a multiple
mation to the TV terminal 30 , based on Bluetooth . The TV terminal and a user interface configuration 100 . In FIG . 28 ,
home application mechanism 35 of the TV terminal 30 the distance between the input device 1 and each terminal is
determines the arrangement of screen 32 of TV terminal 30 , approximately the same. In this embodiment, the weighting
and screen 22 of the portable terminal 20 displayed on the of each terminal is equal, and therefore , there is no setting
active screen of TV screen display 31. 45 of one terminal as a preferred control object terminal. In
In step S128 , a control object terminal begins input FIG . 28 , screen 52 of the tablet- type terminal50 is displayed
registration by Bluetooth from the input device 1 . In step in the center of the screen display 31 of the TV terminal 30
S129 , the control object terminal determines whether to as an active screen . Screen 22 of the portable terminal 20 and
switch the control object terminal according to acceleration screen 32 of the TV terminal 30 are on the back side of
detected in a depth direction from the input device 1 . In step 50 screen 52 and are displayed on the right side and the left
S130 , when switching of the control object terminal occurs side , respectively of screen 52 . Screens 52 , 22 , and 32 are
(a “ yes ” decision in step S129 ), the portable terminal 20 and rotated transversely according to an operation of a user with
the TV terminal 30 are switched as the control object respect to the wearable input device 1, wherein the active
terminal. In step S131 , when switching of the control object screen interchanges accordingly . In one embodiment, the
terminal does not occur ( a “ no ” decision in step S129 ) , a 55 active screen can interchange by operation of a depth
control object terminal determines whether to close opera - direction of the input device 1 . In another embodiment, the
tion . When operations are to be terminated (a “ yes” decision active screen may interchange by operation of a horizontal
in step S131) , the input device 1 ends the operation in step direction , i.e . the x - axis direction .
S132 . When a closing operation is not conducted ( a “ no ” FIG . 29 illustrates transferring a file 100 by utilizing a
decision in step S131 ) , the input device 1 is transferred to 60 wearable input device . A file is transferred between termi
step S129 . nals with a drag function in the z -axis direction using the
FIGS. 26A - 26B illustrate a first embodiment of a multiple input device 1. The upper drawing of FIG . 29 illustrates a
terminal arrangement and a user interface configuration . In wearable input device 1 mounted on a finger to input and / or
one embodiment, a user interface structure displays each manipulate files A and B in the screen display 41 of glasses
terminal on the front screen sequentially according to a 65 40 using a Bluetooth function on glasses 40 of a user 2 .
distance from the input device 1. For the input system 100 Screen 52a of the tablet-type terminal 50 , screen 22 of the
of FIG . 26A , the distance between the input device 1 portable terminal 20 , and screen 32 of the TV terminal 30 are
 US 10 ,055, 064 B2
19 20
displayed on the screen display 41 of glasses 40 . The middle 73 of targets 70A and 70B of FIG . 10 . The cloud server 61
drawing of FIG . 29 illustrates a user wearing the input manages operations control of each portable terminal and
device 1 and operating the input device 1 in a depth the screen structure . The wearable input device 1 and the
direction , i.e. Z -axis direction to designate file A . By an wearable output device 80 perform input and output as a
upward dragging gesture of the input device 1 , designated 5 physical interface with respect to each portable terminal.
file A in screen 52a of the tablet-type terminal 50 is trans - FIG . 31 is a block diagram of an exemplary computing
ferred to screen 22 of portable terminal 20 in the screen device used in conjunction with one or more of the cloud
display 41 of glasses 40 . The transfer of designated file A server 61 , wearable input device 1 , wearable output device
from the tablet-type terminal 50 to the portable terminal 20 80 , or the first through the third portable terminals, 90 - 1
is executed as shown in the lower drawing of FIG . 29 . In 10 through 90- 3 illustrated in FIG . 30 . Computer 110 includes
addition to transferring a file, an operation to copy or delete central processing unit (CPU ) 101 , read -only memory
a file can also be executed . (ROM ) 102 , and random - access memory (RAM ) 103, which
FIG . 30 is a schematic structure of an exemplary input are interconnected via bus 104 . Computer 110 also includes
system for performing a file transfer via a cloud mechanism . a display mechanism 105 , an operation mechanism 106 , a
Input system 100A includes a wearable input device 1 , a 15 non - volatile storage mechanism 107 , and a network inter
wearable output device 80 , a cloud structure 60, a first face mechanism 108. CPU 101 reads from ROM 102
portable terminal 90 - 1, a second portable terminal 90 -2 , and program code of software to execute functions described
a third portable terminal 90 - 3 . Terminals 90 - 1 through 90 -3 herein . CPU 101 generates data packets or replaces the
are examples of target terminal devices . transmission order of a data packet to transmit . CPU 101
The wearable input device 1 includes an operation key 12 , 20 replaces the transmission order of the received data packet
an accelerometer 13 , a gyroscope 14 , an operation detection and restores the original data or relays a data packet.
mechanism 15 , and a wearable Bluetooth mechanism 16 . Processing apparatuses, such as a micro -processing unit can
Wearable output device 80 is an output device which can be used in place of CPU 101.
be mounted or attached to a user's body. One example of a Arithmetic processing or variables received and transmit
wearable output device 80 is spectacles or glasses, a TV 25 ted are temporarily stored in RAM 103. RAM 103 is also
terminal, an interactive white board , etc . The wearable used for various buffers in embodiments described herein .
output device 80 includes a cloud application mechanism liquid crystal display monitor can be used for the
81, a wearable Bluetooth mechanism 82 , a screen output display mechanism 105 , and results of processing by com
mechanism 83 , and a network Wi-Fi/modem mechanism 84 . puter 110 are displayed . A keyboard , mouse , or other periph
The cloud application mechanism 81 includes one or more 30 erals can be used with the operation mechanism 106 . The
processors , such as a central processing unit to read and non -volatile storage mechanism 107 can be a hard disk drive
perform program code of software retrieved from a storage (HDD ). In the non - volatile storage mechanism 107 , the
mechanism . The cloud application mechanism 81 controls program for computer functions other than an operating
each component of the wearable output device 80 . Wearable system can be recorded and stored in the non - volatile
Bluetooth mechanism 82 communicates with the wearable 35 storagemechanism 107 . A network interface card (NIC ) can
Bluetooth mechanism 16 of the wearable inputdevice 1 . The be used for the network interface mechanism 108 , where
screen output mechanism 83 corresponds to the screen various data is transmitted and received between networks in
display 31 of the TV terminal 30 or the screen display 41 of it through a local area network (LAN ) . The network inter
glasses 40 . face mechanism 108 corresponds to the communication
The screens of the first portable terminal 90 - 1 through the 40 mechanism 62 of the cloud server of FIG . 30 to communi
third portable terminal 90 - 3 are displayed via the cloud 60 cate with an external terminal. Although computer 110
through network Wi-Fi modems 92 to and from respective illustrates a display mechanism 105 and an operation mecha
portable desktops 91. Network Wi- Fi/modem 84 transmits nism 106 , an alternative embodiment does not include these
and receives to and from the cloud server 61 of the cloud 60 features in the cloud server 61.
to the wearable output device 80 . Network Wi-Fi/modem 84 45 Portable terminals 90 - 1 through 90 - 3 can be equipped
also performs modulation and demodulation of the data to with the structure and function of computer 110 . In an
transmit and receive . exemplary system , the first portable terminal 90 -1 can be
The cloud server 61 within the cloud 60 includes a equipped with a remote desktop 91 and a network Wi-Fi/
communication mechanism 62, a screen structure mecha - modem 92 . The remote desktop 91 is configured to receive
nism 63 , and an operation control mechanism 64 . The 50 operations from another computer connected to the network .
communication mechanism 62 transmits and receives vari- A graphical user interface (GUI) and the desktop environ
ous data to and from the wearable output device 80 via m ent of the first portable terminal 90 - 1 are configured to
Wi- Fi. The communication mechanism 62 also transmits operate with instructions from the wearable input device 1
and receives data with the first through third portable through the cloud server 61 and the wearable output device
terminals 90 - 1 through 90 - 3 via Wi- Fi. The communication 55 80 . Network Wi- Fi/modem 92 corresponds to network Wi
mechanism 62 corresponds to the Wi-Fi communication Fi/modem 84 of the cloud server 61 . Network Wi-Fi/modem
mechanism 79 of targets 70A and 70B of FIG . 10 . 92 transmits and receives data to and from the cloud server
Screen structure mechanism 63 includes the screen output 61 of the cloud 60 via Wi- Fi. Network Wi-Fi/modem 92
to the screen output mechanism 83 of the wearable output performs modulation and demodulation of the transmitted
device 80 . Screen structure mechanism 63 corresponds to 60 and received data .
the view manager 74 and the graphics mechanism 76 of FIG . 32 illustrates a sequence in which a synthetic screen
targets 70A and 70B of FIG . 10 . The operation control of screen 32 of a TV home screen 32 and screen 22 of the
mechanism 64 analyzes the inputevent of thewearable input portable terminal 20 are displayed on a TV screen display 31
device 1 transmitted from the wearable output device 80 of the TV terminal 30 . The input system includes the
according to the analyzed result . The operation control 65 wearable input device 1, the wearable otput device 80 , the
mechanism 64 corresponds to the gesture drag analysis cloud server 61, and the first through third portable terminals
mechanism 72 and the input event transmission mechanism 90- 1 through 90 - 3 .
 US 10 ,055 , 064 B2
The user 2 presses down the power supply key 11 of the down the power supply key 11 of the input device 1 . In step
input device 1 in step S141. The operation detection mecha - S162, while the user 2 presses the operation key 12 , file A
nism 15 of the input device 1 detects that the power supply is dragged to the second portable terminal 90 -2 from the first
key 11 has been depressed and turns on a power supply in portable terminal 90 - 1 . In step S163 , the operation detection
step S142 . In step S143 , wearable Bluetooth mechanism 16 5 mechanism 15 of the wearable input device 1 transmits an
of the wearable input device 1 sends notification to the input event toward the cloud server 61 through wearable
wearable output device 80 that the wearable input device 1 Bluetooth mechanism 16 . In step S164, wearable Bluetooth
is active as a Bluetooth device. Wearable Bluetooth mecha - mechanism 82 of the wearable output device 80 receives an
nism 82 of the wearable output device 80 receives notifica input event from the wearable input device 1 . The cloud
tion from the wearable input device 1 and searches for a 10 application mechanism 81 of the wearable output device 80
peripheral Bluetooth device that might exist . The wearable transfers the received input event to the cloud server 61
input device 1 responds to the wearable Bluetooth mecha - through network Wi- Fi/modem 84.
nism 82 when a peripheral Bluetooth device exists . Wear - In step S165 , the communication mechanism 62 of the
able Bluetooth mechanism 82 of the wearable output device cloud server 61 receives an input event from the wearable
80 established a connection based on the wearable input 15 output device 80 . The operation control mechanism 64
device 1 and Bluetooth when a response from the wearable analyzes the content of the input event. As a result of the
input device 1 is detected . analysis , the operation control mechanism 64 detects that it
In step S144 , wearable Bluetooth mechanism 16 of the is an operation in which the input event dragged file A to the
wearable input device 1 begins communication with the display area of the second portable terminal 90 -2 from the
wearable Bluetooth mechanism 82 of the wearable output 20 display area of the first portable terminal 90 - 1 . In step S166 ,
device 80 . In step S145 , the cloud application mechanism 81 the operation control mechanism 64 of the cloud server 61
of the wearable output device 80 begins a cloud mode of a sends notification that file A is transferred to the second
multi-terminal connection . In step S146, the cloud applica - portable terminal 90- 2 with respect to the first portable
tion mechanism 81 of the wearable output device 80 begins terminal 90 - 1 . In step S167, the operation control mecha
a remote - control mode. 25 nism 64 of the cloud server 61 sends notification to the
In step S147, the cloud application mechanism 81 of the second portable terminal 90- 2 that file A is transferred from
wearable output device 80 notifies the cloud server 61 that the first portable terminal 90 - 1 to the second portable
remote -control mode has started through network Wi- Fi/ terminal 90 -2 . File A is transferred to the second portable
modem 84 . In step S148 , the operation control mechanism terminal 90 - 2 from the first portable terminal 90 - 1 via the
64 of the cloud server 61 receives notification of having 30 cloud server 61.
started remote - control mode from the wearable output In step S168, the communication mechanism 62 of the
device 80 . The communication mechanism 62 of the cloud cloud server 61 receives the image data corresponding to the
server 61 begins a connection with the device which can be screen of the remote desktop 91 after transferring file A from
operated . the first portable terminal 90 - 1 to the second portable
In steps S149 , S150 , and S151, the communication 35 terminal 90 -2 . The screen structure mechanism 63 of the
mechanism 62 of the cloud server 61 begins a connection cloud server 61 updates a synthetic screen based on the
process via Wi-Fi with each of the first through third received image data . The image data corresponding to the
portable terminals, 90- 1 through 90 - 3. The first through third updated synthetic screen is transferred to the wearable
terminals connect with the cloud server 61, wherein each output device 80 through the communication mechanism 62 .
remote desktop 91 of the first through third portable termi- 40 In step S169 , network Wi- Fi/modem 84 of the wearable
nals transmits image data corresponding to the screen of the output device 80 receives image data corresponding to the
remote desktop 91 to the cloud server 61 through network synthetic screen on which the first portable terminal through
Wi- Fi/modem 92 . the third portable terminal was updated . The cloud applica
In step S152 , the communication mechanism 62 of the tion mechanism 81 outputs the image data corresponding to
cloud server 61 receives the image data corresponding to the 45 the synthetic screen output mechanism 83, and updates a
screen of the remote desktop 91 from the first portable display screen .
terminal 90 - 1 through the third portable terminal 90 - 3 . The Input events , such as a file transfer were described as
screen structure mechanism 63 of the cloud server 61 being transmitted to the first through the third portable
generates a synthetic screen based on the received image terminals via the cloud 60 . In other embodiments , an opera
data . The image data corresponding to the synthetic screen 50 tion is not required to operate through the cloud 60 . Instead ,
is transferred to the wearable output device 80 through the the wearable output device 80 and the first through the third
communication mechanism 62 . portable terminals could perform immediate data commu
In step S153 , network Wi-Fi/modem 84 of the wearable nication without going through the cloud 60.
output device 80 receives the image data corresponding to FIG . 34 illustrates a sequence of events, via remote
the synthetic screen of the first through the third portable 55 control for a synthetic screen of each screen of the first
terminals 90 - 1 through 90 -3 . The cloud application mecha through the third portable terminals, displayed to the wear
nism 81 outputs and displays the image data corresponding able output device 80 . In this example , the first portable
to the synthetic screen of the first through the third portable terminal 90 - 1 is set to be the control object terminal. In step
terminals 90 - 1 through 90 - 3 on the screen outputmechanism S171 , the user 2 presses down the power supply key 11 of
83 . In this processing , the screen of each remote desktop 91 60 the input device 1 . In step S172 , the user 2 presses the
of the first through the third portable terminals is combined , operation key 12 of the input device and performs a drag or
and it displays on the screen output mechanism 83 of the gesture operation .
wearable output device 80 . In step S173 , the operation detection mechanism 15 of the
FIG . 33 illustrates a sequence of input operations when a wearable input device 1 transmits an input event through
synthetic screen of each screen for the first through the third 65 wearable Bluetooth mechanism 16 towards the first portable
portable terminals 90 - 1 through 90 - 3 is displayed to the terminal 90 - 1 , which is the active control object terminal. In
wearable output device 80 . In step S161, the user 2 presses step S174 , wearable Bluetooth mechanism 82 of the wear
 US 10 ,055 ,064 B2
23 24
able output device 80 receives the input event from the The functions, processes , and algorithms described herein
wearable input device 1. The cloud application mechanism may be performed in hardware or software executed by
81 of the wearable output device 80 transfers the received hardware, including computer processors and /or program
input event to the cloud server 61 through network Wi- Fi/ mable processing circuits configured to execute program
modem 84 . In step S175 , the communication mechanism 625 code and /or computer instructions to execute the functions,
of the cloud server 61, receives the input event from the processes , and algorithms described herein . A processing
wearable output device 80 . The operation control mecha circuit includes a programmed processor, as a processor
nism 64 of the cloud server 61 transfers the received input includes circuitry. A processing circuit also includes devices
event to the first portable terminal 90 - 1 through the com - . such as an application specific integrated circuit (ASIC ) and
munication
m mechanism 62. conventional circuit components arranged to perform the
In step S176 , the first portable terminal 90 - 1 receives an recited functions.
input event through network Wi-Fi/modem 92 . The remote The functions and features described herein may also be
desktop 91 of the first portable terminal 90 - 1 analyzes and executed by various distributed components of a system . For
processes the input event. In step S177 , the remote desktop 16 example , one or more processors may execute these system
91 of the first portable terminal 90 - 1 transmits the image functions, wherein the processors are distributed across
data corresponding to the screen of the remote desktop 91 on multiple components communicating in a network . The
which the input event was reflected to the cloud server 61 distributed components may include one or more client
through network Wi-Fi/modem 92 . and /or server machines, in addition to various human inter
In step S178 , the communication mechanism 62 of the 20 face and /or communication devices (e.g., display monitors,
cloud server 61 receives the image data corresponding to a smart phones, tablets, personal digital assistants (PDAs)).
screen from the first portable terminal 90 - 1 . The operation The network may be a private network , such as a LAN or
control mechanism 64 of the cloud server 61 transfers the WAN , or may be a public network , such as the Internet.
received image data to the wearable output device 80 Input to the system may be received via direct user input
through the communication mechanism 62 . In step S179, 25 and / or received remotely either in real- time or as a batch
network Wi- Fi/modem 84 of the wearable output device 80 process . Additionally, some implementations may be per
receives the image data corresponding to the screen of the formed on modules or hardware not identical to those
first portable terminal 90 - 1 in which the input event was described . Accordingly, other implementations are within
reflected . The cloud application mechanism 81 outputs the the scope that may be claimed .
image data corresponding to the synthetic screen of the first 30 It is noted that, as used in the specification and the
portable terminal 90 - 1 through the third portable terminal appended claims, the singular forms “ a ," " an , ” and “ the ”
90 - 3 to the screen output mechanism 83 , and updates a include plural referents unless the context clearly dictates
display screen . otherwise .
FIGS. 35A - 35F illustrate an exemplary user interface The above disclosure also encompasses the embodiments
display for different modes , such as a gesture mode and a 35 noted below .
drag mode in the form of an auxiliary guide . FIG . 35A ( 1 ) A wearable electronic input device comprising cir
illustrates left- right- up -down directional keys in screen 111 cuitry configured to : detect an input event from acceleration
in a gesture mode . This provides the mechanism for a data received in the wearable electronic input device ; send
flicking gesture . FIG . 35B illustrates switching to a drag operation content from the wearable electronic input device
mode in screen 112 . A circle is displayed with a broken line 40 to a first portable electronic device according to the detected
in a user interface to show a starting point. When a user input event; designate a first screen of the first portable
moves a hand with a wearable input device in the state of electronic device as a screen - to -be - controlled ; and combine
screen 112 , without pressing the operation key , the starting the first screen of the first portable electronic device onto a
point shown with a broken line circle will move , as illus - second screen of a second portable electronic device .
trated in screen 113 of FIG . 35C . If a user presses the 45 ( 2 ) The wearable electronic input device according to ( 1 ) ,
operation key while in the state of screen 112 , a down start wherein the wearable electronic input device comprises a
will be displayed in the user interface . ring configured to be worn on a finger of a user.
In FIG . 35D in the state of screen 114 , while the user has ( 3 ) The wearable electronic input device according to ( 1 )
pressed the operation key of the input device , a drag will or ( 2 ), wherein the wearable electronic input device com
continue when the hand is moved , as illustrated in screen 50 prises a portable hand -held device .
115 of FIG . 35E . In the state of screen 115 , when the user (4 ) The wearable electronic input device according to any
releases the operation key of the input device, the drag o ne of ( 1 ) to ( 3 ), further comprising an operation controlkey
function will be ended with an upward operation . In FIG . configured to detectmovement data and initiate an operation
35F , the broken line circle is displayed on screen 116 to based upon the detected movement data .
indicate the newly created up location . 55 (5 ) The wearable electronic input device according to any
The embodiments illustrated in FIGS. 35A -35F provide a one of (1) to (4 ), further comprising an accelerometer
mechanism for confirming a user interface display , where configured to receive 3 - dimensional linear movement.
the user can recognize each specific event . By using embodi- (6 ) The wearable electronic input device according to any
ments described herein ,multiple terminals can be controlled one of ( 1 ) to (5 ), further comprising a gyroscope configured
with 3 - D input by a single input device . 60 to receive 3 -dimensional rotational movement.
Numerous modifications and variations of the present (7 ) The wearable electronic input device according to any
invention are possible in light of the above teachings. The one of (1 ) to (6 ), further comprising circuitry configured for
embodiments described with reference to the drawings may Bluetooth communication to measure a distance between the
be practiced individually or in any combination thereof. It is wearable electronic input device and a plurality of portable
therefore to be understood that within the scope of the 65 electronic devices , and to designate a screen -to -be-con
appended claims, the invention may be practiced otherwise trolled of one of the plurality of portable electronic devices
than as specifically described herein . based upon the distance.
 US 10 ,055 , 064 B2
25 26
( 8 ) The wearable electronic input device according to any electronic device and the second electronic device are con
one of ( 1 ) to ( 7 ), wherein the circuitry is further configured figured based upon the measured distance .
to initiate user interfaces for the first screen and the second ( 18 ) The electronic input system according to any one of
screen based upon the measured distance. ( 12 ) to ( 17 ), wherein the wearable input device has circuitry
(9 ) The wearable electronic input device according to any 5 configured to receive an inputmode of a rolling gesture , via
one of ( 1 ) to (8 ), wherein the wearable electronic input a gyroscope within the wearable input device .
device has circuitry configured to receive an input mode of ( 19 ) The electronic input system according to any one of
a rolling gesture , via a gyroscope within the wearable (12 ) to ( 18 ), wherein the input event is detected via an
electronic input device .
( 10 ) The wearable electronic input device according to 10 ( 20 ) Thekeyelectronic
operation of the wearable input device.
input system according to any one of
any one of ( 1 ) to ( 9 ) , wherein the circuitry of the wearable ( 12 ) to ( 19 ), wherein the wearable input device is configured
electronic input device is controlled via a cloud structure . to obtain acceleration data and rotational data while the
( 11) The wearable electronic input device according to
any one of (1 ) to ( 10 ), wherein the wearable input device operation key is depressed during the input event.
includes circuitry configured to detect a flicking gesture or 15 (21) The electronic input system according to any one of
a drag gesture input event . ( 12 ) to ( 20 ), wherein the circuitry of the wearable input
( 12 ) An electronic input system comprising: a wearable device , the first electronic device , and the second electronic
input device having an accelerometer and circuitry config device is controlled via a cloud structure.
ured for an operation detection mechanism and an input ( 22 ) The electronic input system according to any one of
communication mechanism , wherein the operation detection 20 ( 12 ) to (21) , wherein the first electronic device and the
mechanism acquires acceleration data from the accelerom second electronic device each include one of a portable
eter to detect an input event; a first electronic device having electronic device , a TV electronic device , or a head -mounted
circuitry for a first communication mechanism , a first event electronic device .
analyzer, a screen display , and a first data communication ( 23 ) The electronic input system according to any one of
mechanism ; and a second electronic device having circuitry 25 ( 12 ) to (22 ), wherein the wearable input device includes
configured for a second communication mechanism , a sec - circuitry configured to detect a flicking gesture or a drag
ond event analyzer, a second data communication mecha gesture input event .
( 24 ) A method of communicating an event between elec
nism , a screen combining mechanism , and a screen display.
The input communication mechanism , the first communica - tronic portable devices , the method comprising : acquiring
tion mechanism , and the second communication mechanism 30 acceleration data of the event from an accelerometer of a
have circuitry configured to send notification of the input wearable input device; communicating the event to a first
event of the wearable input device . The first event analyzer electronic portable device having a first communication
and the second event analyzer have circuitry configured to mechanism ; analyzing the event via a first event analyzer of
analyze the input event detected by the wearable input the first electronic portable device ; sending screen data from
device . The first data communication mechanism and the 35 a first data communication mechanism of the first electronic
second data communication mechanism have circuitry con portable device to a second data communication mechanism
figured to send and receive screen data to and from one of a second electronic portable device ; and combining a
another. Circuitry of the wearable input device is configured screen of the first electronic portable device with a screen of
to wirelessly send operation content from the wearable input the second electronic portable device onto a display of the
device to a screen -to -be - controlled of the first electronic 40 second electronic portable device .
device or the second electronic device as a switching event,
via a depth direction movement of the wearable input The invention claimed is :
device . Circuitry of the screen combining mechanism is 1. A wearable electronic input device , comprising :
configured to combine a screen of the second electronic circuitry configured to
device with a screen obtained from screen data received 45 detect an input event from acceleration data received in
from the first electronic device . the wearable electronic input device ;
(13 ) The electronic input system according to ( 12 ), send operation content from the wearable electronic
wherein the input communication mechanism , the first com input device to a first portable electronic device
munication mechanism , and the second communication according to the detected input event;
mechanism communicate via Bluetooth communication . 50 designate a first screen of the first portable electronic
( 14 ) The electronic input system according to ( 12 ) or (13 ), device as a screen -to -be- controlled ; and
wherein the first data communication mechanism and the combine the first screen of the first portable electronic
second data communication mechanism communicate via device with a second screen of a second portable
wireless fidelity (Wi- Fi) communication . electronic device , select a combination of the first
( 15 ) The electronic input system according to any one of 55 screen and the second screen , and transfer a file
( 12 ) to ( 14 ), wherein circuitry of the screen -to -be- controlled between the first and second portable electronic
of the first electronic device or the second electronic device devices in accordance with a portion of the input
is configured to notify a current screen - to -be- controlled of a event in a depth direction of the second screen , the
switch to a new screen -to -be - controlled . file being selected in accordance with the portion of
( 16 ) The electronic input system according to any one of 60 the input event in the depth direction of the second
( 12 ) to (15 ), wherein the wearable input device has circuitry screen , wherein
configured to measure a distance between the wearable input the circuitry is configured for Bluetooth communication
device and either of the first electronic device or the second to measure a distance between the wearable electronic
electronic device located within a vicinity of a received input device and the first and second portable electronic
Bluetooth signal. 65 devices, and to designate the first screen of the first
( 17 ) The electronic input system according to any one of portable electronic device to be the screen -to -be- con
( 12 ) to ( 16 ), wherein user interfaces for screens of the first trolled based upon the distance .
 US 10 ,055 , 064 B2
27 28
2. The wearable electronic input device of claim 1, 12 . The electronic device of claim 11, wherein the elec
wherein the wearable electronic input device comprises a tronic device is a wearable electronic device .
13 . The electronic device of claim 11 , wherein the elec
ring configured to be worn on a finger of a user.
3. The wearable electronic input device of claim 1, tronic device is a ring configured to be worn on a finger of
wherein the wearable electronic input device comprises a 5 aa us14 . The electronic device of claim 11 , wherein the elec
portable hand -held device .
4 . The wearable electronic input device of claim 1 , further tronic device is a portable hand - held device .
15 . The electronic device of claim 11, further comprising:
comprising: a user interface configured to detect movement data and
an operation control key configured to detectmovement initiate an operation based upon the detected movement
data and initiate an operation based upon the detected 10 data .
movement data . 16 . The electronic device of claim 11 , further comprising:
5 . The wearable electronic input device of claim 1, further an accelerometer configured to detect 3 - dimensional lin
comprising:
ear movement.
an accelerometer configured to detect 3 -dimensional lin 15 17 . The electronic device of claim 11 further comprising:
ear movement . a gyroscope configured to detect 3 -dimensional rotational
6 . The wearable electronic input device of claim 1 , further movement .
comprising: 18 . The electronic device of claim 11 , wherein the cir
a gyroscope configured to detect 3 -dimensional rotational cuitry is configured to initiate user interfaces for the first
movement.
7 . The wearable electronic input device of claim 1. 20 screen and the second screen based upon the measured
wherein the circuitry is further configured to initiate user distance .
19. The electronic device of claim 11 , wherein the elec
interfaces for the first screen and the second screen based tronic
upon the measured distance . device includes circuitry configured to detect a rolling
8 . The wearable electronic input device of claim 1 , gesture via a gyroscope within the electronic device .
wherein the wearable electronic input device includes cirin - 2525 20 . The electronic device of claim 11 , wherein the cir
cuitry is controlled via a cloud structure.
cuitry configured to detect an inputmode of a rolling gesture
via a gyroscope within the wearable electronic input device . 21. The electronic device of claim 11 , wherein the cir
9 . The wearable electronic input device of claim 1, cuitry is configured to detect a flicking gesture or a drag
wherein the circuitry of the wearable electronic input device gesture input event.
is controlled via a cloud structure . 30 80 22 . The electronic device of claim 11, wherein the cir
10 . The wearable electronic input device of claim 1 , cuitry is conngured
wherein the wearable input device includes circuitry con control the combined first and second screens to be
figured to detect a flicking gesture or a drag gesture input displayed by another electronic device in an overlap
event. ping manner in order based on the measured distance .
11 . An electronic device , comprising : 35 23 . An electronic device , comprising:
circuitry configured to circuitry configured to
measure a distance between the electronic device and a determine a distance between the electronic device and
plurality of portable electronic devices based on a plurality of portable electronic devices based on
short- range wireless communication performed Bluetooth communication performed between the
between the electronic device and the plurality of 40 electronic device and the plurality of portable elec
portable electronic devices ; tronic devices ;
designate a first screen of one of the plurality of designate a first screen of one of the plurality of
portable electronic devices as a screen - to -be- con portable electronic devices as a screen -to -be - con
trolled based on the measured distance ; trolled based on the determined distance ;
detect an input event based on acceleration data 45 detect an input event at the electronic device ;
detected at the electronic device ; send operation content from the electronic device to the
send operation content from the electronic device to the one of the plurality of portable electronic devices
one of the plurality of portable electronic devices according to the detected input event, wherein
according to the detected input event, wherein the first screen of the one of the plurality of portable
the first screen of the one of the plurality of portable 50 electronic device is combined with a second screen of
electronic device is combined with a second screen of a second one of the plurality of portable electronic
a second one of the plurality of portable electronic devices, a combination of the first screen and the
second screen is selected , and a file is transferred
devices, a combination of the first screen and the between the first and second portable electronic devices
second screen is selected , and a file is transferred
between the first and second portable electronic devices 55 in accordance with a portion of the input event in a
in accordance with a portion of the input event in a depth direction of the second screen , the file being
depth direction of the second screen , the file being selected in accordance with the portion of the input
selected in accordance with the portion of the input event in the depth direction of the second screen .
event in the depth direction of the second screen . * * * * *