Digital Speech Processing

數位語音處理概論

李琳山
Speech Signal Processing

x(t) x[n]
Processing
LPF Algorithms
output

• Major Application Areas • Speech Signals

1. Speech Coding:Digitization and Compression – Carrying Linguistic
Knowledge and Human
x[n] ^
x[n] Information: Characters,
Processing xk Inverse Words, Phrases, Sentences,
110101… Processing Concepts, etc.
Storage/transmission – Double Levels of
Information: Acoustic
Considerations : 1) bit rate (bps)
Signal Level/Symbolic or
2) recovered quality
Linguistic Level
3) computation
complexity/feasibility – Processing and Interaction
of the Double-level
2. Voice-based Network Access — Information
User Interface, Content Analysis, User-content
Interaction
Sampling of Signals

X[n]
X(t)

t n
Double Levels of Information

字(Character)
詞(Word)
人人用電腦
句(Sentence)
Speech Signal Processing – Processing of Double-Level
Information
• Speech Signal • •
Sampling Processing

今 天 的
Algorithm

天 氣 非
Chips or Computers

• Linguistic
常 好 Structure

• Linguistic Knowledge
今天的 天氣 非常 好

Lexicon Grammar
今天 的
Well-Known Application Examples of Speech and
Language Technologies – Speaking Personal Assistant
• Examples • Special Questions:
– Weather in New York next week ? – 唐詩宋詞, 出師表…
– Who is the president of US ? What did he say today ? – 說個笑話…
– How can I go to National Taiwan University ?
– Short messaging, personal scheduling, etc.

Speech Language
Synthesis Generation
Output Speech
Signals

Speech Language Dialogue

Recognition Understanding Manager
Input Speech
Signals

Information Knowledge Question Machine

Wikipedia
Retrieval Graph Answering Translation

• Examples:
– Siri (Apple), Google Now (Google), Cortana (Microsoft)
Voice-based Network Access

Internet

User Content Analysis

Interface

User-Content
Interaction
 User Interface
—when keyboards/mice inadequate
 Content Analysis
— help in browsing/retrieval of multimedia content
 User-Content Interaction
—all text-based interaction can be accomplished by spoken language
User Interface —Wireless Communications Technologies
have Created a Whole Variety of User Terminals

Text Content

Internet

Multimedia
 at Any Time, from Anywhere Content
 Smart phones, Hand-held Devices, Notebooks, Vehicular Electronics, Hands-
free Interfaces, Home Appliances, Wearable Devices…
 Small in Size, Light in Weight, Ubiquitous, Invisible…
 Post-PC Era
 Keyboard/Mouse Most Convenient for PC’s not Convenient any longer
— human fingers never shrink, and application environment is changed
 Service Requirements Growing Exponentially
 Voice is the Only Interface Convenient for ALL User Terminals at Any Time,
from Anywhere, and to the point in one utterance
 Speech Processing is the only less mature part in the Technology Chain
Content Analysis—Multimedia Technologies have Created
a World of Multimedia Content

Internet
Real–time
Information Private Services
– weather, traffic – personal notebook
Intelligent Working
– flight schedule Environment – business databases
Knowledge Special Services – home appliances
– stock price – e–mail processors
Archieves – Google – network
– sports scores – intelligent agents
– digital libraries – Facebook entertainments
– virtual museums – teleconferencing
–YouTube – distant learning
– Amazon – electric commerce

• Most Attractive Form of the Network Content is Multimedia, which usually Includes
Speech Information (but Probably not Text)
• Multimedia Content Difficult to be Summarized and Shown on the Screen, thus
Difficult to Browse
• The Speech Information, if Included, usually Tells the Subjects, Topics and Concepts of
the Multimedia Content, thus Becomes the Key for Browsing and Retrieval
• Multimedia Content Analysis based on Speech Information
User-Content Interaction — Wireless and Multimedia
Technologies are Creating An Era of Network Access by Spoken
Language Processing
text
Text-to-Speech information
Text
voice Synthesis Content
information Multimedia
Content
Spoken and Voice-based Internet
multi-modal Information
Dialogue Retrieval
Multimedia
Text Information Content
Retrieval Analysis

• Hand-held Devices with Multimedia Functionalities Commonly used Today

• Network Access is Primarily Text-based today, but almost all Roles of Texts can be
Accomplished by Speech
• User-Content Interaction can be Accomplished by Spoken and Multi-modal Dialogues
• Using Speech Instructions to Access Multimedia Content whose Key Concepts Specified
by Speech Information
Voice-based Information Retrieval

Voice Instructions
Text Instructions
請問鼎泰豐的地址？

Text Information
d1
Voice Information
d1 d2

d2 d3

d3 鼎泰豐台北101分店在…

•Both the User Instructions and Network Content Can be in form of

Speech
Spoken and Multi-modal Dialogues
• Almost All User-Content Interaction can be Accomplished by
Spoken or Multi-modal Dialogues

Sentence Generation
Users Internet
and Speech Synthesis
Output
Speech Response to
the user

Wireless Discourse Dialogue

Context Manager Databases
Networks
User’s
Intention
Dialogue
Input
Speech Recognition Server
Speech
and Understanding
Outline
• Both Theoretical Issues and Practical Problems will be Discussed
• Starting with Fundamentals, but Entering Research Topics in the Second Half
• Part I: Fundamental Topics
1.0 Introduction to Digital Speech Processing
2.0 Fundamentals of Speech Recognition
3.0 Map of Subject Areas
4.0 More about Hidden Markov Models
5.0 Acoustic Modeling
6.0 Language Modeling
7.0 Speech Signals and Front-end Processing
8.0 Search Algorithms for Speech Recognition
• Part II: Advanced Topics
9.0 Speech Recognition Updates
10.0 Speech-based Information Retrieval
11.0 Spoken Document Understanding and Organization for User-content Interaction
12.0 Computer-assisted Language Learning(Call)
13.0 Speaker Variabilities: Adaption and Recognition
14.0 Latent Topic Analysis
15.0 Robustness for Acoustic Environment
16.0 Some Fundamental Problem-solving Approaches
17.0 Spoken Dialogues
18.0 Conclusion
References
• 教科書：無
• 主要參考書：
1. X. Huang, A. Acero, H. Hon, “Spoken Language Processing”, Prentice
Hall, 2001,松瑞
2. F. Jelinek, “Statistical Methods for Speech Recognition”, MIT Press,
1999
3. L. Rabiner, B.H. Juang, “Fundamentals of Speech Recognition”,
Prentice Hall, 1993, 民全
4. C. Becchetti, L. Prina Ricotti, “Speech Recognition- Theory and C++
implementation”, Johy Wiley and Sons, 1999, 民全
5. D. Jurafsky, J. Martin, “Speech and Language Processing- An
Introduction to Natural Language Processing, Speech Recognition, and
Computational Linguistics, 2nd edition”, Prentice-Hall, 2009
6. G. Tur, R. De Mori, “Spoken Language Understanding- Systems for
Extracting Semantic Information from Speech”, John Wiley & Sons,
2011
7. 其他參考文獻課堂上提供
Other Information
• 教材：
available on web before the day of class (http://speech.ee.ntu.edu.tw)
• 適合年級：三、四（電機系、資工系）
• 成績評量方式
Midterm Exam 25%
Homeworks (I) (II) (Ⅲ) 15%、5%、15％
Final Exam 10%
Term Project 30%
Goals
• 課程目的：
提供同學進入此一充滿機會與挑戰的新領域所需的基本知識，體
驗數學模型與軟體程式如何相輔相成，學習進入一個新領域由基
礎進入研究的歷程，體會吸收非結構性知識(Unstructured
Knowledge)的經驗

• Unstructured Knowledge • Math & Programming

A B C
Mathematical
Programming
Models

D Hardware
 1.0 Introduction — A Brief Summary of Core
Technologies and Example Application Seenarios

References for 1.0

1.“Speech and Language Processing over the Web”, IEEE Signal
Processing Magazine, May 2008
Speech Recognition as a pattern recognition problem

X W
x(t)
Feature Pattern Decision
Extraction Matching Making
unknown output
speech feature word
signal vector
sequence

y(t) Y
Feature Reference
Extraction Patterns
training
speech
Basic Approach for Large Vocabulary Speech Recognition

• A Simplified Block Diagram

Input Speech Feature Output
Linguistic Decoding
Front-end Vectors Sentence
and
Signal Processing Search Algorithm

Acoustic Language
Speech Model Acoustic Language Text
Lexicon Model
Corpora Training Models Model Corpora
Construction

• Example Input Sentence

this is speech
• Acoustic Models (聲學模型)
(th-ih-s-ih-z-s-p-ih-ch)
• Lexicon (th-ih-s) → this
(ih-z) → is
(s-p-iy-ch) → speech
• Language Model (語言模型) (this) – (is) – (speech)
P(this) P(is | this) P(speech | this is)
P(wi|wi-1) bi-gram language model
P(wi|wi-1,wi-2) tri-gram language model,etc
Speech Recognition Technologies, Applications and
Problems
• Word Recognition
– voice command/instructions
• Keyword Spotting
– identifying the keywords out of a pre-defined keyword set from input voice
utterances
• Large Vocabulary Continuous Speech Recognition
– entering longer texts
– remote dictation/automatic transcription
• Speaker Dependent/Independent/Adaptive
• Acoustic Reception/Background Noise/Channel Distortion
• Read/Spontaneous/Conversational Speech
Text-to-speech Synthesis
• Transforming any input text into corresponding speech signals
• E-mail/Web page reading
• Prosodic modeling
• Basic voice units/rule-based, non-uniform units/corpus-based, model-
based

Lexicon Prosodic Voice Unit

and Rules Model Database

Output
Input Speech
Text Text Analysis Signal Signal
and Letter-to- Prosody Processing
sound Generation and
Conversion Concatenation
Speech Understanding
• Understanding Speaker’s Intention rather than Transcribing into
Word Strings
• Limited Domains/Finite Tasks

acoustic phrase concept phrase/concept

models lexicon set language model

input understanding
utterance Syllable syllable lattice Key Phrase phrase graph results
Semantic
Recognition Matching concept graph Decoding

• An Example
utterance: 請幫我查一下 台灣銀行 的 電話號碼 是幾號?
key phrases: (查一下) - ( 台灣銀行) - (電話號碼)
concept: (inquiry) - (target) - (phone number)
Speaker Verification
• Verifying the speaker as claimed
• Applications requiring verification
• Text dependent/independent
• Integrated with other verification schemes

input
speech yes/no
Feature
Verification
Extraction

Speaker
Models
Voice-based Information Retrieval
• Speech Instructions
• Speech Documents (or Multi-media Documents including Speech
Information)

speech instruction
text instruction
請問鼎泰豐的地址？

text documents
speech documents d1
d1 d2
d2 d3
d3 鼎泰豐台北101分店在…

• Locate exactly the desired utterances

• Text descriptions not needed for indexing/retrieving purposes
Spoken Dialogue Systems
• Almost all human-network interactions can be accomplished by spoken
dialogue
• Speech understanding, speech synthesis, dialogue management
• System/user/mixed initiatives
• Reliability/efficiency, dialogue modeling/flow control
• Transaction success rate/average number of dialogue turns
Sentence Generation
Users Internet
Output and Speech Synthesis
Speech Response
to the user
Discourse Dialogue
Networks Context Manager Databases

User’s
Intention Dialogue
Input
Speech Speech Recognition Server
and Understanding
Spoken Document Understanding and Organization
• Unlike the Written Documents which are easily shown on the
screen for user to browse and select, Spoken Documents are just
Audio Signals
— the user can’t listen each one from the beginning to the end during browsing
— better approaches for understanding/organization of spoken documents becomes
necessary
• Spoken Document Segmentation
— automatically segmenting a spoken document into short paragraphs, each with
a central topic
• Spoken Document Summarization
— automatically generating a summary (in text or speech form) for each short
paragraph
• Title Generation for Spoken Documents
— automatically generating a title (in text or speech form) for each short paragraph
• Key Term Extraction and Key Term Graph Construction for
Spoken Documents
— automatically extracting a set of key terms for each spoken document, and
constructing key term graphs for a collection of spoken documents
• Semantic Structuring of Spoken Documents
— construction of semantic structure of spoken documents into graphical hierarchies
Multi-lingual Functionalities
• Code-Switching Problem
– English words/phrases inserted in spoken Chinese sentences as an example
人人都用Computers，家家都上Internet
OK不OK？OK啦!
– the whole sentence switched from Chinese to English as an example
準備好了嗎？Let’s go!
• Cross-language Information Processing
– globalized network with multi-lingual content/users
– cross-language network information processing with a certain input language
• Dialects/Accents
– hundreds of Chinese dialects as an example
– code-switching problem─ Chinese dialects mixed with Mandarin (or plus
English) as an example
– Mandarin with a variety of strong accents as an example
• Global/Local Languages
• Language Dependent/Independent Technologies
• Code-Switching Speech Processing, Speech-to-speech Translation,
Computer-assisted Language Learning
Computer-Assisted Language Learning
 Globalized World
– every one needs to learn one or more languages in addition
to the native language
 Language Learning
– one-to-one tutoring most effective but with high cost
 Computers not as good as Human Tutors
– software reproduced easily
– used repeatedly any time, anywhere
– never get tired or bored
 Learning of
– pronunciation, vocabulary, grammar, sentences, dialogues,
etc.
– sometimes in form of games