1

Integrating VoiceXML with SIP services

Kundan Singh, Ajay Nambi and Henning Schulzrinne
Department of Computer Science, Columbia University
{kns10,an2029,hgs}@cs.columbia.edu

Abstract— We describe our Session Initiation Protocol on the web server or dynamically generated based on some
(SIP)-based VoiceXML browser, sipvxml, that allows pro- server side programming logic like HTTP-CGI (Common
gramming interactive voice response applications that are
accessible from telephones as well as IP phones. We also Gateway Interface), Java servlet or Java server pages. The
describe how we have used sipvxml in our multi-party mul- media files can either be stored on the web server or can
timedia conferencing server. We propose other applications be streamed in real-time from an RTSP [10] media server,
and extensions that can benefit from this technology in our
IP telephony test bed. (keywords: Internet telephony; in-
such as rtspd, directly to the SIP caller using RTP [8].
teractive voice response; SIP; VoiceXML; sipvxml)
PSTN SIP-PSTN
gateway Fetch
I. Introduction Telephone VoiceXML
pages Web server
(CGI, Servlet, JSP)
People are familiar with traditional interactive voice re- Call request
sponse (IVR) systems found in voice mail access, dial- Get streaming media
in conferences, phone-based customer support and tele- IP soft-phone SIP based
VoiceXML
banking. VoiceXML is an XML-based language developed browser
by the W3C [1] to create voice dialogs that feature syn- Media server
IP hardware phone
thesized speech, digitized audio, recognition of spoken and (media files)

DTMF key input and recording of audio for telephony Fig. 1. Example sipvxml scenario
applications. It enhances the traditional proprietary and
closed IVR systems to an open programmable architecture.
It brings the advantage of web technologies to a telephony A. VoiceXML page
user by providing programmable dialogs, similar to HTML The following example VoiceXML page prompts the
forms or CGI scripts. caller with spoken audio: “Enter the ZIP code ...”. When
the user presses a sequence of digits, say 10027#, the vari-
The Session Initiation Protocol (SIP [7]) is an Inter- able zipcode gets the value “10027” that gets passed to
net telephony signaling protocol used for establishing and the URL http://myserver.com/weather.cgi?zipcode=10027.
terminating Internet multimedia sessions. A SIP-based It is up to the script weather.cgi to process the input and
VoiceXML browser (or SIP-VoiceXML browser) allows a generate further VoiceXML pages. If there is some error or
user doesn’t press anything, then the prompt is repeated.
SIP user to take part in application-specific IVR systems,
e.g., voice mail or tele-banking. It also brings the advantage <?xml version="1.0"?>
<vxml version="1.0">
of VoiceXML technology to a regular telephone user via a <form>
SIP-PSTN gateway. We have developed a SIP-VoiceXML <field name="zipcode">
browser, sipvxml, to enhance the services of our CINEMA <prompt>Enter the ZIP code of the location for which you
want weather information.</prompt>
test-bed [3], [4], [11]. In particular, we have extended </field>
our multimedia conferencing server, sipconf [12], and uni- <catch event="noinput error help">
fied messaging (voicemail) server, sipum [13] to provide en- Enter the ZIP code again followed by the pound key.
hanced services and convenience to a telephone user. </catch>
<block>
We describe the architecture of sipvxml in Section II. <submit
Section III describes use of sipvxml with our conferenc- next="http://myserver.com/weather.cgi" namelist="zipcode"/>
</block>
ing server. More examples of SIP services enabled by
</form>
VoiceXML are described in Section IV. We list some other </vxml>
related work in Section V and conclude in Section VI.
We have implemented a very simple DTMF grammar. A
typical explicit dtmf tag in the VoiceXML page looks like:
II. Architecture
<dtmf type="application/x-dtmf">
A SIP-VoiceXML browser is similar to a web browser for 1 | 2 | 3 | 4 | *
</dtmf>
a telephone instead of a desktop PC. The browser fetches
the VoiceXML pages or pre-recorded media files from a web The MIME type for this grammar is “application/x-dtmf”.
server and presents an interactive dialog to the telephone Input is either a fixed length string or terminated by a “#”.
user. Fig. 1 shows an example scenario where the browser An implicit timeout of 5 seconds is implemented so that the
can be accessed from SIP phones as well as a regular tele- input is automatically accepted if the user does not press
phone. The VoiceXML pages can either be statically stored the terminating “#” key for some time. If no grammar is
 2

specified, then the interpreter will accept any input. Users On the other hand, if the request-uri is
can press “**#” anytime to signal the help event. sip:[email protected], then the interpreter
We have implemented only a subset of VoiceXML tags is invoked with the default pre-configured initial
as needed in our application: assign, audio, block, catch, VoiceXML URL, e.g., that of the conferencing service.
clear, disconnect, dtmf, error, exit, field, filled, form, goto, 2. The interpreter thread calls the XML parser with the
help, noinput, nomatch, prompt, submit, value, var and vxml. initial URL.
We do not support any client side script (e.g., JavaScript) 3. The XML parser fetches the page from the web server
usually needed for arithmetic or string operations in the or a local file system (based on the initial URL).
browser, as the same effect can be achieved using server 4. It presents the returned XML document into a tree
side processing. data structure.
5. The interpreter thread invokes the Form Interpreta-
B. Operation of the browser tion Algorithm (FIA [1]) on the selected form from the
VoiceXML document.
SIP interface 6. FIA invokes various other modules based on the con-
SIP On new incoming
XML
(3) Web
server
tent of the VoiceXML document. For example, it
INVITE SIP call
(1) (2) parser
can invoke the text-to-speech SDK to synthesize any
Interpreter
thread
(4)
prompts. The current implementation does not use
RTP any speech recognition engine because user input is
interface
RTP/RTCP RTP
(12) Detect
DTMF
(5)
via touch-tone keys.
receive
thread (14)
(13)
7. FIA can also invoke the HTTP fetcher module to fetch
Speech Grammar
(11)
Form an external grammar file or a media file for an audio
interpretation
recognition matching
rules (15) algorithm prompt. XML parser internally has its own HTTP
(7) client to fetch VoiceXML pages.
(6) (8)
RTP
send (9) Http Web 8. The HTTP fetcher implements a simple HTTP GET
fetcher server
RTP/RTCP
(16)
thread
text to
method to retrieve a document.
(10) speech SDK 9. The media file retrieved from the web server using
HTTP fetcher is fragmented into 20 ms packets for
Fig. 2. Operation of sipvxml
interactive telephony, and enqueued for streaming out
to the caller by the send thread.
Fig. 2 shows the components of our SIP-VoiceXML
10. The speech synthesizer output is also fragmented
browser, sipvxml. We use our SIP library1 for implement- and enqueued for sending out to the caller.
ing a SIP interface, the RTP library2 for the RTP/RTCP 11. The VoiceXML document can specify the grammar
interface, the Apache’s XML parser3 with DOM interface,
rules in various scopes in the document. FIA can set
an HTTP fetcher4 for getting non-XML pages and IBM the active grammar for the matching engine based on
ViaVoice Text-To-Speech SDK5 for speech synthesis. the current execution scope in the VoiceXML page.
1. When the browser receives a new incoming SIP call 12. The RTP receive thread receives the RTP media
it creates three different threads: RTP receive thread, packets and invokes the DTMF detector.
RTP send thread, and the VoiceXML interpreter 13. Any detected DTMF digit is passed to the grammar
thread. The RTP receive thread receives media matching engine.
packets from the caller and invokes the DTMF 14. DTMF tones can be transported from the caller to
detection module. The RTP send thread streams the browser in a number of ways. One approach is to
out media packets to the caller. A separate send not distinguish them from the spoken voice by encod-
thread helps in maintaining the constant band- ing them using the same audio codec. However, a low
width (e.g., 64 kb/s for G.711 audio) for outgoing bandwidth audio codec may distort the properties of
packets and irrespective of the speed of the speech the in-band DTMF tones making them hard to detect.
synthesizer. The initial VoiceXML page URL can A second, preferred way is to use “telephone-event” [9]
be preconfigured in the browser or encoded in containing the digit codes instead of the encoded audio
the SIP request [6]. For example, if the caller in RTP packets. In the first case, the browser has to do
dials sip:dialog.vxml.http%3a//dialogs.server.com/ the DTMF detection, whereas in the second case the
[email protected] then the call will caller or the gateway has to do the DTMF detection.
reach the browser running at vxmlservers.com The RTP receive module forwards telephone-events di-
and it will fetch the initial VoiceXML page from rectly to the grammar matching engine. We have im-
http://dialogs.server.com/script32.vxml. plemented both these methods. A third method of
transporting DTMF in SIP INFO message is not used
1 http://www.cs.columbia.edu/˜ kns10/software/siplib
2 http://www.cs.columbia.edu/˜
in our implementation.
hgs/rtp/rtp-library.html
3 http://xml.apache.org 15. The grammar matching engine tries to match the
4 http://cs.nmu.edu/˜ lhanson/http fetcher/ received digits with any active grammar, and informs
5 http://www-4.ibm.com/software/speech/dev/ttssdk linux.html
 3

the FIA if a match is found. 7. The browser again checks if Alice is allowed to join
16. The RTP send thread periodically sends media pack- the conference identified by number 23, which in this
ets to the caller. No packets are sent during silence. example is sip:[email protected].
8. Once the authentication is done, the browser transfers
III. Multi-party Conferencing the call to the actual conference server using the SIP
REFER method [14] containing the SIP URI of the
Consider a SIP conferencing system where users
conference.
join the conference by dialing in a conference URI
9. Alice’s phone accepts the transfer and initiates a new
sip:[email protected]. A regular telephone user
call to the conference server.
with only a touch-tone phone cannot dial such a generic
10. Alice’s phone exchanges audio with the conference
URI. We can assign one phone number per conference for
server directly, without going through the browser.
direct inward dialing. However, it is preferred that the
user always dials the number of the VoiceXML browser Note that the user authentication, conference look up
and transfer are actually invoked by the conference ser-
that in turn prompts him for the authentication PIN (per- vice CGI scripts, whereas the browser just interprets the
sonal identification number) and conference number. Once VoiceXML pages generated by the scripts to do the actual
the user is authenticated the browser transfers the call to transfer or prompt the caller. For instance, the service
the selected conference. One can also use a single PIN to script may generate the following transfer tag for the call
transfer in step (9).
identify both the participant as well as the conference.
<block><prompt>Your call is being transferred,
(a) Message flow please wait.</prompt></block>
<transfer dest="sip:[email protected]" bridge="false" />
(b) Architecture
User SIP phone VoiceXML browser (4) (7) The transfer can be done in two modes: blind and
(1) INVITE sip:[email protected] bridged. The former is the transfer of the call to the con-
(1)
200 OK (accepted) ference server without consulting the server whereas the
ACK (cnfirmed) (8) latter is the transfer after consulting such that the browser
(2) Welcome, please enter your (9) may choose to be in the media path. We have implemented
four digit PIN code. Database the blind call transfer as shown in Fig. 3.
(3) 1-2-3-4-# (4) user auth/
identification 1234=>Alice (a) Message flow (b) Architecture

(5) enter the conference

identifier you want to join

(6) 2-3-# User SIP phone VoiceXML browser

(7) is user
allowed to join? 23=>staffmeet Call request+authentication (8) INVITE sip:[email protected] (4) (7)
Alice is allowed (1)-(7) with credentials

(8) REFER to sip:[email protected] 202 Accepted (1)

with credentials
(9) User talks/listens to the browser
202 Accepted (8)
Conference server
BYE BYE (user ends the call)
BYE
200 OK (call closed) 200 OK (call closed)
200 OK
(9) INVITE sip:[email protected]
200 OK (accepted)

Fig. 4. Method 2: Joining conference using bridged mode

(10) User talks/listens to the conference server directly

BYE (user ends the call)

200 OK (call closed) Fig. 4 shows the bridged transfer case with the browser
as a “back-to-back-user-agent” bridging the audio path be-
Fig. 3. Method 1: Joining conference in blind transfer mode
tween the user phone and the conference server. Steps
1 to 7 are same as the blind transfer case. Instead
of sending REFER, the browser sends a new call re-
Fig. 3 shows how an user, say Alice, interacts with the
quest to the conference server identifying the conference
browser before joining the conference.
sip:[email protected] in the Request-URI of the SIP
1. Alice dials the browser’s phone number (212-9397137) INVITE message. The browser acts as an application level
or SIP URI (sip:[email protected]). packet forwarder in both directions for RTP/RTCP media
2. The browser accepts the call and prompts the caller traffic.
to enter the PIN for identification. The advantage of bridged transfer is that the browser
3. Alice keys in her PIN, 1-2-3-4, followed by a termi- remains in the media path and can accept future con-
nating # key. The DTMF digits are sent in RTP. trol commands (using DTMF) from the user phone. For
4. The browser looks up the database and identifies the conferencing, it may be useful to interpret DTMF, e.g.,
caller as “Alice”. 6-6-# to mute your audio or 6-8-# to join another vir-
5. Based on the privileges, the browser prompts her with tual chat/conference room. Secondly, the browser needs
a list of conferences to choose from. to forward other signaling messages also, e.g., re-INVITE
6. Alice picks up the conference with identifier 23. from the caller to the conference server. Moreover, main-
 4

taining packet forwarding states for the duration of the Java Servlet. These programs generate VoiceXML pages
conference limits the scalability of the browser on how based on the mail box content and user input. More infor-
many simultaneous callers it can handle. The browser mation can be found at [5].
may issue re-INVITEs with updated transport addresses
for RTP/RTCP to both the caller and the conference server C. Event notification and scheduling
such that the media path is direct. However, this still needs Asynchronous event notification is useful when polling
the signaling state to be maintained for the duration of the for the event is inefficient. For example, the email-by-phone
call. On the other hand, a blind transfer does not require system can be modified to notify the user of any important
any call state in the browser for the duration of the con- email by calling user’s cell phone. Text-to-speech is used to
ference. But it expects that the caller’s IP phone supports play out the email content on the phone. Alternatively, the
the REFER method. user can go to a web page and schedule a birthday reminder
or wakeup call by recording his own audio announcement
IV. Other services or a text message. The system notifies the user by phone
This section describes some of the current and future at the scheduled time. These simple systems do not need
services that can be provided using VoiceXML in our SIP VoiceXML. However, a VoiceXML browser is needed to al-
environment. low the user to schedule events from phone, or to “snooze”
and notify again after a short while. We are extending
A. Unified messaging – voice mail sipvxml to allow initiating a new call for notification.
Sipum is a SIP/RTSP-based unified messaging sys- D. Audio volume level for conference
tem that provides a centralized voice mail and answer-
ing machine service. For example, when Alice calls Multi-party audio conferencing among heterogeneous
[email protected], the SIP server on cs.columbia.edu do- clients with different audio devices causes annoying distor-
main forks the call request to both Bob’s IP phone and tion of audio. Some participants are heard very loud and
the answering machine (sipum). If Bob picks up the phone, some are not heard at all. Ideally, the conference server
the call request to sipum is cancelled. If Bob does not pick should balance the input audio level from the participants
up the phone after 10 seconds, sipum accepts the call on before mixing. However this imposes additional processing
Bob’s behalf and prompts the caller, Alice, to leave any requirement on the server for every audio packet. Another
voice message. The same application, sipum, is also used approach is to tell the participant to adjust his volume
by Bob to retrieve his voice mails, for example, by dialing a level for both microphone and speaker. The participant
URL sip:[email protected] to re- connects to a “audio level feedback” system before joining
trieve the message with ID 672. There are other ways of the conference and speaks into it. The systems announces
retrieving voice mails, for instance, using a web browser or if the user’s microphone volume is acceptable, too high or
a media client. However none of these are appropriate for a too low. The system also plays back a pre-recorded au-
telephone user with limited touch tone capability. We use dio file and allows the user to adjust his speaker volume.
sipvxml with application-level logic for voice mail service to This processing is built in a server side CGI script that is
allow more interactive interface to access and manage the accessible via a VoiceXML browser.
voice mail box. From a user’s perspective this is similar to
E. Advanced conference control
the traditional voice mail service. However, use of SIP and
VoiceXML allows easy integration with web, email, instant Our current conference server implementation provides
messaging, and telephone. a web interface for floor control by the moderator and par-
The application logic to perform voice mail service is ticipant list display. We can extend it such that conference
built as CGI scripts executed in the browser’s context. control can be done using the same telephone that the par-
Once the user is authenticated using PIN, the main menu is ticipant or moderator is using for the conference.
spoken out. This includes options for playing out new voice
messages and other details like sender, subject and times- F. Integrating speech recognition
tamp of the message. An RTSP media server (e.g., rtspd) Our current implementation accepts user input only via
can be used to stream the actual voice message directly DTMF digits. VoiceXML is designed for spoken audio in-
to the caller phone using RTP. We also provide additional put as well as DTMF. Allowing both mechanisms will im-
options like saving or deleting the message. prove the user experience.

B. Email by phone G. Security

Email is one of the most convenient forms of communi- Every Internet system should deal with security. Our ar-
cation. However, convenience is limited by the necessity of chitecture has three places where security needs to be con-
an Internet connected computer. Our “Email By Phone” sidered: telephony gateway, SIP signaling and RTP media
system solves this problem by allowing people to check and transport, and HTTP/RTSP access to the backend servers.
send their emails over any telephone. The server side ap- Telephony gateway and SIP security are explained in the
plication logic is implemented using Java Server Pages and CINEMA technical report [11]. HTTP and RTSP can use
 5

simple shared secret (Digest authentication [2]) or more VI. Conclusions

secure Transport Layer Security (TLS). VoiceXML is a powerful technology that allows telephone
In the bridged transfer case, sipvxml authenticates the users to access services that are typically available to web
caller using PIN and provides its own credentials to the users. SIP interface to a VoiceXML browser allows such
conference server in SIP authnetication. In the blind trans- services from a IP telephone as well as a regular telephone,
fer case, if the caller is using a regular phone connected via via a gateway, using the call transfer feature in an interop-
a gateway, the SIP authentication will contain the creden- erable manner. We have implemented a simple SIP based
tials of the gateway. To prevent malicious users connecting VoiceXML browser and used it to allow telephone users to
to a restricted conference, we pass the credentials from connect to our conferencing server. This along with other
sipvxml to the caller’s gateway in the REFER message, services that we have implemented will help users to join a
which in turn is used by the caller’s gateway to call the con- conference, check mails and stay informed from anywhere
ference server. The credentials can be that of the browser thus enabling ubiquitous availability. We are extending our
or the caller. For example, Refer-To header may contain IP telephony test bed to incorporate various voice based
sip:staffmeet-<timestamp:hash>@conference.com, services like accessing unified messages, event notification
where <timestamp:hash> is base-64 encoded and hash and scheduling, email notification and conference control
is MD5 hash of “browser-host:timestamp:shared-secret”. via a telephone.
The scheme works only when the conference server can
interpret this URI. The server should reject the call if the VII. Acknowledgements
timestamp is old, to prevent replay attack. Daniel Liu and Naho Ogasawara implemented Email
The browser should use HTTP POST method, instead by phone using Tellme studio. Sean Mandel and Visda
of GET, to avoid storing CGI input in the web server log. Vokhshoori implemented parts of sipvxml. The work is sup-
ported by grants from SIP Communications, Inc.
V. Related work References
[1] VoiceXML specification. http://www.w3c.org/voice/.
The Voice Browser working group of world wide web con- [2] J. Franks, P. Hallam-Baker, J. Hostetler, S. Lawrence, P. Leach,
sortium (W3C) is improving the VoiceXML [1] specifica- A. Luotonen, and L. Stewart. HTTP authentication: Basic and
tion. VoiceXML applications for interactive voice response digest access authentication. RFC 2617, Internet Engineering
Task Force, June 1999.
are developed by many commercial organizations. Plum [3] W. Jiang, J. Lennox, S. Narayanan, H. Schulzrinne, K. Singh,
Voice Portal Technology6 can present existing websites or and X. Wu. Integrating internet telephony services. IEEE In-
intranet applications to a phone user. It can also deliver ternet Computing, 6(3):64–72, May 2002.
[4] W. Jiang, J. Lennox, H. Schulzrinne, and K. Singh. Towards
follow-up information via email or fax. Open VXI7 is an junking the PBX: deploying IP telephony. In Proc. International
open source VoiceXML interpreter. IBM’s WebSphere pro- Workshop on Network and Operating System Support for Digital
vides HTML-to-VoiceXML transcoding that can be con- Audio and Video (NOSSDAV), Port Jefferson, New York, June
2001.
verted to speech by a VoiceXML browser. Talking E-Mail8 [5] D. Liu and N. Ogasawara. Project:
allows users to access emails from various interfaces includ- Email by phone using voicexml, May 2001.
ing voice, i-mode, Web and WAP. None of these applica- http://www.cs.columbia.edu/~kns10/projects/spring2001/
emailbyphone/EmailByPhone.html.
tions use SIP for call control. [6] J. Rosenberg. A SIP interface to voiceXML dialog servers. In-
Rosenberg et al. [6] describe a SIP interface to VoiceXML ternet Draft, Internet Engineering Task Force, July 2001. Work
in progress.
dialog servers including SIP URI for VoiceXML service and [7] J. Rosenberg, H. Schulzrinne, G. Camarillo, A. Johnston, J. Pe-
message flows for blind and bridged call transfer. Tellme terson, R. Sparks, M. Handley, and E. Schooler. SIP: session
initiation protocol. RFC 3261, Internet Engineering Task Force,
studio 9 provided the first SIP based VoiceXML develop- June 2002.
ment platform that allows users to test custom VoiceXML [8] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson. RTP: a
pages or scripts. We used this for initial testing of our transport protocol for real-time applications. RFC 1889, Internet
Engineering Task Force, Jan. 1996.
Email by phone system. Our work describes the design of [9] H. Schulzrinne and S. Petrack. RTP payload for DTMF dig-
a SIP based VoiceXML browser and its application in our its, telephony tones and telephony signals. RFC 2833, Internet
IP telephony test bed. At the time of writing, ours is the Engineering Task Force, May 2000.
[10] H. Schulzrinne, A. Rao, and R. Lanphier. Real time streaming
only known implementation that associates the VoiceXML protocol (RTSP). RFC 2326, Internet Engineering Task Force,
transfer tag with the SIP REFER message for a conferencing Apr. 1998.
application. Moreover, sipvxml can be used as a third-party [11] K. Singh, W. Jiang, J. Lennox, S. Narayanan, and
H. Schulzrinne. Cinema: Columbia internet extensible multime-
voice application server like Tellme or an integrated compo- dia architecture. technical report CUCS-011-02, Department of
nent in the CINEMA architecture [3], [4], [11] for campus Computer Science, Columbia University, New York, May 2002.
or enterprise VoIP services. [12] K. Singh, G. Nair, and H. Schulzrinne. Centralized conferencing
using SIP. In Internet Telephony Workshop 2001, New York,
Apr. 2001.
6 http://www.voicexml-ivr.com/solutions/intro.html
[13] K. Singh and H. Schulzrinne. Unified messaging using SIP and
RTSP. In IP Telecom Services Workshop, pages 31–37, Atlanta,
7 http://www.speechworks.com/products/voicexml/openvxi.cfm
Georgia, Sept. 2000.
8 http://www.voice3g.com/appblocks.htm
[14] R. Sparks. SIP call control - transfer. Internet Draft, Internet
9 http://www.tellme.com Engineering Task Force, July 2001. Work in progress.