IMXDSPUG

i.MX DSP User's Guide

Rev. LF5.10.72_2.2.0 — 17 December 2021 User Guide

Contents
1 Introduction 1 Introduction......................................1
This document provides an overall introduction to the DSP including system 2 System Architecture........................ 1
3 File organization.............................. 5
architecture, file organization, DSP-related toolchain, and so on. This
4 Building DSP Framework on Linux
document helps with the overall understanding of the DSP-related code.
OS................................................... 6
Currently, the DSP is used to decode and encode audio streams on the i.MX 5 Building DSP Framework on
8QXP, 8QuadMax, 8M Plus, and 8ULP platforms. Windows OS....................................9
The current DSP framework can support several clients. They support 6 Building DSP Wrapper and Unit Test
...................................................... 15
these codecs:
7 Usage of DSP Binary Files............15
Decoder: 8 Building Codec Wrapper and Codec
Library........................................... 16
• AAC-LC 9 Revision History............................ 18
• AAC plus(HE-AAC/HE-AACv2) 10 Appendix A. Memory Allocation for
DSP............................................... 19
• BSAC
• DAB+
• MP2
• MP3
• DRM
• SBC
• OGG
• AMR-NB
• AMR-WB
• WMA
• WAV
• OPUS
Encoder:
• SBC

2 System Architecture
Figure 1 and Figure 2 provide the overall system architecture of the DSP-related code.
NXP Semiconductors
System Architecture

Application

GStreamer Android

DSP /unit_tests/DSP/dsp_test.out
Wrapper

User
space
Kernel
space
rpmsg
/dev/rpmsgx /dev/rpmsg_ framework Dma-buf
ctrlx

Vritio framework

virtio

Remote proc
Remoteproc framework

Mailbox

Hardware SOC
MU HIFI4

Figure 1. Software architecture for Cortex-A cores running Linux OS

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 2 / 21
NXP Semiconductors
System Architecture

DSP Framework/firmware
Rpmsg-lite (openamp)
HIFI4 Core

Execution

Memory

Library loader

Component Component

Dsp codec wrapper Dsp render

Dsp codecs SAI/ESAI/DMA DAC

Figure 2. Software architecture for DSP processor

The DSP-related code includes the DSP framework, DSP remoteproc driver, DSP wrapper, unit test, DSP codec wrapper, and
DSP codec.
• The DSP framework is a firmware code which runs on the DSP core. The DSP driver is used to load the DSP firmware into
the memory and transfer messages between the user space and the DSP framework.
• The remoteproc and RPMsg framework is used to transfer messages between the Cortex-A cores and the DSP cores. The
Message Unit (MU) is used to trigger interrupts between the Cortex-A cores and DSP cores when messages are placed into
the vring buffer.

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 3 / 21
NXP Semiconductors
System Architecture

• The DSP wrapper and the unit test are the application code in the user space, which uses the rpmsg_char interface to transfer
messages between the DSP remoteproc driver and the user space. In addition, the DSP wrapper is used to provide unified
interfaces for the GStreamer.
• The DSP codec provides the actual decoding and encoding functions.
• The DSP codec wrapper is a wrapping code for the DSP codec and provides unified interfaces for the DSP framework.

2.1 Remote processor start

To start the firmware, use the following command.

Board $> echo start >/sys/class/remoteproc/remoteprocX/state

2.2 Remote processor stop

To stop the firmware, use the following command.

Board $> echo stop >/sys/class/remoteproc/remoteprocX/state

2.3 Resource table example

#define NUM_VRINGS 0x02

/* Place resource table in special ELF section */

__attribute__((section(".resource_table")))

const struct remote_resource_table resources = {

/* Version */
1,

/* NUmber of table entries */

NO_RESOURCE_ENTRIES,
/* reserved fields */
{
0,
0,
},

/* Offsets of rsc entries */

{
offsetof(struct remote_resource_table, user_vdev),
},

/* SRTM virtio device entry */

{
RSC_VDEV,
7,
0,
RSC_VDEV_FEATURE_NS,
0,
0,
0,
NUM_VRINGS,
{0, 0},
},

/* Vring rsc entry - part of vdev rsc entry */

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 4 / 21
NXP Semiconductors
File organization

{VDEV0_VRING_DA_BASE, VRING_ALIGN, RL_BUFFER_COUNT, 0, 0},

{VDEV0_VRING_DA_BASE + VRING_SIZE, VRING_ALIGN, RL_BUFFER_COUNT, 1, 0},
};

3 File organization
The DSP framework, DSP wrapper, and unit test code are in the github.com/NXP imx-audio-framework repository. The DSP
remoteproc driver code belongs to the Linux OS kernel.
DSP codecs originated from Cadence are license-restricted: A license authorization is required from NXP Marketing to access
them in binary format.

3.1 DSP remoteproc driver

The driver is under the remoteproc framework. The remote processor (RPROC) framework allows the different platforms/
architectures to control (power on, load firmware, power off) remote processors while abstracting the hardware differences. For
more details, refer to the below link.
https://www.kernel.org/doc/Documentation/remoteproc.txt
The DSP remoteproc driver code is in the Linux OS kernel. It includes the following files:
• drivers/remoteproc/imx_dsp_rproc.c
• drivers/rpmsg/rpmsg_char.c
• drivers/rpmsg/rpmsg_ctrl.c
• drivers/rpmsg/rpmsg_ns.c

3.2 DSP framework

The DSP framework code is in this folder:
• imx-audio-framework/dsp_framework
• imx-audio-framework/dsp_framework/rpmsg-lite
The rpmsg-lite code is copied from https://github.com/NXPmicro/rpmsg-lite.

3.3 DSP wrapper and unit test

The DSP wrapper and unit test are in these folders:
• imx-audio-framework/dsp_wrapper
• imx-audio-framework/unit_test

3.4 Interface header files

The DSP-related code includes these four interface header files:
• imx-audio-framework/include/mxc_dsp.h
• imx-audio-framework/dsp_framework/plugins/audio_codec/dsp_codec_interface.h
• imx-audio-framework/dsp_wrapper/include/uni_audio/fsl_unia.h
• imx-audio-framework/dsp_wrapper/include/uni_audio/fsl_types.h
The mxc_dsp.h file is the same as the header file in the Linux OS kernel. This file includes the interfaces and command definitions
that are used by the DSP wrapper and unit test. The dsp_codec_interface.h file wraps the DSP codec’s header files. It includes
unified interfaces and command definitions which can be used by the DSP framework. The fsl_unia.h and fsl_types.h header files
include the interfaces and command definitions which can be used by GStreamer.

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 5 / 21
NXP Semiconductors
Building DSP Framework on Linux OS

4 Building DSP Framework on Linux OS

Before you compile the DSP-related code, set up the DSP-related toolchains. The DSP framework, DSP codec wrapper, and DSP
®
codec use Xtensa development toolchain.

4.1 Installing Xtensa development toolchain

The Xtensa development toolchain consists of two components which are installed separately in the Linux OS, including:
• Configuration-independent Xtensa Tool
• Configuration-specific core files and Xtensa Tool
®
The configuration-independent Xtensa Tool is released by Cadence . For the current code, the version of the tool
is XtensaTools_RI_2020_4_linux.tgz, which is updated from XtensaTools_RF_2016_4_linux.tgz. The two versions are
compatible. The XtensaTools_RF_2016_4_linux.tgz version is used as an example. You can download this package from
the Xtensa XplorerTM.
The configuration-specific core files and the Xtensa Tool are released by NXP. There are two specific Xtensa
tools. The hifi4_nxp_v4_3_1_3_dev_linux.tgz tool is updated from the hifi4_nxp_v3_3_1_2_dev_linux.tgz tool. The
hifi4_nxp_v3_3_1_2_dev_linux.tgz tool is used as an example for the i.MX8QXP and i.MX8QM boards. The second
tool is hifi4_mscale_v1_0_2_prod_linux.tgz that is updated from the hifi4_mscale_v0_0_2_prod_linux.tgz tool. The
hifi4_mscale_v0_0_2_prod_linux.tgz tool is used as an example for the i.MX8MP board.
When you have these two components, you can set up the toolchain as follows:
• Open the imx-audio-framework folder and execute these commands:

mkdir -p ./imx-audio-toolchain/Xtensa_Tool/tools mkdir -p ./imx-audio-toolchain/Xtensa_Tool/builds

• Set up the configuration-independent Xtensa Tool:

cd imx-audio-toolchain/Xtensa_Tool
tar zxvf XtensaTools_RF_2016_4_linux.tgz -C ./tools

• Set up the configuration-specific core files and the Xtensa Tool:

cd imx-audio-toolchain/Xtensa_Tool
tar zxvf hifi4_nxp_v3_3_1_2_dev_linux.tgz -C ./builds

• Install the Xtensa development toolchain:

cd imx-audio-toolchain/Xtensa_Tool
./ builds/RF-2016.4-linux/hifi4_nxp_v3_3_1_2_dev/install --xtensa-tools ./tools/RF- 2016.4-linux/
XtensaTools --registry ./tools/RF-2016.4-linux/XtensaTools/config

• Set the PATH environment variable:

export PATH= ./imx-audio-toolchain/Xtensa_Tool/tools/RF-2016.4- linux/XtensaTools/bin:$PATH

• Set the LM_LICENSE_FILE environment variable.

The Xtensa development tools use FLEXlm for license management. The FLEXIm licensing is required for tools such as the
Xtensa Xplorer, TIE Compiler, and Xtensa C and C++ compiler. If you want to use a floating license, install the FLEXIm license
manager and set the LM_LICENSE_FILE environment variable. If there is any problem, you can find useful information in the
Xtensa Development Tools Installation Guide User’s Guide.doc document provided by Cadence.
After the above steps, the Xtensa development toolchain is set up successfully. In addition, the Xtensa Tools and additional tools
are provided as 32-bit (x86) binaries. They are supported on 32-bit (x86) systems, and also on recent 64-bit (x86-64) systems that

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 6 / 21
NXP Semiconductors
Building DSP Framework on Linux OS

have appropriate 32-bit compatibility packages installed. If you use a 64-bit system (for example; Ubuntu 16.04), install the 32-bit
compatibility packages first. Use these commands:

sudo apt-get install lib32ncurses5 lib32z1 sudo dpkg --add-architecture i386

sudo apt-get install libc6:i386 libstdc++6:i386

4.2 Building DSP framework

After installing the DSP-related toolchains on your Linux OS server, you can compile the DSP framework. Execute the “make”
command in the imx-audio-framework folder to compile the DSP framework. This way also builds the DSP wrapper and unit test.
If you want to compile the DSP framework separately, see the README file in the imx-audio-framework folder. After the compiling
process, you can find the binary files in the imx-audio-framework/release folder.
For the DSP framework, different commands generate different frameworks for different platforms:
• imx-audio-framework/release/hifi4_imx8qmqxp.bin
• imx-audio-framework/release/hifi4_imx8mp.bin
• imx-audio-framework/release/hifi4_imx8ulp.bin
By default, the command generates the hifi4_imx8qmqxp.bin file. With the “PLATF=imx8m” attribute, it generates the
hifi4_imx8mp.bin file. With the “PLATF= imx8ulp” attribute, it generates the hifi4_imx8ulp file. With the “DEBUG=1” attribute, it
generates the firmware with the debug information. You can see the debug information using UART. For the i.MX QXP/QM board,
modify the “LPUART_BASE” from 0x5a090000 to 0x5a080000, disable “lpuart” in “lpuart_probe”, which is in the fsl_lpuart.c file
of the kernel, and the debug information can print in the UART console.

4.3 DSP DEBUG

Building the DSP framework with the extra “DEBUG=1” attribute, the DSP can print the debug information using the UART
console. To enable this feature, do some changes in the kernel and in the DSP side. For a different platform, prepare a different
board and different changes. The following sections describe what need to change for different platforms.

4.3.1 Enabling DSP debug on i.MX 8M Plus

Enable the UART for DSP print debug information in the i.MX 8M Plus board and add the UART clock in the DTS file and the UART
module driver in the DSP.
1. Add the UART clock and pinctrl in the DTS.
Add the UART clock and pinctrl in the DSP node as follows:

&dsp {
compatible = "fsl,imx8mp-dsp-v1"; memory-region = <&dsp_reserved>; reg = <0x0 0x3B6E80000x0
0x88000>;
pinctrl-0 = <&pinctrl_uart4>;
clocks = <&audiomix_clk IMX8MP_CLK_AUDIOMIX_OCRAMA_IPG>,
…
…
<&audiomix_clk IMX8MP_CLK_AUDIOMIX_ASRC_IPG>,
<&clk IMX8MP_CLK_UART4_ROOT>,
<&clk IMX8MP_CLK_UART4_ROOT>;
clock-names =
"ocram", "audio_root", "audio_axi", "core", "debug", "mu2", "sdma_root", "sai_ipg",
"sai_mclk","asrc_ipg", "uart_ipg", "uart_per";
…
fsl,dsp-firmware = "imx/dsp/hifi4.bin"; status = "okay";
};

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 7 / 21
NXP Semiconductors
Building DSP Framework on Linux OS

Then generate the DTB file, replacing the old one.

2. Add the UART driver in the DSP.
By default, the DSP side already supports enabling the UART. The code is located in the dsp_framework/arch/peripheral.c
file. Build the DSP firmware with the “DEBUG=1” attribute to generate the hifi4_imx8mp.bin file, rename it to hifi4.bin, and
copy it to the board.
3. Run the DSP and print the debug information.
Run one instance and the following debug information is printed on the fourth serial COM port:

DSP Start.....
core initialized
Response queue: write = 0x0 / read = 0x0 Command queue: write = 0x10001 / read = 0x0 ext_msg:
[client:0]:(80008004,4,1000) Response queue: write = 0x0 / read = 0x0 Command queue: write =
0x10001 / read
= 0x10001
alloc size out: 943feff8 4104 avail mem: 16773104 Response queue: write = 0x0 / read = 0x0
Response[client: 0]:(80048000,4,1000)
Command queue: write = 0x10001 / read = 0x10001 Response queue: write = 0x10001 / read = 0x10001
Command queue: write = 0x20002 / read = 0x10001 ext_msg: [client: 0]:(80008004,80000001,15)
Response
queue: write = 0x10001 / read = 0x10001

4.3.2 Enabling DSP DEBUG on i.MX QuadXPlus

To enable the DSP DEBUG on the i.MX QuadXPlus platform, you need only one base board, as shown in Figure 3.

Figure 3. i.MX8-8X-BB

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 8 / 21
NXP Semiconductors
Building DSP Framework on Windows OS

Connect the RS-232 to a PC and connect the base board to the i.MX QuadXPlus board.
1. Add the UART clock and pinctrl in the DTS.
Add the UART clock and pinctrl in the DSP node as follows:
dsp: dsp@596e8000 {

compatible = "fsl,imx8qxp-dsp"; reg = <0x596e8000 0x88000>;

clocks = ...

<&uart2_lpcg 1>, <&uart2_lpcg 0>; clock-names = ... "uart_ipg", "uart_per";

assigned-clocks = <&clk IMX_SC_R_UART_2 IMX_SC_PM_CLK_PER>; assigned-clock-rates = <80000000>;

...

status = "disabled";

};

Then build the image instead of the old one.

2. Modify the DSP side.
The DSP supports the LPUART driver in the dsp_framework/arch/peripheral.c file. Change the LPUART_BASE from
0x5a090000 to 0x5a080000:
diff --git a/dsp_framework/arch/board.h b/dsp_framework/arch/board.h

index 9e04e64e821c..75a15fd09f0d 100644

--- a/dsp_framework/arch/board.h

+++ b/dsp_framework/arch/board.h

@@ -138,7 +138,7 @@ enum {

#define MUB_BASE (MU_PADDR)

#define SYSTEM_CLOCK (600000000UL)

-#define LPUART_BASE (0x5a090000)

+#define LPUART_BASE (0x5a080000)

#define UART_CLK_ROOT (80000000)

#endif /*PLATF_8ULP */

Then build the DSP with “DEBUG=1” and copy it to the board.
3. Run the DSP and print the debug information.
This part is the same as on the i.MX 8M Plus board. Select the proper serial COM port and you will see the debug
information. The debug information cannot print on the i.MX 8QuadMax board, because the UART is taken.

5 Building DSP Framework on Windows OS

The DSP framework can be built also on Windows OS. The Xplorer software can be used to build the DSP framework on Windows
OS. This chapter explains how to use Xplorer to build the DSP framework. First, install the Xtensa Xplorer IDE. You can download
the Xplorer IDE and Xplorer license form Cadence.

NOTE
Log into the XPG Cadence website to download installers for the Xplorer IDE, Xtensa tools, and so on. For NXP
internal use, contact the DSP owner to get the NXP common XPG login credentials. The Xplorer 7.0.8 version is
used as an example and its default installation folder is C:\usr\xtensa.

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 9 / 21
NXP Semiconductors
Building DSP Framework on Windows OS

5.1 Adding new configuration packages

Currently, the hifi4_nxp_v4_3_1_3_dev_win32.tgz configuration package, which is updated from the
hifi4_nxp_v3_3_1_2_dev_win32.tgz configuration package, is used to build the DSP framework on Windows OS. The
hifi4_nxp_v3_3_1_2_dev_win32.tgz configuration package is used as an example. Add this configuration package into Xplorer
before building the code. You can get this configuration package and the corresponding memory map linker files from NXP. The
required files are as following:
• hifi4_nxp_v3_3_1_2_dev_win32.tgz
• memmap/mainsim folder
When you have the DSP configuration package and memory map linker files, you can add a new configuration package into
Xplorer as follows:
• Download and install Xtensa Tools for Xplorer.
If you do not have the Xtensa Tools, you shall download and install it using Xplorer. Currently, the Xtensa Tool that we use is
XtensaTools_RF_2016_4_win32.tgz. You can first open the Xplorer software and click the “RF-2016.4” option in the “XPG View”
panel and select the “tools->Xtensa Tools->Xtensa Tools 11.04 for Windows” option. After you select it, you can click the download
button to start the downloading process.

Figure 4. “XPG View” panel

After the download finishes, right-click the “Xtensa Tools 11.0.4 for Windows” option and select the “Install Xtensa Tools…” option
in the new dialog. The installing process takes some time.
The Xtensa Tool is installed successfully after this step. You can see this folder in the Xplorer’s installing folder if everything is OK:
C:\usr\xtensa\XtDevTools\install\tools\RF-2016.4-win32
• Add the configuration package into Xplorer.
When you have the hifi4_nxp_v3_3_1_2_dev_win32.tgz package from NXP, you can add it into Xplorer. The first thing to do is to
create a folder called build in Xplorer’s installing path if the build folder is not created already. The total path after this operation
is as follows:
C:\usr\xtensa\XtDevTools\downloads\RF-2016.4\build
• Place the hifi4_nxp_v3_3_1_2_dev_win32.tgz package into the new build folder.
C:\usr\xtensa\XtDevTools\downloads\RF-2016.4\build\ hifi4_nxp_v3_3_1_2_dev_win32.tgz
• After you have performed the above steps, you can click the refresh button in the “XPG View” panel and find the “build”
option in this panel.

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 10 / 21
NXP Semiconductors
Building DSP Framework on Windows OS

Figure 5. The “build” option

• Right-click the build->hifi4_nxp_v3_3_1_2_dev_win32.tgz package and click the “Install Build…” option in the new dialog
to start the installing process. This takes some time. You can see the following folder in the Xplorer’s installing folder if
everything is OK.
C:\usr\xtensa\XtDevTools\install\builds\RF-2016.4-win32\hifi4_nxp_v3_3_1_2_dev
• Add the new memmap linker files into Xplorer.
After you add the hifi4_nxp_v3_3_1_2_dev_win32.tgz configuration package into Xplorer, you can add the new memmap linker
files. You can copy the mainsim folder into Xplorer’s install folder to finish this process. The complete folder after this process is
as follows:
C:\usr\xtensa\XtDevTools\install\builds\RF-2016.4-win32\hifi4_nxp_v3_3_1_2_dev\xtensa- elf\lib\mainsim
After you complete the above three steps, the new configuration package and the corresponding memory map linker files are
successfully added into Xplorer.

5.2 Creating the DSP framework Xplorer project

The DSP framework project must be created before using Xplorer to build it. The DSP framework code is in the imx-audio-
framework package:
• imx-audio-framework\dsp_framework
You can create the DSP framework as follows:
• Open Xplorer and click the “File->New->Xtensa C/C++ project” option in the menu bar. You will see this dialog:

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 11 / 21
NXP Semiconductors
Building DSP Framework on Windows OS

Figure 6. New Xtensa project

• Enter the project name and import DSP framework source code into the “New Xtensa C/C++ Project” dialog. Then click
the “Finish” button.

Figure 7. The “Finish” button

After the above two steps, the DSP framework project is successfully created. You can see the project in Figure 7.

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 12 / 21
NXP Semiconductors
Building DSP Framework on Windows OS

Figure 8. DSP framework project

5.3 Building DSP framework

When you created the DSP framework project, you can build its code. Choose the memmap linker files before the building process.
• Right-click the name of the DSP framework project in the “Project Explorer” panel and choose the “Build Properties…”
option. You will see the “Build Properties for dsp_framework” dialog. The dialog is shown in Figure 9.

Figure 9. “Build Properties” dialog

• Click the “Addl” compiler to add options. Add the “-DPLATF_8m” attribute to build the firmware for i.MX8mp. Add the
“-DPLATF_8MP_LPA” attribute to build the firmware for LPA. Add the “-DDEBUG” attribute to build the firmware with the
print debug information.

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 13 / 21
NXP Semiconductors
Building DSP Framework on Windows OS

• Click the “Linker” option and configure the custom LSP path as shown in Figure 10. Click the “OK” button to finish this
process.

Figure 10. “Linker” option

• When you configured the memmap linker files, you can choose the dsp_framework project and the required DSP
configuration to start the building process. The configuration is as follows:

Figure 11. Build process configuration

• Click the “Build Active->Build Active” option to start building the DSP framework. This takes some time.

NOTE
You must remove the file rpmsg_queue.c and directory porting in rpmsg_lite, otherwise it reports error.

• After performing the above steps, you get the binary file called dsp_framework (which is the firmware of the DSP) in the
following folder.
C:\usr\xtensa\Xplorer-7.0.8- workspaces\workspace\dsp_framework\bin\hifi4_nxp_v3_3_1_2_dev\Debug\dsp_framework
If you want to use this binary file to run on a real board, rename the dsp_framework binary file as
hifi4.bin and place it to a right place of “rootfs”.

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 14 / 21
NXP Semiconductors
Building DSP Wrapper and Unit Test

6 Building DSP Wrapper and Unit Test

Before you compile the DSP wrapper and the unit test, set up the related toolchain. The DSP wrapper and the unit test use the
Linaro compiler toolchain for the Yocto platform.

6.1 Installing Linaro compiler toolchain

Currently, the “gcc-linaro-4.9-2015.02-3-x86_64_aarch64-linux-gnu” toolchain is used to compile the DSP wrapper and the unit
test’s code for the Yocto platform. This toolchain shall be placed into the /usr folder of your Linux OS server. If you want to
successfully build the code, you can get more information from the Makefile file of the DSP wrapper and the unit test.

6.2 Building the code

When the Linaro toolchain is successfully installed on your server, you can compile the DSP-related code. You can execute the
“make” command in the imx-audio-framework folder to compile the DSP wrapper and the unit test. If you want to compile them
separately, see the README file in the imx-audio-framework folder. After the compiling process, you can find the binary files in
the imx-audio-framework/release folder.
For the DSP wrapper:
• imx-audio-framework/release/wrapper/lib_dsp_wrap_arm_elinux.so
For the unit test:
• imx-audio-framework/release/exe/dsp_test

7 Usage of DSP Binary Files

7.1 Getting DSP binary files

You can get the DSP binary files of the DSP framework, DSP wrapper, and unit test directly from NXP or compile the source code
to produce them yourself. You can also obtain DSP codec binary files directly from NXP. DSP codecs originated from Cadence
are license-restricted: A license authorization is required from NXP Marketing to access them.
The location for all prebuilt binaries not requiring any NXP Marketing authorization is on the Yocto mirror server.

7.2 Binary files in Linux OS rootfs

To run these binary files, place them into the Linux OS rootfs. The location of the DSP framework is determined by the DSP
remoteproc driver, so you shall keep it in the specified place. The location of the DSP wrapper is determined by the GStreamer
and you shall keep it in the specified place. You can change the location of the unit test. The binary files are in these folders:
• The unit test is here (default path):
/unit_tests/DSP/dsp_test.out
• The DSP framework is here:
/lib/firmware/imx/dsp/hifi4.bin
• The DSP wrapper is here:
/usr/lib/imx-mm/audio-codec/wrap/lib_dsp_wrap_arm_elinux.so
• You can keep the DSP codec wrapper and the DSP codec in these folders of the Linux OS rootfs (These libraries require
authorization from NXP Marketing):
/usr/lib/imx-mm/audio-codec/dsp/lib_dsp_codec_wrap.so
/usr/lib/imx-mm/audio-codec/dsp/lib_dsp_mp3_dec.so
/usr/lib/imx-mm/audio-codec/dsp/lib_dsp_aac_dec.so

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 15 / 21
NXP Semiconductors
Building Codec Wrapper and Codec Library

/usr/lib/imx-mm/audio-codec/dsp/lib_dsp_bsac_dec.so
/usr/lib/imx-mm/audio-codec/dsp/lib_dsp_dabplus_dec.so
/usr/lib/imx-mm/audio-codec/dsp/lib_dsp_drm_dec.so
/usr/lib/imx-mm/audio-codec/dsp/lib_dsp_mp2_dec.so
/usr/lib/imx-mm/audio-codec/dsp/lib_dsp_sbc_dec.so
/usr/lib/imx-mm/audio-codec/dsp/lib_dsp_sbc_enc.so
• Add DSP NXP codec wrapper library (WMA10 library requires authorization from NXP Marketing, others are on the Yocto
Mirror Server):
/usr/lib/imx-mm/audio-codec/dsp/lib_mp3d_wrap_dsp.so
/usr/lib/imx-mm/audio-codec/dsp/lib_aacd_wrap_dsp.so
/usr/lib/imx-mm/audio-codec/dsp/lib_vorbisd_wrap_dsp.so
/usr/lib/imx-mm/audio-codec/dsp/lib_wma10d_wrap_dsp.so
/usr/lib/imx-mm/audio-codec/dsp/lib_nbamrd_wrap_dsp.so
/usr/lib/imx-mm/audio-codec/dsp/lib_wbamrd_wrap_dsp.so

7.3 Unit test and playing

7.3.1 dsp_test
After placing the binary files into the correct location of the rootfs, you can decode or encode audio streams directly using the unit
test binary file. To decode one *.mp3 file, use this command:
./dsp_test -f1 -d16 -itest.mp3 -otest.pcm

dsp_test.out -f3 -r32 -t49 -d16 -ithetest_48000ps_chbr32.nac -othetest_48000ps_chbr32.pcm dsp_test.out -f4

-il2-fl11.mp2 -ol2-fl11.pcm

For more information about the dsp_test, use this command:

./dsp_test

To play one music file using the GStreamer and DSP wrapper, use this command:
gplay-1.0 test.mp3

The dsp_test also supports compress playback. This feature is now usable on i.MX 8MP board, after this change the dtb filename
is: imx8mp-evk-dsp.dtb, you can hear the sound by running the command:
./dsp_test -f1 -c33 -d16 -isyz.mp3

8 Building Codec Wrapper and Codec Library

The library of the DSP codec wrapper and DSP codec is the loadable library. This chapter describes how to make the loadable
library for the DSP.
The DSP loadable library is available as two different types: a fixed-location overlay and a position-independent library.
• For a fixed-location overlay, you can load the code into a predetermined location in the memory.
• For a position-independent library, you can load the code at an address determined during runtime.
You can link the loadable library using a special LSP named “piload” or “pisplitload” (see the Xtensa Linker Support Packages
(LSPs) Reference Manual). The binary files that are used by the DSP framework belong to the position-independent library, so
this chapter briefly discusses how to generate the position-independent library. For more detailed information, see Chapter 4 of
the Xtensa System Software Reference Manual.

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 16 / 21
NXP Semiconductors
Building Codec Wrapper and Codec Library

A position-independent library can be loaded and run at any address that supports both code and data, like a normal system RAM.
Alternatively, you can use the “pisplitload” LSP to load the code and data into separate memory blocks located in local RAMs. The
library location must be decided before the runtime.
The Xtensa development toolchain must be installed before making a loadable library. After that, you can follow the steps below.

8.1 Finding custom LSPs

The loadable libraries must be linked to a custom linker support package. For the position-independent libraries, you do not have
to generate or edit an LSP. Instead, you must link your position-independent library using the standard “pisplitload” LSP that is
provided as a part of your configuration.

8.2 Source code modifying and compiling

The API only allows the main program to directly access a single symbol in the library, the “_start” symbol. The library cannot
access any symbols in the main program directly. Any other symbol’s address must be passed to or from the library as an argument
to the “_start” function. This code is an example:
#include <stdio.h>

/* declare a printf function pointer */ int (*printf_ptr)(const char *format, …);

/* replace all calls to printf with calls through the pointer */ #define printf printf_ptr

/* This is the function provided by the library */ char * interface_func(unsigned int input)

printf(“executing function interface_func\n”); 13

return “this is string returned from interface_func”;

void * _start(int (*printf_func)(const char *format, …))

printf_ptr = printf_func;

/* The main application wants to call the function interface_func, but can’t directly reference it.
Therefore, this function returns a pointer to it, and the main application will be able to call it via this
pointer. */

return interface_func;

The main application calls the “_start” function, passes a pointer to “printf”, and takes a pointer to interface_func() in return. If the
library and the main program must communicate a value of more than one symbol, then the “_start” function call can return arrays
of pointers, rather than single pointers.
After finishing your source code, you can use “xt-xcc” of the Xtensa development toolchain to compile the code. Because the
position-independent libraries can be loaded at any address, make sure that the code in the library is position-independent using
the “-fpic” flag along with your normal compile options, as shown here:
xt-xcc -O3 -o library.o -c library.c

8.3 Linking the library code

In this step, link the library code into a loadable library using the appropriate LSP. For position-independent library, you can use
this command:
xt-xcc -mlsp=pisplitload -Wl,--shared-pagesize=128 -Wl,-pie -lgcc -lc -o library.so library.o

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 17 / 21
NXP Semiconductors
Revision History

After this command, you can get a position-independent library with the code and data loadable separately. If you want to get a
contiguous position-independent library, you can use this command.
xt-xcc -mlsp=piload -Wl,--shared-pagesize=128 -Wl,-pie -lgcc -lc -o library.so library.o

After the linking stage, you can get a loadable library which can be loaded by the DSP framework. The current DSP framework
only supports loading the code and data sections separately.

9 Revision History
The following table provides the revision history for this document.

Table 1. Revision history

Revision number Date Substantive changes

0 06/2018 Initial release

1 01/2019 Added details about using the sound card feature that allows users to
play mp3 files over ALSA compressed interface.

2 05/2020 Updated sections Introduction, Building DSP framework, and Binary

files in Linux OS rootfs.

3 09/2020 Added support for the i.MX8 MP board. Added support for *.wav files
playback by the ALSA compressed interface. Added details about DSP
framework building.

4 01/2021 Updated the version of the toolchain. Added details about the firmware
generation and the LPA.

5 09/2021 • Added 8ulp support

• Removed the cplay and LPA support
• Added Remote processor start, Remote processor stop, and
Resource table example
• Updated System Architecture, DSP framework, DSP remoteproc
driver, Enabling DSP debug on i.MX 8M Plus, Enabling DSP
DEBUG on i.MX QuadXPlus, and Appendix A. Memory Allocation
for DSP
• Updated Figure 1
• Added Figure 2
• Changed "DSP driver" to "DSP remoteproc driver" in
File organization

6 11/2021 • Added OPUS decoder in Introduction

• Updated Figure 10
• Added note in Building DSP framework
• Added compress playback feature for i.MX 8MP board in dsp_test
• Removed the path of configurable memory map file from Installing
Xtensa development toolchain

Table continues on the next page...

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 18 / 21
NXP Semiconductors
Appendix A. Memory Allocation for DSP

Table 1. Revision history (continued)

Revision number Date Substantive changes

LF5.10.72_2.2.0 12/2021 This document is published with the Linux software document
package from this release.

10 Appendix A. Memory Allocation for DSP

The DSP firmware is loaded into the memory by the DSP remoteproc driver. The loading address is defined by the memory map
linker files of the Xtensa development toolchain. You may change the loading address based on the memory map list of i.MX
8QuadXPlus, shown in Table 2.

Table 2. Memory allocation on i.MX8 QXP

Cortex-A35/Cortex-M4 DSP Content

— 0x80000000 - 0x806FFFFF Reserved (cannot be used)

0x59700000 - 0x5971FFFF 0x80700000 - 0x8071FFFF DSP OCRAM-system RAM

0x59720000 - 0x5973FFFF 0x80720000 - 0x8073FFFF DSP OCRAM-system ROM

— 0x80740000 - 0x80FFFFFF Reserved (cannot be used)

0x80700000 - 0x8073FFFF — Linux OS kernel (not visible from DSP)

0x81000000 - 0x9FFFFFFF 0x81000000 - 0x9FFFFFFF SDRAM

Currently, the Linux OS kernel reserves the memory for the DSP in the SDRAM separately. The range of the reserved
memory is 0x92400000 ~ 0x943fffff (32 MB). You may set this reserved memory by changing the imx8x-mek.dtsi file in the
linux-kernel/arch/arm64/boot/dts/freescale folder.

reserved-memory {
…………….
dsp_reserved: dsp@92400000 {
reg = <0 0x92400000 0 0x1000000>;
no-map;
};
dsp_reserved_heap: dsp_reserved_heap {
reg = <0 0x93400000 0 0xef0000>;
no-map;
};
dsp_vdev0vring0: vdev0vring0@942f0000 {
reg = <0 0x942f0000 0 0x8000>;
no-map;
};
dsp_vdev0vring1: vdev0vring1@942f8000 {
reg = <0 0x942f8000 0 0x8000>;
no-map;
};
dsp_vdev0buffer: vdev0buffer@94300000 {
compatible = "shared-dma-pool";
reg = <0 0x94300000 0 0x100000>;
no-map;
};

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 19 / 21
NXP Semiconductors
Appendix A. Memory Allocation for DSP

………….....
}

The DSP remoteproc driver splits the current reserved memory into five parts. One part is used to store the DSP firmware and the
other part is a scratch memory for the DSP framework. The detailed information about these five parts is shown in Table 3.

Table 3. Five memory parts

0x92400000 - 0x933FFFFF DSP firmware (16 MB)

0x93400000 - 0x942EFFFFF Scratch memory (16 MB)

0x942F0000 - 0x942F7FFF vdev0vring0

0x942F8000 - 0x942FFFFF vdev0vring1

0x94300000 - 0x943FFFFF vdev0buffer

NOTE
If you make changes in the memory map linker files of the Xtensadevelopment toolchain, make the related changes
for the DSP remoteproc driver.

i.MX DSP User's Guide, Rev. LF5.10.72_2.2.0, 17 December 2021

User Guide 20 / 21
How To Reach Us Limited warranty and liability — Information in this document is provided solely to enable system and software implementers to use NXP
products. There are no express or implied copyright licenses granted hereunder to design or fabricate any integrated circuits based on
Home Page:
the information in this document. NXP reserves the right to make changes without further notice to any products herein.
nxp.com
NXP makes no warranty, representation, or guarantee regarding the suitability of its products for any particular purpose, nor does
Web Support: NXP assume any liability arising out of the application or use of any product or circuit, and specifically disclaims any and all liability,
nxp.com/support including without limitation consequential or incidental damages. “Typical” parameters that may be provided in NXP data sheets and/or
specifications can and do vary in different applications, and actual performance may vary over time. All operating parameters, including
“typicals,” must be validated for each customer application by customer's technical experts. NXP does not convey any license under
its patent rights nor the rights of others. NXP sells products pursuant to standard terms and conditions of sale, which can be found at
the following address: nxp.com/SalesTermsandConditions.

Right to make changes - NXP Semiconductors reserves the right to make changes to information published in this document, including
without limitation specifications and product descriptions, at any time and without notice. This document supersedes and replaces all
information supplied prior to the publication hereof.

Security — Customer understands that all NXP products may be subject to unidentified or documented vulnerabilities. Customer
is responsible for the design and operation of its applications and products throughout their lifecycles to reduce the effect of these
vulnerabilities on customer’s applications and products. Customer’s responsibility also extends to other open and/or proprietary
technologies supported by NXP products for use in customer’s applications. NXP accepts no liability for any vulnerability. Customer
should regularly check security updates from NXP and follow up appropriately. Customer shall select products with security features
that best meet rules, regulations, and standards of the intended application and make the ultimate design decisions regarding its
products and is solely responsible for compliance with all legal, regulatory, and security related requirements concerning its products,
regardless of any information or support that may be provided by NXP. NXP has a Product Security Incident Response Team
(PSIRT) (reachable at [email protected]) that manages the investigation, reporting, and solution release to security vulnerabilities of
NXP products.

NXP, the NXP logo, NXP SECURE CONNECTIONS FOR A SMARTER WORLD, COOLFLUX,EMBRACE, GREENCHIP, HITAG,
ICODE, JCOP, LIFE, VIBES, MIFARE, MIFARE CLASSIC, MIFARE DESFire, MIFARE PLUS, MIFARE FLEX, MANTIS, MIFARE
ULTRALIGHT, MIFARE4MOBILE, MIGLO, NTAG, ROADLINK, SMARTLX, SMARTMX, STARPLUG, TOPFET, TRENCHMOS,
UCODE, Freescale, the Freescale logo, AltiVec, CodeWarrior, ColdFire, ColdFire+, the Energy Efficient Solutions logo, Kinetis,
Layerscape, MagniV, mobileGT, PEG, PowerQUICC, Processor Expert, QorIQ, QorIQ Qonverge, SafeAssure, the SafeAssure logo,
StarCore, Symphony, VortiQa, Vybrid, Airfast, BeeKit, BeeStack, CoreNet, Flexis, MXC, Platform in a Package, QUICC Engine, Tower,
TurboLink, EdgeScale, EdgeLock, eIQ, and Immersive3D are trademarks of NXP B.V. All other product or service names are the
property of their respective owners. AMBA, Arm, Arm7, Arm7TDMI, Arm9, Arm11, Artisan, big.LITTLE, Cordio, CoreLink, CoreSight,
Cortex, DesignStart, DynamIQ, Jazelle, Keil, Mali, Mbed, Mbed Enabled, NEON, POP, RealView, SecurCore, Socrates, Thumb,
TrustZone, ULINK, ULINK2, ULINK-ME, ULINK-PLUS, ULINKpro, µVision, Versatile are trademarks or registered trademarks of Arm
Limited (or its subsidiaries) in the US and/or elsewhere. The related technology may be protected by any or all of patents, copyrights,
designs and trade secrets. All rights reserved. Oracle and Java are registered trademarks of Oracle and/or its affiliates. The Power
Architecture and Power.org word marks and the Power and Power.org logos and related marks are trademarks and service marks
licensed by Power.org. M, M Mobileye and other Mobileye trademarks or logos appearing herein are trademarks of Mobileye Vision
Technologies Ltd. in the United States, the EU and/or other jurisdictions.

© NXP B.V. 2018-2021. All rights reserved.

For more information, please visit: http://www.nxp.com
For sales office addresses, please send an email to: [email protected]

Date of release: 17 December 2021

Document identifier: IMXDSPUG