Logistics Extraction Cockpit

Agenda

What is the LO Extraction Cockpit?

Why do we need it?
How does it work?
How can I check the data flow?
How to get there?
When is it available? (R/3, BW)
Which applications are supported?

SAP R/3 Extraction Overview Diagram

The Transaction Data that arrives in the BW by means of Extractors originates

from very different modules. For historical reasons, this requires very different
extractions mechanisms.
2

What is the Logistics Extraction Cockpit?

New technique to extract logistics information
Consists of new
Standard Extract Structures
Standard DataSources

Comparison: LIS vs LO Extraction Cockpit

BW

d
l
o
Initial-Load

BW

w
e
n

Delta-Load

Initial-Load

Delta-Load

R/3

S260

S260BIW1
S260BIW2

Statistical
Setup

Interface

Application

Cluster
Tables
MC11VA0ITMSETUP

V3 Update

V1 / V2
Update

Statistical
Setup

Interface

MCVBAK
MCVBAP

Sales Order

Delta-Queue

VBAK

VBAP

Application

Job

Infostructure

R/3

Update
Queue

MCVBAK
MCVBAP

Sales Order

VBAK
VBAK

VBAP
VBAP

Dataflow with new Logistics Extraction

Comm.
Sales
Document Structure
Sales
Order

Delivery

Invoice

Extract
Structure

Data/ InfoSource

Cubes & Queries

MC11VA0HDR
MC11VA0ITM
MC11VA0SCL
MC11V_0ITM
MC11V_0SCL

2LIS_11_VAHDR
2LIS_11_VAITM
2LIS_11_VASCL
2LIS_11_V_ITM
2LIS_11_V_SCL

0SD_C04
0SD_C04

MCLIKP
...

MC12VC0HDR
MC12VC0ITM
MC12VC0SCL

2LIS_12_VCHDR
2LIS_12_VCITM
2LIS_12_VCSCL

0SD_C03

MCVBRK
...

MC13VD0HDR
MC13VD0ITM

2LIS_13_VDHDR
2LIS_13_VDITM

MCVBAK
...

R/3

0SD_C05

BW

The same R/3 Communication Structures are used to transfer data directly to BW
using new Datasources. Note: The LO data flow bypasses the LIS InfoStructures
5

New LO Data Extraction: Why?

Performance &
Volume

Simple Handling

Advantages

Standardized Logistics
Solution

No LISFunctionality

Advantages (1) : Performance & Data Volume

Detailed Extraction
You can deactivate the extraction of scheduling data, for ex: slim Extractors
means smaller volumes of data to upload.

Document Changes
Only changes to data that are relevant to BW are updated (smaller volumes
of data to upload.)

LIS-Tables are not updated

Reduces the volume of data to upload by removing redundant data.

Update using Batch Processes

No Pressure on the day to day running of business.

Advantages (2) : Simple Handling

No LIS-Functions
No Knowledge of LIS customizing, Update settings and so on is required.

Function Enhancements
Simple to create without modifications.

BW Customizing Cockpit:
Central, Standardised maintenance tool for logistics applications

Advantages (3) : Others

No Delta-Tables (SnnnBIW1/2)
No double update, no double Data-Storage

LO Extraction: IMG (SBIW)

Customizing Cockpit:
LBWE

BW Log:
LBWF

Delete Setup Tables

LBWG

Initialization/Setup:
OLI*BW

LO Extraction Step-by-Step
11. Infopackage for the Delta Upload
10. Set up Periodic V3 Update
9. Infopackage for the Delta Initialization
8. Delete Setup Tables / Setup Extraction
7. Activate Extract Structures
6. Maintain InfoCubes and Update Rules
5. Maintain Communication Structure / Transfer Rules
4. Assign InfoSource
3. Replicate DataSource in BW
2. Maintain DataSource
1. Maintain Extract Structures

10

1. Maintain Extract Structures

Grouped by Communication
Structure (CS)
SAP Standard Extract Structures
can be enhanced (user appends
on CS are available here)
Missing fields like units and
compound characteristics will be
filled automatically

Extract Structure Maintenance

Field Selection
Creating New Fields

11

Enhancements of Extract-Structures (2 Options)

Take existing fields from the

communication structure

go to the customizing cockpit

maintain extract-structure

select the fields from the different

catalogs on the right hand side

move the fields to the extract-structure

on the left hand side

save your settings

system automatically creates an append

to the delivered extract-structure which
contains the selected fields

Generate DataSource again

Reactivate Extract-Structure

Use the User-Exit to fill your own

fields
add your fields to an append
structure
in
the
appropriate
Communication Structure
Identify the appropriate user-exit
for
filling
the
fields
e.g. MCS10001 for Sales Orders
now the fields will also be available
in the customizing cockpit and you
can therefore proceed with the
steps on the left side

12

2. Maintain DataSource

Selection:
Selection Fields
Hide Fields
Cancellation Fields

At this point, you call up the general maintenance screen for Datasources by
selecting the extract structure fields that you want to transfer into the BW
13
system. This selection forms the basis for the process of generating the Datasource.

3. Replicate DataSource in BW

Replication of Datasource

You Replicate Datasources in the Source Systems tab strip of the BW

Administration Workbench select the relevant source system, and choose the
Replicate Datasources option from the context menu. All the Datasources that do
not yet exist in BW are transferred from the Source System to the BW System.
14

4. Assign InfoSource

15

5. Maintain Communication Structure / Transfer Rules

The Communication
Structure is the GLOBAL
(BW Definition of the Data).
Some Infoobjects may have
had their beginnings as
template datasource
Fields, that were mapped
to BW Infoobjects on the
Transfer rules Screen.

16

6. Maintain InfoCubes and Update Rules

17

7. Activate Extract Structures

Activate the Extract Structures

18

8. Delete Setup Tables / Setup Extraction

Transaction LBWG

19

Setup Extraction
SAP OLTP System

SetUp
Tables
Full Load

Transaction
Data

Central Delta Management

--------------------------Delta Extraction

Staging
Engine

20

Initialization/Simulation (OLI*BW)
Selection Criteria to limit
historical Data

Name of the Run

Date or Time to stop the program

to allow the R/3 System to better
support for Production

New Run Flag

Simulation:
No Update!

21

Extracting Data from SAP R/3 Archives

Infosource

SAP BW Server

Mapping and
Transfer Rules
Replicated
Metadata

DataSource
DataSource

I(nitial), F(ull)
Setup
Tables

Delta Queue
Update
Technology

Refresh
Statistical data
Application
Tables

D(elta)

SAP R/3 Source

System

Archived
Documents

Document
Processing

22

Filling in the Setup Table

Completing setup tables is a critical action, that should be carried out with caution.
For large data amounts, this can take longer than one night and may have to be
done at the weekend. In the setup log (transaction NPRT) you can see more
information on setups that have already been carried out.

23

9. Infopackage for the Delta Initialization

The Selected Dataset

in OLTP is completely
updated into BW and
the delta process is
Initialized.

24

10. Set up Periodic V3 Update

25

Selection of Update Mode

LBWE, Update Mode
Selection Screen as
PI.2003.1
LBWE, Update Mode
Selection Screen as
PI.2002.1

Details of each Method is given in coming Slides

26

11. Infopackage for the Delta Upload

Only the data

since the last
upload is
requested.

27

Tools to check the data flow

Data Updates

Simulation
Some Applications support Logs for Setup (e.g. SD)

Extraction Log (LBWF)

Log for last processed Document

Data-Queues

Update Queue
Transaction : SM13 (shows the data being updated)
Double click on MCEX_UPDATE_XX to see data

Delta Queue
Transaction: RSA7
(shows to be transferred data (delta) for a DataSource
and a Target system)

28

Extraction Log (LBWF)

BW Log
You can control transfers to BW using a log. Log will be updated from the last
processed document. A user-related log is defined for each application, if the user
parameter MCL is set. The last posting procedure for each application is always
recorded; existing log entries are overwritten.
Recommendation
The log is only for test purposes. You should deactivate it during productive operation.

29

Update Queue (SM13)

1. Double
Click
4. View
Data

2. Display

3. Double
Click

30

Delta-Queue (RSA7)
Mark a line and
press this button

31

Technical Overview of the Delta Queue

The Delta Queue is Constructed of three tables
1. ARFCSDATA: Raw Data, Based on the Extract Structure, but
compressed.
Pointer tables to Access and control the flow of
2. ARFCSSTATE:
data to multiple BW systems.
3. TRFCQOUT:

CDM and the Delta Queue provides these features.

1. Support multiple initializations of Delta.
2. Multiple receiving BW systems with out repeated extractions.
3. Support for repeat requests for delta, in the event the data was not
loaded successfully in BW.
4. Easy access and review of delta records, before they are sent to
BW, ideal for debugging and verification purposes.

32

Tables of the Delta Queue

Compressed (Encrypted)
lines of data (one or more) to
support one LUW. Data
based on the Extract
Structure.

ARFCSDATA

Client Dependent sender

pointer-table by queue name
(Datasource) and destination
(Receiver). WHO HAS WHAT.

ARFCSSTATE
Pointer #1 =
Pointer #2 =
TRFCQOUT
BW #1 has Pointer #1

Client Dependent Pointer

towards the Data Posted into
the Queue in one LUW. Entries
are not Target Dependent.

BW #1 has Pointer #2
BW #2 has Pointer #1

LUW = Logical Unit of work.

One or more logically grouped records, depending on programs Commit logic.

33

Supported Applications - DataSources

Purchasing (02)

2LIS_02_HDR Purchasing Data (Header Level)

2LIS_02_ITM Purchasing Data (Item Level)
2LIS_02_SCL Purchasing Data (Schedule Line Level)

Inventory Controlling (3)

2LIS_03_BF Material Movements

2LIS_03_UM Revaluations

Shop Floor Control (4)

2LIS_04_PEARBPL Reporting Points

2LIS_04_PECOMP REM Component View
2LIS_04_PEMATNR REM Material View
2LIS_04_P_ARBPL Work Center View PP
2LIS_04_P_COMP Component View PP
2LIS_04_P_MATNR Material View PP(PI)

Quality Management (5)

2LIS_05_Q0ACTY Quality Notifications - Activities

2LIS_05_Q0CAUSE Quality Notifications - Causes
2LIS_05_Q0ITEM Quality Notifications - Items
2LIS_05_Q0NOTIF Quality Notifications
2LIS_05_Q0TASK Quality Notifications - Tasks
2LIS_05_QE1 Inspection result: General data
2LIS_05_QE2 Inspection result: Quantitative data

34

Supported Applications - DataSources

Shipment (8)

2LIS_08TRFKP
2LIS_08TRFKZ
2LIS_08TRTK
2LIS_08TRTLP
2LIS_08TRTS

Sales (11)

2LIS_11_VAHDR Sales Document Header

2LIS_11_VAITM Sales Document Item
2LIS_11_VASCL Sales Document Schedule Line
2LIS_11_V_ITM Sales-Shipping Allocation Item Data
2LIS_11_V_SCL Sales-Shipping Allocation Schedule Line

Shipping (12)

2LIS_12_VCHDR Delivery Header Data

2LIS_12_VCITM Delivery Item Data
2LIS_12_VCSCL Sales-Shipping Schedule Line Delivery

Billing (13)

2LIS_13_VDHDR Billing Doc. Header Data

2LIS_13_VDITM Billing Document Item Data

Shipment Costs at Item Level

Shipment Costs at Delivery Item Level
Shipment: Header Data
Shipment: Delivery Item Data per Section
Shipment: Section Data

35

Supported Applications - DataSources

36

New Update Methods for Logistics Extraction

Agenda
Delta Extraction with the V3 Update
Problems with the V3 Update

Essential Features

New Update Methods

Technical Details

37

Agenda
Delta Extraction with the V3 Update
Problems with the V3 Update

Essential Features

New Update Methods

Technical Details

38

Delta Extraction with the V3 Update (I)

This means that the data is collected for the BW system (Delta
Queue) in the R/3 update tables before the transfer to the
interface. The data is then retrieved there by means of a periodic
update process that needs to be started.

During this V3 collective run, the data is transferred to the Delta

Queue, from which they are retrieved by means of requests
from the BW system.

39

Delta Extraction with the V3 Update (II)

Data Flow Schematic for Logistics Extraction with the V3 Update

Docu.
Tables

V1

Document 2

V3 Collective Run
(for example,
daily or hourly)

Docu.
Tables

V1

Docu.
Tables

Call

V3 Module

Call

V3 Module

V3 Module

Document n

Call

One LUW,
One Commit

A
(Stopped qRFC)

Time

Transfer
to BW

B
Delta Queue

Update Tables

Delta
Request

Re
ex adi
ist ng
ing an
LU d p
Ws roc
for ess
a D ing
ata of
So a l l
ur
ce

V1

Re
ad
all ing
a
ex
ist nd p
r
i
for ng u oces
p
s in
a m da
od te d g o
f
ule at
a

Document 1

BW (PSA,
ODS, Cube)

40

Agenda
Delta Extraction with the V3 Update
Problems with the V3 Update

Essential Features

New Update Methods

Technical Details

41

Problems with the V3 Update

Multiple Language Problems in a Serialized V3 Update

Normal V3 Collective Run

VBHDR
(Each step is a
full table scan)

Step 1:
Language :EN:

Step 2:
Language :DE:

:EN:
:EN:
:DE:
:EN:
.......
:EN:
:DE:
.......
:DE:
:EN:
.......
:EN:
.......

VA01
VA02
VA02
VA01
VA01
VA02
VA02
VA01
VA02

Serialized V3 Collective Run

Step 1:
Language :EN:
Step 2:
Language :DE:
Step 3:
Language :EN:
Step 4:
Language :DE:
Step 5:
Language :EN:

42

Problems with the Serialized V3 Update

The following problems continue to occur in conjunction with the V3 update in the
logistics extraction of transaction data:
The serialized V3 update can only ensure the correct sequence of extraction
data for a document if the document is not repeatedly changed within the
span of a second.
Furthermore, the serialized V3 update can only ensure the correct sequence
of extraction data for a document if the times are permanently and exactly
synchronized for all instances in a system. This is because the creation time
of the update record, which is determined by the local time for the application
server, is used for sorting the update data.
In addition, the serialized V3 update can only ensure the correct sequence of
extraction data for a document if it previously had no errors in the V2 update.
This is because the V3 update only processes the update data that is
successfully processed with the V2 update.
Independently of the serialization, update errors that occur in the V2 update
of a transaction and which cannot be reposted have the consequence that
the V3 updates for the transaction that are still open can never be processed.
This can thus lead to inconsistencies in the data in the BW system.
43

Agenda
Delta Extraction with the V3 Update
Problems with the V3 Update

Essential Features

New Update Methods

Technical Details

44

New Update Methods (I)

With PI 2002.1 the following new update methods for logistics extraction will be
offered:
Direct Delta:
With this update mode, the extraction data is transferred with each document
posting directly into the BW delta queue. In doing so, each document posting with
delta extraction is posted for exactly one LUW in the respective BW delta queues.
Queued Delta:
With this update mode, the extraction data is collected for the affected application
instead of being collected in an extraction queue, and can be transferred as usual
with the V3 update by means of an updating collective run into the BW delta
queue.
In doing so, up to 10000 delta extractions of documents for an LUW are
compressed for each DataSource into the BW delta queue, depending on the
application.
Non-serialized V3 Update:
With this update mode, the extraction data for the application considered is written
as before into the update tables with the help of a V3 update module. They are
kept there as long as the data is selected through an updating collective run and
are processed. However, in contrast to the current default settings (serialized V3
update), the data in the updating collective run are thereby read without regard to
sequence from the update tables and are transferred to the BW delta queue.
45

New Update Methods (II)

Data Flow Schematic for Logistics Extraction with Direct Delta

Document 1

V1

Docu.
Tables

Document 2

V1

Delta
Request

Docu.
Tables

V1

a ll
of urce
g
in aSo
s
es at
oc a D
r
p
d for
n
S
a
ng LUW
i
a d ng
e
R sti
i
ex

Docu.
Tables

e
ul
od
M ate
n
io Upd
ct
tra V1
Ex th
wi
e
ul
od
M ate
n
io Upd
ct
tra V1
Ex th
wi
le
du
Mo te
a
on
cti Upd
tra
1
Ex th V
wi

Document n

Transfer
to BW

B
Delta Queue

BW (PSA,

(Stopped qRFC)

ODS, Cube)

Time

46

New Update Methods (III)

Benefits and Features of the Direct Delta:
By writing in the delta queue within the V1 update process, the
serialization of documents is ensured by using the enqueue concept for
applications.
For customers with a low occurrence of documents, the process is
recommended if a downtime is possible in the initialization process
during the reconstruction and the delta-init request.
Extraction is independent of V2 update.
Additional monitoring of update data or extraction queue does not
apply.
No need to schedule a job at regular intervals to transfer the data to the
BW delta queues.
We cannot post any documents during delta initialization in an
application from the start of the recompilation run in the OLTP until all
delta init requests have been successfully updated in BW. Otherwise,
data from documents posted in the meantime is irretrievably lost.

47

New Update Methods (IV)

Data Flow Schematic for Logistics Extraction with Queued Delta

Docu.
Tables

V1

ue
ion que
extract
Filling
date
e V1 up
from th

Docu.
Tables

Document n

V1

Docu.
Tables

ue
on que
extracti
Filling
date
e V1 up
from th

eue
action qu
Filling extr
1 update
from the V

Document 2

Extraction Collective Run

(recommended hourly)

One LUW,
One Commit

A
(Stopped qRFC)

Time

Transfer
to BW

B
Delta Queue

Extraction Queue

Delta
Request

Re
ex adin
ist
ing g an
LU d pr
Ws oc
for essi
a D ng
ata of a
ll
So
ur
ce

V1

Re
a
all din
en g a
qu trie nd
p
s
eu
e f in t roce
h
or
s
an e ex sing
tr
ap
pli acti of
o
ca
tio n
n

Document 1

BW (PSA,
ODS, Cube)

48

New Update Methods (V)

Benefits and Features of the Queued Delta:
By writing in the extraction queue within the V1 update process, the
serialization of documents is ensured by using the enqueue concept for the
applications.
By collecting data in the extraction queue that is processed regularly
(preferably hourly, as recommended), this process is especially recommended
for customers with a high occurrence of documents.
The collective run uses the same reports as before (RMBWV3<Appl.-No.>,...).
Report RSM13005 will not be provided any more.
By collecting new document data during the delta-init request, the downtime in
the initialization process can be reduced for the reconstruction run (filling of the
setup tables).
V1 immeasurably more burdened than by using V3.
Collective run clearly performs better than the serialized V3. Especially the
critical aspect of multiple languages does not apply here.
49

Benefits and Features of the Queued Delta:

Extraction is independent of success of the V2 update.
In the case of a delta initialization, the document postings of the
affected application can be included again after successful execution
of the recompilation run in the OLTP, provided that you make sure that
the update collective run is not started before all delta Init requests
have
been
successfully
updated
in
the
BW.
For an overview of the data of all extraction queues of the logistics
extract structures Customizing Cockpit, use transaction LBWQ

50

The new function from the logistics queue overview (LBWQ) is available
for monitoring the extraction queues

51

Benefits and Features of the Unserialized V3 update :

With this update mode, the extraction data of the application continues to be
written to the update tables using a V3 update module and is retained there
until the data is read and processed by a collective update run.
However, unlike the current default values (serialized V3 update), the data is
read in the update collective run (without taking the sequence from the update
tables into account) and then transferred to the BW delta queues.
The restrictions and problems described in relation to the "Serialized V3
update" do not apply to this update method since serialized data transfer is
never the aim of this update method.
Limitation: Processes the data that has been successfully posted by V2
Update.

52

Affected Applications - Availability

02 Purchasing
03 Inventory Control
04 Manufacturing
05 Quality Management
08 Transport
11 SD Verkauf BW
12 LE Versand BW
13 SD Faktura BW
17 Plant Maintainance
18 Customer Care
45 Agency Business
40 Retail
43 Retail POS Kassierer
44 Retail POS Bons

PI 2002.1

PI 2003.1
53

New Update Methods (PI 2002.1)

The new update methods are available for selection together with the
update method Serialized V3 Update in the Logistics Extraction
Structures Customizing Cockpit (LBWE) for each application:

PI 2002.1
Old Method
New Methods
New methods are
optional

54

PI 2003.1 - Available Update Methods

PI 2003.1

New methods are

mandatory

55

Migration Procedure
The migration to the new update methods will happen automatically
during the upgrade.
The following preconditions are escential:
The Update tables of the particular application have to be empty.
All entries within V3 Update for the particular application have to be
processed.
All entries in the Extraction Queue (transaction LBWQ) have to be
processed. After successfull migration the Extraction Queue entries
have to be collected into the Delta Queue via the Reports
RMBWV3<Appl.-No.>
56

Agenda
Delta Extraction with the V3 Update
Problems with the V3 Update

Essential Features

New Update Methods

Technical Details

57

Technical Details

In the first delivery with PI 2002.1, the update method Serialized V3

Update is delivered for the first time as a default setting in order to
ensure upward compatibility and to allow the customers to get to know
the new methods themselves.

The update method Serialized V3 Update will no longer be offered as

of a later Plug-In (planned with PI 2003.1).

The update methods are stored in the new customizing table

TMCEXUPD.

For applications that are delivered with PI 2002.1 and have no entries in
table TMCEXUPD, no selection of alternative methods is possible for
the time being.

When using the method Queued Delta, it is strongly recommended,

that you use the newest qRFC version. (QRFC-Version 6.20.045,
Supplement 9)
58

Technical Details (III)

Call Hierarchy for the Application 11 Example
Direct Delta

Queued Delta

V3 Update

MCEX_UPDATE_CALL_11
Dialog
MCEX_UPDATE_11_V1

V1 Update

MCEX_UPDATE_11

RSC1_TRFC_QUEUE_WRITE

MCEX_UPDATE_11_QRFC

Extraction
Collective Run

MCEX_UPDATE_11

RSC1_TRFC_QUEUE_WRITE

MCEX_UPDATE_11

V3 Collective Run
RSC1_TRFC_QUEUE_WRITE

59

Thanks
Thanks to Madhuchhanda Pandit, Uma Mahesh &
Rajesh Sarkar

60