Lecture 12.1 - Approaches To Load Data

Uploaded by

Trang Nguyen

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

17 views

Lecture 12.1 - Approaches To Load Data

Uploaded by

Trang Nguyen

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 21

WEB524

WEB PROGRAMMING ON WINDOWS

WEEK 12 – LECTURE 1
APPROACHES TO LOAD DATA
This Week
• New web apps often need vast amounts of data before they’re
deployed.
• Existing web apps sometimes need additional data to support a new
feature.
• You’ll learn how this data loading task works.
• We’ll cover some techniques for common data loading scenarios.

WEB 524 - Approaches to Load Data 2

What is the problem?
• Assume that we’re designing and creating a web app.
• It needs data before we can publicly deploy it to the user community.
• In the past, you have used project templates that included an existing
data store with thousands of objects.
• You have also used other project templates that enabled you to
define your own data model.
• You probably created – hopefully programmatically – a few data items
for the app.
• Programmatically creating new objects is easy and easy to understand
but it is not well suited for creating hundreds or thousands of objects.

WEB 524 - Approaches to Load Data 3

How do we design a data loading solution?
• To design a data loading solution, we must answer these fundamental
questions (and maybe others):

1. Where is the source data?

2. What is the format of the source data?
3. Do we want the source data to be located within the project’s file
system?

WEB 524 - Approaches to Load Data 4

Where is the source data?
• In a file in the file system?
• On the web?
• In a database?

• Answering this question and understanding its implications is very

important.

WEB 524 - Approaches to Load Data 5

Maybe you have a comma-separated values
file (CSV)

WEB 524 - Approaches to Load Data 6

Or an Excel workbook file (XLSX)

WEB 524 - Approaches to Load Data 7

Where is the source data?
• Maybe the data is on the web… on a web page contained in an HTML
table.
• It can by manually extracted with a drag-and-drop “select” action,
copied, and pasted to a destination.
• Microsoft Excel can be a useful tool for this purpose, as it is nicely-
equipped to receive pasted HTML table data.

WEB 524 - Approaches to Load Data 8

WEB 524 - Approaches to Load Data 9
WEB 524 - Approaches to Load Data 10
WEB 524 - Approaches to Load Data 11
WEB 524 - Approaches to Load Data 12
WEB 524 - Approaches to Load Data 13
Maybe the data is in a Web API
• This can be a good thing – if you know how to programmatically
access a web service
• The data is ready to be requested and used.

WEB 524 - Approaches to Load Data 14

Maybe the data is in an existing database
• We used this approach for many weeks in the early part of the course.
• You used the Chinook music business database.
• For a new app, maybe you’ll have another database.

WEB 524 - Approaches to Load Data 15

What is the format of the source data?
• Is the source data in a form that we can use?
• Depending on its format, source data may be used with a small
amount of conversion or transformation.
• Some data formats need more work before they can be used and
loaded.
• In general, your goal should be to get or transform the data into a
rows-and-columns format that makes it easier to understand, and
then process on a row-by-row basis.

WEB 524 - Approaches to Load Data 16

Location of the source data
• Do we want the source data to be located within the project’s file system?
• Sometimes, it can make sense to copy-paste a data source file into the
project’s file system (using the Visual Studio Solution Explorer panel).
• When you do this, your program can reference and use the file because
you know exactly where it is.
• Other times, you will need to allow the user (admin or developer) to select
and use the data source at runtime.
• When you do this, you will need some UI that has a file upload element,
and logic that saves the file in the web app’s file system, so that it can be
processed.

WEB 524 - Approaches to Load Data 17

Too many answers – how to organize them
• We have created a table that attempts to organize the many ways to
approach this problem.
• In general, when loading data, continue to use the approach that we
have used for the past several weeks:
1. In the Manager class, create one-or-more methods that will load data.
2. In a controller, create an action/method that will call the method in the
manager object.

WEB 524 - Approaches to Load Data 18

Handling associations during data loading
• Can you load data that has associations?
• Yes, but be careful and plan your work.

• Here are some suggestions:

• Load all unassociated data first.
• If you have entity collections that are not associated with other objects but do
provide some useful function (static data for lookups etc.) then you can load
them any time. We suggest that you do this first.
• Carefully study your data model. (The design model classes – NOT the
database model.)
• If it helps, use a class diagram.

WEB 524 - Approaches to Load Data 19

Locate a “root”
• Locate a “root” in the data model.
• A “root” is a class that has objects that can be added, without
configuring an association.
• For example, in a ProductsSupplier scenario, we CAN add a Supplier
object, so it is a “root” class.
• Load that data - it can be used for lookups.

WEB 524 - Approaches to Load Data 20

Dependent Objects
• When adding dependent objects (e.g. products – to add a Product
object, we MUST have an association to an existing Supplier object).
• Load dependent objects.
• Maybe the CSV or XLSX source data includes a value that you can use
to look up the associated object (e.g. a Supplier name).
• Other times, you may have to organize and manage your source data
so that all dependent objects for a specific “root” object are added in
a batch.
• Continue this process until all data gets loaded