1.

Data Wrangling

Data wrangling refers to the process of transforming and mapping data from its raw form into a
usable format for analysis. It typically involves cleaning, restructuring, and enriching the data to
ensure it’s in a format suitable for analysis.

Key Steps in Data Wrangling:

Data Collection: Gathering data from various sources.

Data Cleaning: Handling missing or inconsistent data (removing or imputing).

Data Transformation: Changing the structure of the data (e.g., normalizing or aggregating).

Data Enrichment: Adding additional data or context for deeper analysis.

Data Formatting: Converting data types or ensuring consistency across datasets.

2. Data Cleaning

Data cleaning is the process of detecting and correcting (or removing) corrupt or inaccurate records
from a dataset.

Common Data Cleaning Tasks:

Removing Duplicates: Identifying and removing duplicate rows.

Handling Missing Values: Filling missing values with meaningful substitutes (e.g., mean, median,
mode) or removing rows/columns with missing data.

Correcting Errors: Fixing inconsistent data entries (e.g., "yes" vs. "Yes" or incorrect date formats).

Filtering Outliers: Identifying and dealing with extreme or incorrect data points.

3. Data Preparation

Data preparation involves preparing the data for the analysis phase. This includes transforming the
data into a consistent, normalized form.

Key Tasks in Data Preparation:

Data Transformation: Converting categorical data into numerical values (e.g., one-hot encoding).

Normalization/Standardization: Scaling features to a specific range or distribution.

Feature Engineering: Creating new features from existing ones (e.g., creating a new column for year
from a date column).

4. Handling Large Data Sets

Handling large datasets comes with challenges like memory constraints, processing speed, and data
accessibility. There are techniques to optimize performance.
Common Problems:

Memory Constraints: Large datasets may not fit in memory, causing performance issues.

Slow Processing: Large data takes time to process, which can slow down computations.

Data Storage: Storing and retrieving large datasets can be inefficient.

General Techniques for Handling Large Data Volumes:

Chunking: Breaking large datasets into smaller chunks and processing them in batches.

Streaming: Reading and processing data in a stream, one piece at a time, instead of loading
everything into memory.

Parallel Computing: Distributing tasks across multiple CPUs or machines to speed up processing.

Efficient Data Structures: Using data structures optimized for performance, like pandas DataFrames
or NumPy arrays.

Compression: Compressing data to save space.

Programming Tips:

Use Libraries Efficiently: Libraries like pandas and dask are optimized for large data sets.

Avoid Loops: Where possible, use vectorized operations instead of Python loops.

Memory Management: Load only necessary columns and use data types that consume less memory
(e.g., category for categorical data).

import pandas as pd

# Load the dataset

df = pd.read_csv("/mnt/data/solar data edited.csv")

# Step 1: Inspect the dataset

print("Initial Data Overview:")

print(df.head()) # Show the first few rows

print(df.info()) # Display column data types and missing value counts

# Step 2: Data Cleaning

df.columns = df.columns.str.strip().str.replace('\\n', '').str.replace(' ', '_').str.lower()

# Step 3: Handle missing values

# Check for missing values

df.isnull().sum() # Identify the number of missing values per column

# Fill missing numeric values with the column's mean

df.fillna(df.mean(numeric_only=True), inplace=True)

# Step 4: Convert date and time columns

# Ensure 'date_&_time' is in datetime format

df['date_&time'] = pd.to_datetime(df['date&_time'], format='%d-%m-%Y %H:%M:%S',

# Step 5: Data Preparation

# Create additional time-based features (e.g., hour and day of the week) for analysis

df['hour'] = df['date_&_time'].dt.hour

df['day_of_week'] = df['date_&_time'].dt.day_name()

# Step 6: Final inspection of cleaned and prepared data

print("\nCleaned and Prepared Data Overview:")

print(df.head())

print(df.describe())

# Save cleaned data to a new CSV file

df.to_csv("/mnt/data/solar_data_cleaned.csv", index=False)
Loading the Data:

The dataset is loaded into a DataFrame using pd.read_csv().

Inspecting the Data:

head() shows the first few rows.

info() provides details on the data types and missing values.

Cleaning the Data:

Column names are standardized by removing extra spaces and newline characters, converting to
lowercase, and replacing spaces with underscores.

Missing numeric values are filled with the mean of the respective columns to avoid data loss.

Date Conversion:

The date_&_time column is converted to the datetime type, making it easier to perform time-based
analysis.

Data Preparation:

New time-based features (hour and day_of_week) are added to support trend analysis over different
time periods.

Final Inspection:

describe() summarizes the statistics for numerical columns, and the cleaned data is saved
to a new CSV file.
Before Data Wrangling/Cleaning

This raw dataset includes missing data, inconsistencies, and poorly formatted entries.

---

After Data Wrangling/Cleaning/Preparation

- *Missing Data:* Filled or handled appropriately.

- *Date Format:* Standardized to "YYYY-MM-DD."

- *Inconsistent Data:* Weather condition names are standardized.

- *Negative/Missing Solar Output:* Handled by filling with mean/median or using interpolation.

1. *Missing Data Handling:* Missing solar output filled with average or interpolated values.

2. *Date Standardization:* All dates formatted as "YYYY-MM-DD."

3. *Weather Conditions:* Standardized to consistent terms like "Sunny" or "Overcast."

4. *Temperature Issues:* Missing or "Missing" temperature values filled with a mean value or
interpolated.

5. *Panel Age Handling:* NaN values replaced with a median or estimated age value.