International Journal of Trend in Scientific Research and Development (IJTSRD)

Volume 5 Issue 1, November-December 2020 Available Online: www.ijtsrd.com e-ISSN: 2456 – 6470

Abrasion of Current Weather of a City Using Variant Python

Libraries and Weather Application Programming Interface (API)
H. Sumesh Singha1, Dr. Bhuvana J2
1Master of Computer Application, 2Associate Professor,
1,2Jain Deemed-to-be University, Bengaluru, Karnataka, India

ABSTRACT How to cite this paper: H. Sumesh Singha

Web allows a wide extent of realities and information source set up by people. | Dr. Bhuvana J "Abrasion of Current Weather
However, it will comprise of a gigantic arrangement of divergent and of a City Using Variant Python Libraries and
Weather Application
debilitated coordinated information. In metropolitan urban areas of India,
Programming Interface
there are different institution that measures climate such as The Energy (API)" Published in
Resource and Institute, Indian Institute of Tropical Meteorology and many International Journal
more. There are also different sites that also provide accurate weather data of of Trend in Scientific
cities in India. For this research, I am using web scraping method to collect Research and
current weather report for a specific city in India from two different sources. Development (ijtsrd),
IJTSRD38037
The fundamental target of Web Scraping is to extract data from one or ISSN: 2456-6470,
Volume-5 | Issue-1,
numerous sites and cycle it into basic structures such as JSON, text, CSV files.
December 2020, pp.593-596, URL:
This incorporates web crawler and data extractor. Web crawler slithers all the www.ijtsrd.com/papers/ijtsrd38037.pdf
connections present in a site page and stores them in an information base
though data extractor extricates information from these put away connections. Copyright © 2020 by author(s) and
The information gathered by this web scraping procedure will shape a data set International Journal of Trend in Scientific
or information distribution centre that can be utilized for additional Research and Development Journal. This
is an Open Access
exploration on information digging for climate determining in metropolitan
article distributed
urban areas. These data stored in the database can be utilize to enhance the under the terms of
application and also meteorologist can study the weather pattern from the the Creative Commons Attribution
data and alert the people of the states from upcoming natural disasters. License (CC BY 4.0)
(http://creativecommons.org/licenses/by/4.0)
KEYWORDS: Web scraping, web crawler, data extractor, meteorologist.

1. INTRODUCTION
The weather influences nearly all that activities we do in our downloaded from these sites in the form of JavaScript Object
day by day lives. It influences what we wear and what kinds Notation (JSON), CSV, XML formats. For this research we are
of exercises that we do. In numerous callings, it influences going to use Web Scraping technology. From the term itself
either we should work, at what time to work, where should we can know that it is a technology which is used to extract
we work or we have to work for the whole work time in our hidden html code and the data stored in the database into
daily lives. Weather have become an important part for us the above datasets forms and stored into a local database
people. There are different factors that affect the weather making it uncomplicated and easy to visualize the data
such as temperature, windspeed, latitude, longitude, scraped. There are many Web Scraping tools such as Scraper
humidity and many others. The world wide web is a API, ParseHub, BeautifulSoup, Requests, Scrapy and many
significant wellspring of data for some experts in different more. For this research we will use BeautifulSoup to Scrape
areas. It contains helpful and futile, organized and non- weather report from www.wunderground.com and Requests
organized data, in various types of format, and from diverse from www.openweathermap.org. There are sites that
sources. In India there are multiple organizations, such as provides free or paid web-based APIs for the service
The Energy Resource and Institute, Indian Institute of provided by modern web application for example
Tropical Meteorology and many more which produces daily www.openweathermap.org provide free APIs as well as paid
weather reports. Apart from these organizations, there are services too. Its recommended to use APIs but there are
multiple sites like www.wunderground.com, some sites that doesn’t provides APIs, in such situation Web
www.openweathermap.org that provides real-time current scraping comes into play. It extracts the information from
weather data reports. Every day the weather changes and these sites. Web crawler also known as “spider” is an
the data generated for each day is stored in the database. artificial intelligence that stalk the web pages. Spider follow
These data can be structured or unstructured in nature. links and traverse to index and scout for content by
These data that are produced in large volume is known as browsing the internet much like a person who have plenty
Big Data. Handling such massive quantity of data is very amount of free time on their hands. A web scraper is a
complicated for a normal person. Scientist and researchers specialized tool intended to precisely and rapidly scrape
use various types of technologies to handle such data. For information from a site page. This research will use this
our research, we are going to use these datasets to generate meteorological data generated by web scaping in our web
weather of a city in our application. These datasets can be application and further this information can be put to use to

@ IJTSRD | Unique Paper ID – IJTSRD38037 | Volume – 5 | Issue – 1 | November-December 2020 Page 593
 International Journal of Trend in Scientific Research and Development (IJTSRD) @ www.ijtsrd.com eISSN: 2456-6470
study the weather and climate as well as to enhance the web a) Mimicry Approach
application. In this approach the scrapper is preconfigured with the
location of information gathered from the web page. This
2. OVERVIEW OF WEB SCRAPING approach is well organized, yet it is less adjusted with
Web Scraping, additionally called Web Harvesting, or Web regards to handle different heterogeneous sites.
data extraction is a way towards separating information
from a site. On the off chance that you have ever duplicated b) Weight Measurement Approach
data from a site and glued it into an Excel accounting page or This methodology depends on a nonexclusive calculation
Word doc then in fact you are scraping a site. Scraping which investigations the Document Object Model tree of a
manually every information from a site is very complicated page and measures the heaviness of words in each branch.
and time consuming. Rather than sitting at a PC hitting
Ctrl+C and Ctrl+V for every information you need, you set a c) Differential Approach
web scraping code running that will separate your ideal This approach explains that the content of the page will vary
information from the site and store it in your favoured only from the body of the web page where both of the page
document design very quickly. Today, web scraping is belongs form the same web page.
utilized in everything from land to web-based business.
From the activity perspective, a web scraping look like d) Machine Learning Approach
manual reorder task. But there is some variation that is this The main motive of this approach is to physically examined
task is finished in a coordinated and programmed way, by a sites pages on a huge sample and train an algorithm on it.
virtual PC specialist. The initial step of web scraping script is
Also, paper [1] has explained different categories and tools
to make a https request to the target website for the data of a
specific URL. This step is accomplished by “Scraper”. Once for web scraping. Progression of data retrieval have
the site returns the html file the “scraper” extracts the increased significantly over the period of time. Information
retrieval (IR) was used for web searches even before
information from that HTML file. For this research we are
internet came into appearance. This IR was common since as
extracting the current weather report which is accomplished
ahead of the schedule as 1960 in business and knowledge
by a process called “parsing”. The last advance is for the
applications. The capacity limit, accuracy, handling force of
script to store this information in a CSV excel sheet, JSON or
IR have dramatically increased over the period of time. Such
in an information base so it tends to be utilized physically or
turn of events and headway in the field of IR have likewise
in another program.
brought many changes in the progression of techniques of
querying from the physical library-based approaches. After
the creation of world wide web, the content of pages was
very few and doesn’t need IR to scrape information from
them. Despite the fact that this strategy experiences a
disadvantage of being restricted to thoughtfully homogenous
writings documents permits the archives to be filed by the
manner in which they are referred to in different articles.
The most well-known data recovery is the ad-hoc querying
where a query finds for a bunch of static reports. This
information retrieval technique was used by commercials
search engines like Google and Alta Vista. The downside of
this strategy is that the precision is very low. Agent
paradigm is another type of promising innovation for data
retrieval. IR system became much more scalable, adaptable,
Figure 1. Basic architecture of Web Scraping interoperable with the addition of agent in it.

3. APPLCATIONS OF WEB SCRAPING 5. PROBLEM STATEMENT

Applications of web scraping increases day by day. The There are multiple web applications that provides real-time
number of applications accessible today is more than it was weather report. Obviously, these applications would not
yesterday and in future these applications will keep on deliver false weather report but still there are some
increasing. Below there are a-few number of fields where misbelief whether the weather report is accurate or not. For
web scraping is mostly used: this research I am creating a uncomplicated and
straightforward web application for my academic purpose
 Weather forecasting. which will display current weather of a city. My project guide
 Comparison of prices of different product. came with a problem and asked me whether the report
 Analysing advertisement. displayed in my application is accurate or not? To overcome
 Web indexing & rank checking this problem, I came with an idea and thought to build a
 Analysing of graphical datasets. script that will scrape weather information from two
 Optimizing search engines. different sites.
 Content production.
6. PROPOSED SYSTEM
4. LITERATURE REVIEW Different Strategies and tools are used by web scraping
There are many approaches of web scraping. In paper [1] technology to scrape website data and export them into our
many different approaches are explained. In this paper i am local database in the form of JSON, CSV, XML etc. Python
going to explain only the basics of those approaches. provides different libraries or modules that can be used for
scraping datasets from the HTML web documents such as

@ IJTSRD | Unique Paper ID – IJTSRD38037 | Volume – 5 | Issue – 1 | November-December 2020 Page 594
 International Journal of Trend in Scientific Research and Development (IJTSRD) @ www.ijtsrd.com eISSN: 2456-6470
Requests, BeautifulSoup, Scrapy, Selenium. To overcome the 8. SEQUENCE DIAGRAM
problem of my project guide my application will use
Requests to make an API call to www.openweathermap.org
and display the report in the web application whereas
BeautifulSoup will scrape HTML page from
www.wunderground.com and exhibits the topical weather
temperature of a city on the console. This application is
created only for academic purpose. This application will
show temperature in a graphical representation along with
the other weather factors like windspeed, humidity weather
description any many more also it has a very simple user
interface with only a text field to enter the city name and
search button. This application will be deployed on a cloud
service i.e. www.pythonanywhere.com which provides
three months trail hosting service for your web-based
python application. This cloud service is fully automated,
easy and Simple to use.

Figure 3. Sequence Diagram

9. MODULE DIAGRAM

Figure 2. Proposed System Architecture

7. METHODOLOGY
BeautifulSoup is one of the important python libraries which
is used to handle XML, HTML and other types of mark-up
languages. Whereas Requests library is used to send HTTPs
request without the need of manually querying string to the
URL. Inside the request there is a module urllib3 which keep-
alive the URL and allows constant polling of data. Following
principles are followed to run the application:
Figure 4. Module Diagram
 First the application is accessed by a web browser.
10. RESULTS
 Then we need to entered the city name for the weather
report.
 After that the script sends a Https Get request to the
targeted site(www.openweathermap.org) using the
request module.
 Next the Scripts check whether the city exist in the Figure 5. Running the Application on Localhost
world or not from the database server. Depending on
the city existence a successful or error memo is flashed
on the application.
 Similarly, Beautifulsoup parse the weather temperature
from the target site(www.wunderground.com) which is
a HTML page and display it in the console.
 If any updates are required then a POST request is send
using the Python requests module.

Figure 6. Weather Report of a City (from

www.openweathermap.org)

@ IJTSRD | Unique Paper ID – IJTSRD38037 | Volume – 5 | Issue – 1 | November-December 2020 Page 595
 International Journal of Trend in Scientific Research and Development (IJTSRD) @ www.ijtsrd.com eISSN: 2456-6470
they can take required precautions and safety measures.
Also, this will help farmers to decide whether the weather is
suitable for planting, watering, and other harvesting
activities.

REFERENCE
[1] R. DIOUF, E. N. SARR, O. SALL, B. BIRREGAH, M.
Figure 7. Cities names in the Database BOUSSO and S. N. MBAYE, "Web Scraping: State-of-
the-Art and Areas of Application," IEEE International
Conference on Big Data (Big Data), pp. 6040-6042,
2019.
[2] F. Y. N. Kunang and S. . D. Purnamasari, "Web Scraping
Techniques to Collect Weather Data in South
Sumatera," INTERNATIONAL CONFERENCE ON
ELECTRICAL ENGINEERING AND COMPUTER
SCIENCE (ICECOS), 2018.
[3] M. S. Parvez, S. S. Rajendra, K. S. A. Tasneem and K. R.
Bodke, "Analysis Of Different Web Data Extraction
Techniques," International Conference on Smart City
and Emerging Technology (ICSCET), Mumbai, 2018.
[4] P. Ashiwal, S. R. Tandan, P. Tripathi and R. Miri , "Web
Information Retrieval Using Python and
BeautifulSoup," International Journal for Research in
Applied Science & Engineering Technology (IJRASET),
pp. 335-339, 2016.
[5] S. Singh and R. Jain, "Weather report on metropolitan
cities in India using web scraping technique,"
International Journal of Advance Research, Ideas and
Innovations in Technology, pp. 1068-1070, 2019.

Figure 8. Error Message of Invalid City [6] A. V. Saurkar , K. G. Pathare and S. G. Gode, "An
Overview On Web Scraping Techniques And Tools,"
International Journal on Future Revolution in
Computer Science & Communication Engineering , pp.
363-367, 2018.
[7] B. G. Dastidar, D. Banerjee and S. Sengupta, "An
Intelligent Survey of Personalized Information
Retrieval using Web Scraper," I.J. Education and
Management Engineering, 2016.
[8] R. Samya and R. Rathipriya, "Predictive Analysis for
Figure 9. weather report of a city (from Weather Prediction using Data Mining with ANN: A
www.wunderground.com) Study," International Journal of Computational
Intelligence and Informatics, pp. 150-154, 2016.
11. CONCLUSION AND FUTURE ENHANCEMENT
With the help of this application generating current weather [9] D. Karthikeyan and M. Shivaranjani, "A Review of
reports become much easier. With minimalistic user Weather Forecasting Using Data Mining Techniques,"
interface anyone can use this application and observe the International Journal Of Engineering And Computer
current weather of their city. There are very few chances for Science, pp. 19784-19788, 2016.
this application to malfunction as it used APIs for generating [10] V. Singrodia, A. Mitra and S. Paul, "A Review on Web
the weather report. In future the database can be modified to Scrapping and its Applications," International
stored not only the names of the cities but also the weather Conference on Computer Communication and
factors that affect the climate. Using that data, meteorologist Informatics (ICCCI -2019), 2019.
can study the weather pattern and generate report for
upcoming weeks, months and years. This will help to alert
people about the natural disasters at an early stage so that

@ IJTSRD | Unique Paper ID – IJTSRD38037 | Volume – 5 | Issue – 1 | November-December 2020 Page 596