WNReader - A Web Novel Scraping App

A PROJECT REPORT
Submitted to the APJ Abdul Kalam Technological University
in partial fulfillment of requirements for the award of degree

Bachelor of Technology
in
Computer Science and Engineering
by

Akhil Nair(CHN19CS007)
Nithin Johnson(CHN19CS086)
S Balamurali(CHN19CS108)
Sujin Skaria Philip(CHN19CS118)

DEPARTMENT OF COMPUTER ENGINEERING

COLLEGE OF ENGINEERING CHENGANNUR
KERALA
July 2022
 DEPARTMENT OF COMPUTER ENGINEERING
COLLEGE OF ENGINEERING CHENGANNUR
2021 - 22

CERTIFICATE

This is to certify that the report entitled WNReader - A Web Novel Scraping
App submitted by Akhil Nair (CHN19CS007), Nithin Johnson (CHN19CS086),
S Balamurali (CHN19CS108) & Sujin Skaria Philip (CHN19CS118) to the APJ
Abdul Kalam Technological University in partial fulfillment of the B.Tech. degree in
Computer Science and Engineering is a bonafide record of the project work carried
out by them under our guidance and supervision. This report in any form has not been
submitted to any other University or Institute for any purpose.

Internal Supervisor External Supervisor

Project Coordinator Head of Department

 DECLARATION

We hereby declare that the project report WNReader - A Web Novel Scraping App,
submitted for partial fulfillment of the requirements for the award of degree of Bachelor
of Technology of the APJ Abdul Kalam Technological University, Kerala is a bonafide
work done by us under supervision of

This submission represents our ideas in our own words and where ideas or words of
others have been included, we have adequately and accurately cited and referenced the
original sources.

We also declare that we have adhered to ethics of academic honesty and integrity
and have not misrepresented or fabricated any data or idea or fact or source in our
submission. We understand that any violation of the above will be a cause for
disciplinary action by the institute and/or the University and can also evoke penal
action from the sources which have thus not been properly cited or from whom proper
permission has not been obtained. This report has not been previously formed the basis
for the award of any degree, diploma or similar title of any other University.

Akhil Nair
Nithin Johnson
Chengannur
S Balamurali
22-07-2022
Sujin Skaria Philip
Acknowledgement

We take this opportunity to express our deepest sense of gratitude and sincere thanks
to everyone who helped us to complete this work successfully. We express our sincere
thanks to , Head of Department, Department of Computer Engineering, College of
Engineering Chengannur for providing us with all the necessary facilities and support.

We would like to express our sincere gratitude to the , Department of Computer

Engineering, College of Engineering Chengannur for the support and co-operation.

We would like to place on record our sincere gratitude to our project guide , As-
sistant Professor, Department of Computer Engineering, College of Engineering
Chengannur for the guidance and mentorship throughout this work.

Akhil Nair
Nithin Johnson
S Balamurali
Sujin Skaria Philip

i
Abstract

Novels are the current sleeping giants of the Internet, where Content Sharing media
platforms reign supreme. Despite the fact that novels are available to read over
numerous platforms, they do not have the same impact as other entertainment
franchises. We believe one of the main reasons for this is the limited availability of
content on each platform. The difficulty of the users is yet another issue. To read their
favourite novels, they must use/download apps for each platform separately.

We are creating an app that will contain all of the novels from various types of
platforms, including both lower and higher rated platforms. We allow users to read
novels from various platforms in one app while adhering to all copyright guidelines
and respecting every Authors’ work. We are scraping the details and content of each
novel from each platform using the Web Scraping concept. As a result, having all
content on one platform gives each novel an equal chance of becoming popular, and
it also makes it easier for users because they won’t have to use multiple apps on their
devices.

ii
Contents

Acknowledgement i

Abstract ii

1 Introduction 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Contribution of the Project . . . . . . . . . . . . . . . . . . . . . . . 4

2 Software Requirement Specification 5

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.2 Intended Audience and Reading Suggestions . . . . . . . . . 5
2.1.3 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.4 Project Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Functional requirements . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Interface requirements . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 Performance Requirements . . . . . . . . . . . . . . . . . . . . . . . 7
2.5 Non functional atributes . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Project Design 9
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 System Architecture Design . . . . . . . . . . . . . . . . . . . . . . 9

iii
 3.3 Application Architecture Design . . . . . . . . . . . . . . . . . . . . 11
3.4 GUI Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.5 Database Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.6 Technology Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Implementation 16
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.2 System Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.3 Testing Plan and Scenarios . . . . . . . . . . . . . . . . . . . . . . . 17

5 Results 19

6 Future Scope and Conclusion 20

6.1 Future Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

References 22

iv
Chapter 1

Introduction

1.1 Introduction
The internet has played a significant role in entertainment. It has changed the way
that entertainment is viewed, consumed and also the way in which it’s received. Many
years ago, entertainment was primarily enjoyed in real time and live. Now, we are able
to watch things live and also watch replays of entertainment thanks to the provision
of the internet. As with everything, this has many implications for the entertainment
industry and has helped artists globally. It has also created an avenue for other sub-
forms of entertainment to emerge as a result.

Figure 1.1: Value of the entertainment from 2011 to 2023

The internet has millions of users who are searching for both information and
entertainment on a daily basis. There is an overload of entertainment available for

1
them to consume because of how easy it is to upload content and share it.

The internet has made entertainment more accessible. Now, people from anywhere in
the world have access to various content that is available. Seeing as 47 percentage of
the world’s population is said to now use the internet, this means that almost half of
the world have the ability to access and consume eclectic forms of entertainment.

The Novel Franchise is one of the entertainment franchises whose popularity is

declining at the same rate as Internet entertainment, which is rising at a continually
accelerating rate. Platforms for reading novels are what we think of as the internet’s
sleeping behemoth. Despite the fact that novels have a wide range of platforms, they
do not have the same influence as other entertainment properties.

Figure 1.2: Reading decline over the years

1.2 Motivation
Prior to a few years ago, traditional enterainment was used by the majority of
entertainment franchises. However, once technology began to affect them, they all
developed into a new shape. Because of this, we sought to include technology into the
Novel Franchise. We expect to observe a dramatic growth spurt for the Novel platforms
by incorporating technology. In order to improve reading rates in the coming years,
we decided to do something for the Novel community.

2
1.3 Problem Statement
The number of people who read novels is rapidly declining, and novel franchises are on
the verge of extinction. We made the decision to look into the causes of this constant
deterioration.

1. Limited Content - The limited availability of Content in each novel-reading

platforms, i.e, The best novels are scattered all over the Internet on different
platforms.

2. User Difficulty - The difficulty of the users. They have to use different platforms
to read their favorite novels, which is, in fact, Time Consuming.

3. Video Entertainment - People began to believe that reading is uninteresting as

video entertainment such as movies, reels, and more grew in popularity.

1.4 Proposed Solution

We came to a single conclusion in order to address the issues we discovered as well as
support Novel Franchise’s growth in popularity. We made the decision to use a mobile
application. A mobile application that enables users to read novels from a variety
of platforms in one app while abiding by all copyright regulations and honouring the
Author’s work. Using the web scraping idea, we are extracting the content of each
novel from each platform. Users won’t have to install different applications on their
devices as this application provides all contents on one platform. This makes it easier
for people to access the novels. Features included in this Application are:

• Multiple Sources - Contents from multiple sources are available.

• Bookmarks - Bookmarks user’s favorite novhecticin single place.

• Auto Updates - Automatically updates the chapters as soon as a new chapter

releases.

• Beautiful UI - A simple, yet a beautiful User Interface.

3
1.5 Contribution of the Project
Our application WNReader plays an important role in one of the most impactful sectors
of the society which forms a large proportion of online content readers. Unfortunately,
despite the numerous number of platforms already available, these are often riddled
with the challenge of not being able to provide favourites of each user.

Heightened amount of choice

WNreader gives access to multiple contents with a potentially infinite number of

resources and selections at hand. The main limit for traditional libraries is represented
by physical space: books consume a lot of it and people often have to walk round in
search of a particular material. Our application overcome this limitation, expanding
readers’ horizons in learning and facilitating the increased interest in reading habits.

Fighting against deterioration

The digital storage of books in our database solves the problem of deterioration of
novels available offline. In traditional libraries, paperbacks are shared among a lot of
people posing the problem to stand time.This makes our app definitely a much better
option.

An easier information retrieval

The difficulty in controlling and tracking the readers’ current reading progression is
one of the biggest issues for the novel communities. It might be possible in case of
one or two novels, but it is near to impossible when an user is reading many novels
concurrently. Thus, WNReader’s bookmarking features enable users to organize all
of the books they are presently reading, saving them from having to do so manually.
The user can keep track of how far he has read thanks to the other function, which
automatically tracks his reading progress.

4
Chapter 2

Software Requirement Specification

2.1 Introduction

2.1.1 Purpose

The purpose of this document is to define and describe the requirements of the project
and to spell out the system’s functionality and its constraints.

2.1.2 Intended Audience and Reading Suggestions

This Software Requirement Specification is for developers, project managers, users

and testers. Further, the discussion will provide all the internal, external, functional
and also non-functional information about the mobile application ”WNReader”.

2.1.3 Overview

WNReader is an app which scrapes the content of available novels from various
types ofplatforms, including both lower and higher rated platforms. This is achieved
by utilising a technology stack that involves Dart Programming Language, Flutter
Frameworks and other Web Scraping Libraries

5
2.1.4 Project Scope

This application software will be a Web Novel Reading system for a user anywhere
in the world. This application will be designed with an aim to maximize the reader’s
productivity by providing features such as regular novel chapter updates to assist in the
process, which would otherwise have to be performed manually by searching various
sites and finding the content of one’s interest and liking.
More specifically, this app is designed to allow a user to manage his/her favourites
and maintain it as a library of their favourite collections,irrespective of the public
website the novel belongs to. Novels can be searched with the help of a search bar
at the top and the progress of a reader reading a particular novel can be saved within
the app which acts as a bookmark. This helps the reader to continue reading the story
from where they left and therefore, helps in maintaining consistency in reading.

2.2 Functional requirements

• Multiple Sources - Contents from multiple novel sources are available in this
application. So user can search and read novels from different websites in this
single application.

• Bookmarks - WNReaders allow users to Bookmarks their novels into a single

place. We call the bookmarking feature as ’Add to Library’. When it is in library,
the app will organise the novels in the My Library page

• Auto Updates - WNReader automatically updates the novels as soon as a new

chapter releases. So, the user will know when a new chapter is released.

• Beautiful UI - WNReader has a simple yet beautiful User Interface.

2.3 Interface requirements

Front-end : Dart Programming Language, Flutter Framework
Back-end : SQLite, Web Scraping

6
Platforms Used : Android Studio

2.4 Performance Requirements

Hardware Requirements for WNReader are:

1. The Android application is required to have RAM that is at least 2GB in size. 4
GB of RAM is required for optimal performance.

2. Internal memory should be greater or equal to 16 GB of memory. However

Internal memory higher than 32 GB is necessary for uninterrupted and efficient
performance.

Software Requirements for WNReader are:

1. The Android version must be higher than version 4.0, however Android versions
higher than 10.0 are necessary for flawless performance.

2. Language: Dart Programming Language

2.5 Non functional atributes

A non-functional requirement (NFR) is a requirement that specifies criteria that can be
used to judge the operation of a system, rather than specific behaviours. Non-functional
attributes of WNReader are:

• Security - WNReader is a highly secure and safe to use application as it doesn’t

store or manipulate user’s data.

• Maintainability - Most faulty or error-producing components that may haunt

WNReader can be rectified in a single day.

• Compatibility - WNReader is compatible with Android version 4.0 and above.

• Capacity - WNReader takes less than 100 MB space of phone storage.

7
• Usability - Utilizing WNReader is pretty straightforward. All the features that
WNReader offers are clearly visible to someone who is unfamiliar with the
application. An intuitive user interface is employed.

8
Chapter 3

Project Design

3.1 Introduction
Project design is an early phase of the project lifecycle where ideas, processes,
resources, and deliverables are planned out. A project design comes before a project
plan as it’s a broad overview whereas a project plan includes more detailed information.

There are seven steps that make up a successful project design process. These include
everything from defining goals and baseline objectives to strengthening your strategy
to help you stay organized while managing a new project:

Step 1. Define project goals

Step 2. Determine outcomes
Step 3. Identify risks and constraints
Step 4. Refine your project strategy with a visual aid
Step 5. Estimate your budget
Step 6. Create a contingency plan
Step 7. Document your milestones

3.2 System Architecture Design

WNReader follows a database-screen transition method. The basic concept we use to
implement our application is Web Data Scraping. Firstly, we load the sources from the

9
source novel websites, scrape the latest novels and display them in a organised manner
on the screen.

The user can now select novels of their interest and choice. The app then checks
whether the requested novel is present in the database or not. If it is not available, we
add the novel into the database. If available, we display the novel from the database

Figure 3.1: System Architecture - Scraping

onto the screen. Information relating to the selected novel is now displayed on the
screen which includes an option to add to Library, a brief summary of the novel and
the corresponding chapters’ list.

If the user selects the ’Add to Library’ icon, the novel is added to the database and also
displayed on the Library screen. The user can remove the novel from the library by
clicking the same icon again. This results the novel to be removed from the Library
screen and the database.

When the user selects a chapter to be read from the chapter list, the database is checked
whether to be containing the requested chapter and if available in the database, the
chapter content is displayed on the reader screen. If the chapter is not available in the
existing database, it is updated to add the chapter..

10
 Figure 3.2: System Architecture - Library

3.3 Application Architecture Design

We first created an architecture design for our application prior to actually implement-
ing an intuitive and understandable interface. The novel titles will be on the first page,
which is the library page, and clicking them will take you to the novel details screen.
The app has bottom navigation that makes it simple for users to switch between pages.
The browse page, which contains several novel sources, is the second page we plan to
create. By clicking on the sources, the user can use various sources and view various
novels.

The page changes to the novel details screen after clicking on the novel titles. The
name, cover page, brief summary, and chapter list of the book are all included in the
details screen. A bookmark option is also provided on this page to aid users in saving

11
 Figure 3.3: Application Architecture Design

the novel. When you click on the respective chapter titles, the page changes to the
reader screen, which contains the entire text of each chapter of the novel.

3.4 GUI Design

The user interface for WNReader is straightforward, dynamic, and visually appealing.
It has a user-friendly, intuitive interface. To implement this Interface, we adhered to
the application architecture design that we created. Dart and the Flutter Frameworks
were used to bring our conceptions and ideas to life. The flutter function is used to
create the bottom navigation bar. It makes it easier for the user to switch between the
different application screens. Using the OnTap flutter feature, the novel’s cover and

12
title are added to a Flutter Card and made clickable. The sources list, which is made up

Figure 3.4: Landing Page-Library Figure 3.5: Settings Page

of list tiles with the logos of the relevant websites, can be found on the browse page.All
the novel titles appear as clickable cards on the website when the source list is clicked.
The Work Details screen, which has all the details about the selected novel, appears
when a novel is clicked. Using the pre-existing Flutter packages, the Add to library
button was created. The chapter list is clickable and is implemented as Flutter cards.
The reading with the entire scraped text from the associated literature appears as soon
as the user clicks the Chapter Name.

13
 Figure 3.6: Application Architecture Design

3.5 Database Design

SQLite is a C-language library that implements a small, fast, self-contained, high-
reliability, full-featured, SQL database engine. SQLite is the most used database
engine in the world.

SQLite is built into all mobile phones and most computers and comes bundled inside
countless other applications that people use every day. SQLite source code is in the
public-domain and is free to everyone to use for any purpose.

14
 Figure 3.7: Entity Relationship Table

3.6 Technology Stack

The tech stack is the combination of technologies that we used to build and run this
application. Tech stacks used for this application are:

1. Dart Programming Language - Dart is an open-source general-purpose pro-

gramming language made by Google with an excellent tooling support. Appli-
cations run in Dart run faster than in other programming languages

2. Flutter Frameworks - Flutter is an open source framework by Google for

building beautiful, natively compiled, multi-platform applications from a single
codebase. Flutter has its own Rendering Engine.

3. HTML and CSS - Both HTML and CSS are two of the core technologies for
building Web pages. HTML provides the structure of the page, CSS the layout,
for a variety of devices.

15
Chapter 4

Implementation

4.1 Introduction
Systems implementation is a set of procedures performed to complete the design (as
necessary) contained in the approved systems design document and to test, install, and
begin to use the new or revised Information System.

The systems implementation goals are as follows:

Complete as necessary the design contained in the approved systems design doc-
ument.For example, the detailed contents of new or revised documents, computer
screens, and database must be laid out and created. Write, test, and document
the programs and procedures required by the approved systems design document.
Ensure, by completing the preparation of user manuals and other documentation and
by training personnel, that the organization’s personnel can operate the new system.
Determine, by thoroughly testing the system with users, that the system satisfies
the users’ requirements. Ensure a correct conversion by planning, controlling, and
conducting an orderly installation of the new system.

4.2 System Implementation

When the app is initially started, the starting page will be library screen, which displays
all the novels followed by the user, if any. The user can navigate to the browse section

16
or the settings section using the bottom navigation bar. The browse screen shows a list
of sources available for the user to find any novels they desire, if it exists.

The novels are fetched from the websites using the http package. The response body
of the fetched data is parsed using the parser package, and the parsed data is processed
further to extract the necessary contents for use.

After the processing data is extracted, the necessary information is displayed on each
specific screen that the information belongs to. All the novels are displayed on the
screen using a GridView widget which builds a grid of cards containing the novel
cover images and the novel names.
When navigating to a specific novel (i.e. the novel screen), extra details of that
novel such as the novel name, cover image, novel summary, follow/unfollow button
and list of chapters is displayed using the Column widget, and since the novel contains
a list of chapters, the chapters are displayed using a ListView widget.

From the novel screen, it is possible for the user to add the novel to library or remove
it. The user can read any chapter by selecting the desired chapter which navigates the
user to the reader screen which displays the chapter title and the chapter text. Since the
chapter content is large, it won’t fit into the screen, and thus we use column widget to
make the screen content scrollable.

4.3 Testing Plan and Scenarios

We created testing steps to make sure the application had all the functionality the users
require. The testing steps we adopted are:

• Step 1: Develop Test Strategy - To check whether WNReader is useful for the
community.

• Step 2: Define Scope of Testing - The functionalities and components to be

tested.

• Step 3: Identify Testing Type - To decide which Testing Types should be focused
on.

17
 • Step 4: Risk and Issues - To Identify the probability of occurrence of an
unwanted event.

• Step 5: Create Test Logistics - To decide who will test and when will the test
occur.

In order to find issues in our application and to determine which features are required
inside of it, we are now employing Alpha testing methodologies. Software testing
known as ”Alpha Testing” is done to find issues before a product is made available to
the general public or real users. Before the official release, we also intend to conduct
beta testing.

18
Chapter 5

Results

WN reader is a Webnovel scraping application which is used to scrap novel contents

from different novel websites. The user has the facility of adding the corresponding
novel to the libarary screen and is working well. The features of the app are perfectly
running which includes scraping part as well as database part.

Figure 5.1: WNReader - Result Pages

19
Chapter 6

Future Scope and Conclusion

6.1 Future Scope

WNReader is an application with a lot of development potential. Currently, it is
available for use in Android devices only. We are planning to release an iOS supported
version of the app soon.
Future Plans:

• Improved Download and Delete features.

• Bookmarking.

• Inclusion of more sources.

We see WNReader as an app which can grow together with the Novel Franchise to
create a platform where novels from round the globe are consolidated for ease of access
to reading.

6.2 Conclusion
In Conclusion, this developed application augers well with Agile development method.
Usability engineering can be adapted to the agile context, for example by using a more
iterative approach and testing throughout the Project Lifecycle.
As a group, we have well understood the importance of planning and designing as
a part of software development. In our perspective of the app, the users are considered

20
as an important indicator for development that would determine further project growth
and success. By minimizing limitations, we aim to evolve an integrated code which
might prove to be useful for the users and make our endeavour a meaningful one.

21
References

1. [Flutter Docs] - https://docs.flutter.dev/

2. [Flutter Packages] - https://pub.dev/packages/

3. [Web Scraping Docs] - https://www.techopedia.com/definition/5212/web-scraping

4. [SQLite Docs] - https://www.sqlite.org/docs.html

5. [Novel Source 1] - https://boxnovel.me/

6. [Novel Source 2] - https://novelfull.com/

22