DMAP: Data Aggregation and Presentation Framework
DMAP (Data Mining and Automation for Platforms) is an online framework that presents a wide variety of official data, news and information about companies. It was developed with an aim to act as a one-stop platform for displaying everything official related to a company and its competitors. It aggregates data from the following sources: Bing News, Companys' official blogs, RSS Sources, Facebook, Twitter, Google Trends, Crunchbase, Financial Data Sources, and Alexa Web Analytics. The aggregated information is shown in an intuitive fashion that allows a user to perform exploratory analysis of a particular company. Specifically, the interface makes it easy to do comparative analysis of a company with respect to its competitors. The user can use filters to select a given set of companies, data sources and date ranges. All the information is presented within the context of the framework such that the user doesn't have to go to different domains. The fetched news articles are cleaned, enriched, and presented in a manner that allows an analyst to navigate through news articles using named entities. For each news article, named entities: people, location, organization, and name of companies are extracted. For a given search query, the interface returns matching news articles along with associated entities which are shown using word clouds. This allows for easy discovery of connections between entities. The framework was developed as a part of the 2015 summer internship with The Center for Global Enterprise (CGE). Conceptualization, Design and Development of the framework was done by me during the three month period. For more information, please visit: http://people.cs.vt.edu/parang/ or contact parang at firstname at cs vt edu
![[DrupalCon] Erase Unconscious Bias From Your AI Datasets](https://cdn.slidesharecdn.com/ss_thumbnails/drupalconeraseunconsciousbiasfromyouraidatasets-190508202014-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (19)
Similar to DMAP: Data Aggregation and Presentation Framework (20)
More from Parang Saraf (20)
Recently uploaded (20)
DMAP: Data Aggregation and Presentation Framework
- 1. DMAP – Data Mining Automa/on for Pla3orms Parang Saraf Ph.D. Candidate Discovery Analy/cs Center Department of Computer Science Virginia Tech [email protected]
- 2. What is DMAP? • DMAP is a system for aggrega/ng official informa/on about a company from wide range of data sources – News Media – Official Company Blog Posts – Official TwiJer and Facebook Handles • The intui/ve presenta/on of ar/cles helps in analyzing them faster and beJer 2
- 3. Objec/ve 1. Overview of the system architecture 2. Example News data files 3. Ravens Interface 4. Overview of the web architecture 5. Example RSS data files 6. Horse’s Mouth Interface 7. Example Facebook and TwiJer data files 8. Gossipers Interface 9. Moving from Proof of Concept to Produc/on System 3
- 4. DMAP -‐ Objec/ve • A one stop pla3orm for displaying everything official related to a company – News Sources (Ravens) – Official Blogs (Horse’s Mouth) – Official TwiJer and Facebook Accounts (Gossipers) – Other Informa/on (Facts) • Company Informa/on • Web ac/vity informa/on • Financial Informa/on 4
- 6. System Architecture Data Collec/on Through Official APIs • We use official APIs to collect data – No Scraping • Why use Bing and not Google News? • Example search result data • Bing Cost Calcula/on 6
- 7. System Architecture Data Cleaning and Enrichment • Structured data doesn’t mean clean data • Fetch the original ar/cle by following the url in the search results • Remove Boilerplate • Filter again for the keyword • Enrich ar/cle with Basis En/ty Extractor 7
- 8. System Architecture Duplicate Detec/on and Data Loading • Search for duplicate content • Load data in the database 8
- 9. Ravens Interface • Results control op/ons • Dynamically generated Google Trends chart • Dynamically generated Word Clouds • Ar/cle Detailed View • Ar/cle specific word clouds What happens behind-‐the-‐scenes when you click the green “SUBMIT” buJon? 9
- 10. System Architecture Web Architecture Model Controller View (MVC) Framework 10 Server-‐Side Model Controller View Client-‐Side 40% 60%
- 11. System Architecture What happens aier clicking “Submit” 11 • Client-‐side Sanity Check • Checks wriJen • If everything is as expected, sends the query parameters to the server
- 12. System Architecture What happens aier clicking “Submit” 12 1. Performs basic sanity checks 2. Get qualifying ar/cle set based on search parameters 3. Iden/fy the 10 search results from this set based on page number 4. Do a frequency count of all en//es (People, Loca/on, Organiza/on and Product) and pick the top 50 in each category and determine their font size based on frequency. 5. Generate Google Trends chart data 6. Return everything back to the client
- 13. System Architecture What happens aier clicking “Submit” 13 1. Display the returned search results 2. Embed dynamically generated Google Search Trends 3. Dynamically Generate word clouds using D3
- 14. RSS Reader • Sample blog feed • Problem: How to know if someone has published a new post? • Use a feed reader – Inoreader: – How to subscribe to a blog in inoreader – Checks blogs at regular intervals for new posts • Wrote scripts that downloads “new” posts found by inoreader • Sample blog post example fetched through API 14
- 15. Horse’s Mouth • Selected companies carry over • Now ar/cles are limited only to the official blogs 15
- 16. TwiJer API • Have to abide by /me-‐based rates while fetching data • Tweets and replies both count towards tweets from an account – We are only interested in tweets – Discard reply tweets – There can be a lag while fetching tweets • Sample Tweet fetched through API 16
- 17. Facebook API • Can’t be automated. Have to generate a new authoriza/on every /me before fetching data • Pictures are provided in low resolu/on • Sample Facebook post fetched through API 17
- 18. Gossipers • Solves the feed personaliza/on problem in true sense • TwiJer allows embedding of tweets: – Leads to exact same presenta/on format as they appear on twiJer.com – The media links are all embedded and works • Facebook doesn’t allow embedding of posts due to privacy issues – Have to recreate the presenta/on look and style – Media links don’t work • Pagina/on works individually for both of them 18
- 19. Facts • Will show the following: – Sta/c informa/on about the company – Financial Informa/on – Web usage data – Other miscellaneous informa/on 19
- 20. 20 Thank You