CSE5656 Complex Networks - Gathering Data from Twitter
1 like323 views
The document discusses gathering data from Twitter using its APIs. It provides an overview of Twitter as a social network and describes the main Twitter APIs including the REST, Search, and Streaming APIs. It also outlines some key Twitter API objects like Tweets, Users, Entities, and Places. The document recommends getting familiar with the Twitter apps, checking rate limits, and considering database needs when collecting large amounts of Twitter data.
CSE5656 Complex Networks - Gathering Data from Twitter
- 1. Gathering Data from Twitter Marcello Tomasini – [email protected] Marcos Oliveira – [email protected] Adapted from 2013 Hugo Serrano’s and 2014 Diogo Pacheco’s Lectures
- 2. Introduction ▪ What is Twitter ▪ Twitter is an online social network that allows users to send and read messages of up to 140 characters, known as tweets. ▪ https://twitter.com
- 3. Introduction ▪ Twitter ▪ Online Social Network ▪ Users can follow and be followed by other users. ▪ A Real Time Information Network ▪ Tweets can be replied, retweeted, and favorited.
- 4. Twitter APIs ▪ REST APIs ▪ Access core primitives of Twitter ▪ provides programmatic access to read and write Twitter data ▪ Author a new Tweet, retweet a Tweet, read author profile and follower data, and more ▪ Search API ▪ Finding a set of Tweets, Tweets from a particular user, or Tweets with specific keywords/hashtags ▪ focused on relevance and not completeness ▪ Streaming APIs ▪ Continuously deliver new responses to REST API queries ▪ Useful for Data Mining & Analytics research ▪ When application is rate-limited for over-polling the REST APIs ▪ Large quantity of keywords ▪ Ads APIs ▪ https://dev.twitter.com/overview/documentation
- 5. REST APIs ▪ Communicate over HTTP with the same HTTP verbs (GET, POST) ▪ Twitter limits the number of requests per 15 minutes intervals ▪ https://dev.twitter.com/rest/public/rate-limiting ▪ https://dev.twitter.com/rest/public/rate-limits ▪ Detailed reference of Search API endpoint ▪ https://dev.twitter.com/rest/public/search ▪ https://dev.twitter.com/rest/reference/get/search/ tweets
- 6. Streaming APIs ▪ Low latency access to Twitter’s global stream of Tweet data. ▪ Twitter offers several streaming endpoints ▪ Public Streams ▪ https://dev.twitter.com/streaming/public ▪ Streams of the public data flowing through Twitter. ▪ Suitable for following specific users or topics, and data mining ▪ User Streams ▪ https://dev.twitter.com/streaming/userstreams ▪ Contain roughly all of the data corresponding with a single user’s view of Twitter. ▪ Site Streams ▪ https://dev.twitter.com/streaming/sitestreams ▪ Multi-user version of user streams. Site streams are intended for servers which must connect to Twitter on behalf of many users. ▪ Require a long-lived HTTP connection
- 7. API Objects ▪ The most frequently observed objects in Twitter ecosystem: ▪ Tweets ▪ Users ▪ Entities ▪ Places
- 8. API Object: Tweets ▪ Basic building blocks ▪ Also known as “status updates” ▪ Fields such as ▪ created_at: UTC time when this Tweet was created. ▪ coordinates: the geographic location of this Tweet. ▪ in_reply_to_screen_name: the name of the original Tweet's author. ▪ in_reply_to_status_id: the id of the original Tweet's author. ▪ retweeted_status: a representation of the original Tweet that was retweeted. ▪ https://dev.twitter.com/overview/api/tweets
- 9. API Object: Users ▪ They tweet, follow, retweet, are mentioned etc. ▪ Fields such as: ▪ id: the unique identification of a user ▪ screen_name: the screen name, handle, or alias that this user identifies themselves with ▪ followers_count: the number of followers this account currently has ▪ https://dev.twitter.com/overview/api/users
- 10. API Object: Entities ▪ Provide metadata and additional information about content posted on Twitter: ▪ media ▪ urls ▪ hashtags ▪ user_mentions ▪ https://dev.twitter.com/overview/api/entities ▪ https://dev.twitter.com/overview/api/entities-in- twitter-objects
- 11. API Objects: Places ▪ Specific, named locations with corresponding geo coordinates ▪ Tweets associated with places are not necessarily issued from that location but could also potentially be about that location ▪ Fields such as ▪ country ▪ bounding_box ▪ full_name ▪ place_type ▪ https://dev.twitter.com/overview/api/places
- 12. Twitter APIs: OAuth Authentication ▪ Create a Twitter profile ▪ Create an application (you need a phone number) ▪ Create Access Tokens ▪ Use the OAuth tool to get your keys, tokens and secrets ▪ https://apps.twitter.com
- 13. Twitter Libraries ▪ For some libraries (python), the Twitter API website is the documentation for the developers ▪ https://dev.twitter.com/overview/api/twitter-libraries -> api.search.tweets(q=“#HowToComplexNetworks”)
- 14. Useful Hints ▪ Get accustomed with Twitter App
- 15. Useful Hints ▪ Try first Twitter Console before coding https://dev.twitter.com/rest/tools/console
- 16. Useful Hints ▪ Check rate limits in your code – try/catch loops ▪ Do you need database? Relational one? ▪ Remember 2-step process: ▪ Collect data ▪ Construct the network ▪ Intensive traffic/volume, try MongoDB
- 17. Useful Hints? ▪ 2014 FIFA World Cup Dataset ▪ +50 millions tweets ▪ +1 million geo-tagged tweets ▪ 1 month window period ▪ Database Optimization ▪ Memory Consuming ▪ Programs Crashes
- 19. Twitter API - Resources ▪ Timelines ▪ Search ▪ Tweets ▪ Streaming ▪ Direct Messages ▪ Friends & Followers ▪ Users ▪ Suggested Users ▪ Favorites ▪ Lists ▪ Saved Searches ▪ Places & Geo ▪ Trends ▪ Spam Reporting ▪ OAuth ▪ Help
- 20. Twitter API - Resources ▪ Search ▪ Let’s search for 5 most recent tweets with “Florida Tech” on it ▪ Let’s search for 5 most recent tweets within 1mi
- 21. Twitter API – Resources ▪ Friends & Followers ▪ Let’s query for the friends of a particular user ▪ Let’s query for the first 100 followers of a particular user
- 22. Twitter API - Resources ▪ Users ▪ Let’s query for the usernames of the first 100 followers of a particular user
- 23. © Ronaldo Menezes, Florida Tech Twitter API - Resources ▪ Streaming ▪ Let’s track tweets with the hashtag #cse5656 and #netsci ▪ Let’s track the tweets in Brevard County
- 24. Questions? Marcello Tomasini – [email protected] Marcos Oliveira – [email protected] http://my.fit.edu/~mtomasini/classes/ComplexNetworks.html