Ab(Using) the MetaCPAN API for Fun and Profit v2013
13 likes6,006 views
The document provides an overview of using the MetaCPAN API, including its architecture based on Elasticsearch and various endpoint functionalities. It discusses convenience endpoints, special cases for pod requests, the necessity of proper content types, and also suggests practices such as using compression and handling scrolling for large queries. Additionally, it emphasizes good habits when working with the API and offers resources for further assistance.
Ab(Using) the MetaCPAN API for Fun and Profit v2013
- 1. (Ab)Using the MetaCPAN API for Fun and Profit v2013 Olaf Alders (OALDERS) @wundercounter Thursday, 13 June, 13
- 2. Architecture • Built on ElasticSearch • Uses Catalyst as a thin wrapper Thursday, 13 June, 13
- 3. Objectives • Rough introduction to the API • Some ElasticSearch pointers Thursday, 13 June, 13
- 4. ElasticSearch • ElasticSearch (ES) has indexes and types • In RDBMS terms? • index = database • type = table • OK to think of it that way Thursday, 13 June, 13
- 5. Let’s look at types • Every type has a corresponding endpoint Thursday, 13 June, 13
- 6. Thursday, 13 June, 13
- 7. Thursday, 13 June, 13
- 9. Endpoints • Convenience endpoints • ElasticSearch endpoints • There *is* some overlap Thursday, 13 June, 13
- 10. Think of it this way • Every type has a corresponding endpoint • Not every endpoint has a corresponding type Thursday, 13 June, 13
- 11. Convenience Endpoints Thursday, 13 June, 13
- 12. Convenience Endpoints • /author/DOY • /distribution/Moose • /release/Moose • /module/Moose • /pod/Moose • /search/autocomplete?q=Moose • /search/reverse_dependencies/Moose Thursday, 13 June, 13
- 13. Versioned Convenience Endpoints • /release/DOY/Moose-2.0001 • /module/DOY/Moose-2.0001/lib/Moose.pm • /pod/DOY/Moose-2.0001/lib/Moose.pm • /search/reverse_dependencies/DOY/Moose-2.0001 Thursday, 13 June, 13
- 14. Special Cases • Some convenience URLs don’t expose ElasticSearch’s DSL Thursday, 13 June, 13
- 15. /pod • GET your /pod • Don’t send JSON in your request • Don’t expect JSON in your response Thursday, 13 June, 13
- 16. Set content-type via param or header • /pod/Moose?content-type=text/html (default) • /pod/Moose?content-type=text/plain • /pod/Moose?content-type=text/x-pod • /pod/Moose?content-type=text/x-markdown Thursday, 13 June, 13
- 17. /pod the easy way use MetaCPAN::API; my $mcpan = MetaCPAN::API->new; my $html_pod = $mcpan->pod( module => 'Carton', 'content-type' => 'text/plain' ); Text (MetaCPAN::API::Tiny has the same syntax) Thursday, 13 June, 13
- 18. Thursday, 13 June, 13
- 19. Thursday, 13 June, 13
- 20. Autocomplete Made Easy • /v0/search/autocomplete?q=HTTP • GET this endpoint • No JSON in • Expect JSON out Thursday, 13 June, 13
- 21. Thursday, 13 June, 13
- 22. Alfred 2 Autocompletion https://github.com/2shortplanks/alfred-metacpan Thursday, 13 June, 13
- 23. The (real) Endpoints Thursday, 13 June, 13
- 24. The (real) Endpoints • /author Thursday, 13 June, 13
- 25. The (real) Endpoints • /author • /distribution Thursday, 13 June, 13
- 26. The (real) Endpoints • /author • /distribution • /favorite Thursday, 13 June, 13
- 27. The (real) Endpoints • /author • /distribution • /favorite • /file Thursday, 13 June, 13
- 28. The (real) Endpoints • /author • /distribution • /favorite • /file • /rating Thursday, 13 June, 13
- 29. The (real) Endpoints • /author • /distribution • /favorite • /file • /rating • /release Thursday, 13 June, 13
- 30. Why is /module missing? • A module is a kind of file • file.module.name = $module_name Thursday, 13 June, 13
- 31. One special endpoint • https://api.metacpan.org/v0/user • You need to be logged in • Note the SSL ^^^ Thursday, 13 June, 13
- 32. Exploring Endpoints • Documentation is weak • You may need to dig around a bit Thursday, 13 June, 13
- 33. MetaCPAN Explorer Thursday, 13 June, 13
- 34. Thursday, 13 June, 13
- 35. Thursday, 13 June, 13
- 36. Thursday, 13 June, 13
- 37. Dig into the API code • https://github.com/CPAN-API/cpan- api/blob/master/lib/MetaCPAN/ Document/Author.pm Thursday, 13 June, 13
- 38. Good Habits • Make your life easier Thursday, 13 June, 13
- 39. Enable Compression • use WWW::Mechanize::Gzip • use WWW::Mechanize::Cached::GZip • Or set the appropriate request header Thursday, 13 June, 13
- 40. POST POST POST • Beyond the most trivial of queries, GET is a PITA. • POSTing will make your life easier Thursday, 13 June, 13
- 41. Are you asking for too much? • There is an upper limit of 5,000 on query size • If you need more data points, use the scrolling API Thursday, 13 June, 13
- 42. Scrolling • The scrolling API allows you to iterate over an arbitrary number of results • Be aware that when you scroll, your docs will come back unsorted • Let ElasticSearch.pm take care of the details Thursday, 13 June, 13
- 43. query vs filter context • generally you just want filters • use a query if you need to sort your results by relevance • https://metacpan.org/module/DRTECH/ElasticSearch- SearchBuilder-0.18/lib/ElasticSearch/SearchBuilder.pm#QUERY- FILTER-CONTEXT Thursday, 13 June, 13
- 44. Let’s Get Started Thursday, 13 June, 13
- 45. Let’s Get Started • We want to fetch some data Thursday, 13 June, 13
- 46. Let’s Get Started • We want to fetch some data • TIMTOWDI Thursday, 13 June, 13
- 47. Code Samples • https://github.com/cpan-api/ metacpan-examples Thursday, 13 June, 13
- 48. In closing... Thursday, 13 June, 13
- 49. Hack on *MetaCPAN* • Download a pre-configured VM • Requires VirtualBox + vagrant • https://github.com/cpan-api/ metacpan-developer Thursday, 13 June, 13
- 50. Production • Before you go live... Thursday, 13 June, 13
- 51. Rule #1 • Be polite Thursday, 13 June, 13
- 52. Rule #2 • There isn’t actually a rule #2. Just be polite. Thursday, 13 June, 13
- 53. Identify Yourself • Not mandatory, but encouraged • Do this when you’re in production • https://github.com/CPAN-API/cpan-api/wiki/API-Consumers Thursday, 13 June, 13
- 54. Getting Help • #metacpan on irc.perl.org • https://metacpan.org/about/resources Thursday, 13 June, 13
- 56. Bonus Slides • These apps all use the MetaCPAN API Thursday, 13 June, 13
- 57. iCPAN - iPhone Thursday, 13 June, 13
- 58. iCPAN - iPad Thursday, 13 June, 13
- 59. Android Thursday, 13 June, 13
- 60. Thursday, 13 June, 13
- 61. Thursday, 13 June, 13
- 62. Thursday, 13 June, 13
- 63. Thursday, 13 June, 13
- 64. Thursday, 13 June, 13
- 65. Thursday, 13 June, 13
- 66. Thursday, 13 June, 13
- 67. Thursday, 13 June, 13
- 68. Thursday, 13 June, 13