Mongo db dla administratora
Download as ODP, PDF0 likes554 views
The document discusses using MongoDB as a log collector. It provides an agenda that includes who the presenter is, how logging is currently done, and ideas for using MongoDB for logging in the future. Specific topics covered include using syslog-ng to send logs to MongoDB, examples of logging Apache traffic, and map-reduce examples for analyzing logs like finding the top 10 IP addresses.
![afmongodb
Przykładowy log z HAProxy:
20120402T21:10:25+02:00 10.x.y.z haproxy[2913]: Connect
from 80.55.238.74:3935 to 195.8.99.31:80 (dom.gratka.pl/
HTTP)
HAProxy -> syslog-ng(local) -> syslog-ng(remote) -> mongodb](https://image.slidesharecdn.com/mongodbdlaadministratora-120403162447-phpapp02/85/Mongo-db-dla-administratora-10-320.jpg)
![afmongodb
destination d_mongodb {
mongodb(
valuepairs(
scope("selectedmacros" "nv
pairs")
exclude("DATE")
exclude("HOST")
exclude("LEGACY_MSGHDR")
[...]};](https://image.slidesharecdn.com/mongodbdlaadministratora-120403162447-phpapp02/85/Mongo-db-dla-administratora-11-320.jpg)
![Pomysły – Top10 IP
m=function () {
emit(this.MESSAGE, {count:1});
}
r=function (key, values) {
var total = 0;
for (var i = 0; i < values.length; i++) {
total += values[i].count;
}
return {count:total};
}
db.messages.mapReduce(m, r, { out : "myoutput" } );
db.myoutput.find().sort({value:1});](https://image.slidesharecdn.com/mongodbdlaadministratora-120403162447-phpapp02/85/Mongo-db-dla-administratora-19-320.jpg)
More Related Content
What's hot (20)
Viewers also liked (7)
Similar to Mongo db dla administratora (20)
Recently uploaded (20)
Mongo db dla administratora
- 2. MongoDB dla administratora Łukasz Jagiełło – [email protected] http://geekandpoke.typepad.com/geekandpoke/2011/01/nosql.html
- 3. Agenda Kto ja i co ja Jak to się robi dzisiaj Jak to można zrobić jutro Pomysły...
- 4. Kto ja i co ja ? Ukradzione z wyszukiwania w Grafika google.com
- 5. Jak to się robi dzisiaj
- 6. Jak to się robi dzisiaj filter f_kern { facility(kern); }; filter f_cron { facility(cron); }; destination d_kern { file("/var/log/$YEAR$MONTH$DAY/kern"); }; destination d_cron { file("/var/log/$YEAR$MONTH$DAY/cron"); }; log { source(src); filter(f_kern); destination(d_kern); }; log { source(src); filter(f_cron); destination(d_cron); };
- 7. Jak to się robi dzisiaj
- 8. Jak to się robi dzisiaj filter f_kern { facility(kern); }; filter f_cron { facility(cron); }; destination d_collector { udp("10.X.Y.Z" port(514)); }; log { source(src); destination(d_collector); };
- 9. Syslog-ng 3.4.0alpha1 Moduły: File JSON MongoDB Stream SQL CSV Socket Network Console
- 10. afmongodb Przykładowy log z HAProxy: 20120402T21:10:25+02:00 10.x.y.z haproxy[2913]: Connect from 80.55.238.74:3935 to 195.8.99.31:80 (dom.gratka.pl/ HTTP) HAProxy -> syslog-ng(local) -> syslog-ng(remote) -> mongodb
- 13. MongoDB „Capped collections are fixed sized collections that have a very high performance auto-FIFO age-out feature (age out is based on insertion order). They are a bit like the "RRD" concept if you are familiar with that.” „In addition, capped collections automatically, with high performance, maintain insertion order for the documents in the collection; this is very powerful for certain use cases such as logging.” > db.createCollection("mycoll", {capped:true, size:100000}) > db.createCollection("messages", {capped: true, size:10000000, autoIndexId:true})
- 14. MongoDB > db.messages.find() { "MESSAGE" : "67.162.55.226", "_id" : ObjectId("4f7a0072c90d34ea7d2495db" ) } { "MESSAGE" : "193.151.114.11", "_id" : ObjectId("4f7a0072c90d34ea7d2495dc" ) } { "MESSAGE" : "89.167.68.120", "_id"
- 15. MongoDB Puchnące mongo ? 17772 root 15 0 1463m 474m 276m S 0.0 8.0 2011 147:00 ./mongod fork logpath /d0/db/mongod.log ponad 5 miesięcy działania średnio co 30min wszystkie dane są wymienione 10mln rekordów w bazie – zawsze ! 4GB na dysku !! Wesje 2.x działają DUŻO lepiej niż wcześniejsze !!
- 19. Pomysły – Top10 IP m=function () { emit(this.MESSAGE, {count:1}); } r=function (key, values) { var total = 0; for (var i = 0; i < values.length; i++) { total += values[i].count; } return {count:total}; } db.messages.mapReduce(m, r, { out : "myoutput" } ); db.myoutput.find().sort({value:1});
- 20. Pomysły – Top10 IP > db.messages.mapReduce(m, r, { out : "myoutput" } ); db.myoutput.find().sort({value:1}); { "_id" : "195.8.99.115", "value" : { "count" : 20771 } } { "_id" : "86.111.244.239", "value" : { "count" : 6816 } } { "_id" : "94.75.116.206", "value" : { "count" : 6716 } } { "_id" : "195.137.162.149", "value" : { "count" : 5638 } } { "_id" : "178.63.0.2", "value" : { "count" : 4902 } } { "_id" : "31.187.3.80", "value" : { "count" : 3573 } } { "_id" : "176.9.73.105", "value" : { "count" : 2734 } } { "_id" : "62.129.228.159", "value" : { "count" : 2092 } } { "_id" : "209.123.8.176", "value" : { "count" : 2016 } } { "_id" : "107.20.40.240", "value" : { "count" : 1927 } }
- 23. Pomysły cd. CAPTCHA Lokalizowanie błędów Kontrola Liczniki (youporn.com 8GB-15GB/godzinę) Itd. Do statystyk mamy ciekawsze narzędzia