Perceval, Graal and Arthur: The Quest for Software Project Data
1 like170 views
Perceval gathers data from software repositories through backends. Graal enhances Perceval to analyze source code from repositories, running tools on checked-out code and embedding results. Arthur allows running Perceval and Graal at scale through distributed Redis queues. The tools provide open source software for software analytics through extraction of data from repositories and analysis of source code changes.
![Perceval, Graal and Arthur
The Quest for Software Project Data
Jesus M. Gonzalez-Barahona, Santiago Dueñas, Valerio Cosentino
@jgbarah, @sduenasd, @_valcos_
[jgb, sduenas, valcos] at bitergia.com
https://speakerdeck.com/bitergia
Open Source Summit North America,
Vancouver,
August 29th-31st, 2018](https://image.slidesharecdn.com/ossnapercevalgraalarthur-190901092332/85/Perceval-Graal-and-Arthur-The-Quest-for-Software-Project-Data-1-320.jpg)
![/perceval As a program
$ pip3 install perceval
$ perceval github chaoss grimoirelab-perceval
--from-date=2017-03-01 --api-token=5…
[2018-08-01 12:31:10] - Sir Perceval is on his quest.
[2018-08-01 12:31:11] - Getting info for
https://api.github.com/users/jgbarah
[2018-08-01 12:31:12] - Getting info for
https://api.github.com/users/sduenas
… producing JSON documents …
[2018-08-01 12:34:22] - Sir Perceval completed his quest](https://image.slidesharecdn.com/ossnapercevalgraalarthur-190901092332/85/Perceval-Graal-and-Arthur-The-Quest-for-Software-Project-Data-13-320.jpg)
![/perceval
As a Python3 library
from perceval.backends.core.github import GitHub
from_date = datetime.datetime(2017, 3, 1)
github = GitHub("chaoss", "grimoirelab-perceval",
api_token="5e3d7...")
for issue in github.fetch(from_date=from_date):
print(issue['data'])](https://image.slidesharecdn.com/ossnapercevalgraalarthur-190901092332/85/Perceval-Graal-and-Arthur-The-Quest-for-Software-Project-Data-14-320.jpg)
![/perceval
{ "backend_name": "GitHub",
"backend_version": "0.17.2",
"category": "issue",
"data": {
"comments": 5,
"comments_data": {...}
"created_at": "2017-02-28T05:33:10Z",
....,
"id": 210691361,
"state": "closed",
"title": "..",
"updated_at": "2017-03-02T09:51:49Z"
},
"origin": "https://github.com/chaoss/grimoirelab-perceval",
"perceval_version": "0.11.7",
"tag": "https://github.com/chaoss/grimoirelab-perceval",
"timestamp": 1517479766.878609,
"updated_on": 1488448309.0,
"uuid": "77a42463d5d0a34b1d58006263b85909a9788b52" }
Data source
data
}
]Perceval
data](https://image.slidesharecdn.com/ossnapercevalgraalarthur-190901092332/85/Perceval-Graal-and-Arthur-The-Quest-for-Software-Project-Data-15-320.jpg)
![/graal As a program
$ pip install graal
$ sudo apt-install cloc
$ graal cocom
https://github.com/chaoss/grimoirelab-perceval
--git-path /tmp/graal-cocom
[2018-05-30 18:22:35,643] - Starting the quest for the
Graal.
[2018-05-30 18:22:39,958] - Git worktree /tmp/...
created!
[2018-05-30 18:22:39,959] - Fetching commits
… producing JSON documents …
[2018-05-31 04:51:56,112] - Quest completed.](https://image.slidesharecdn.com/ossnapercevalgraalarthur-190901092332/85/Perceval-Graal-and-Arthur-The-Quest-for-Software-Project-Data-21-320.jpg)
![/graal
As a Python3 library
from graal.backends.core.cocom import CoCom
repo_uri =
"https://github.com/chaoss/grimoirelab-perceval"
repo_dir = "/tmp/graal-cocom"
cc = CoCom(uri=repo_uri, gitpath=repo_dir)
commits = [commit for commit in cc.fetch()]](https://image.slidesharecdn.com/ossnapercevalgraalarthur-190901092332/85/Perceval-Graal-and-Arthur-The-Quest-for-Software-Project-Data-22-320.jpg)
![/graal
{ "backend_name": "CoCom",
"backend_version": "0.2.1",
"category": “code_complexity",
"data": {
"AuthorDate": “Mon May 28 ...”,
"CommitDate": “Mon May 28 ...”,
"commit": "dc78c25….",
"message": "Increase coverage ...",
"analysis": [{
"ccn": 80, “num_funs”: 33, …,
"comments": "153", “loc”: 341,
"file_path": "perceval/backend.py",
}], ...
},
"origin": "https://github.com/chaoss/grimoirelab-perceval",
"graal_version": "0.1.0",
"tag": "https://github.com/chaoss/grimoirelab-perceval",
"timestamp": 1534782753.878609,
"updated_on": 1392000436.0,
"uuid": "77a42463d5d0a34b1d58006263b85909a9788b52" }
Git metadata
}
]Graal
data
} Code data](https://image.slidesharecdn.com/ossnapercevalgraalarthur-190901092332/85/Perceval-Graal-and-Arthur-The-Quest-for-Software-Project-Data-23-320.jpg)
Perceval, Graal and Arthur: The Quest for Software Project Data
- 1. Perceval, Graal and Arthur The Quest for Software Project Data Jesus M. Gonzalez-Barahona, Santiago Dueñas, Valerio Cosentino @jgbarah, @sduenasd, @_valcos_ [jgb, sduenas, valcos] at bitergia.com https://speakerdeck.com/bitergia Open Source Summit North America, Vancouver, August 29th-31st, 2018
- 2. /chaoss
- 3. /grimoirelab Software development analytics with free, open source software (a CHAOSS project) chaoss.github.io/grimoirelab chaoss.github.io/grimoirelab-tutorial
- 4. /bitergia Software Development Analytics for your peace of mind
- 6. /context
- 7. /context Are there new contributors?
- 8. /context How many bugs were fixed past month?
- 9. /context Has our gender diversity increased lately?
- 10. /context Perceval gathers data for you chaoss/grimoirelab-perceval
- 11. /perceval
- 12. /perceval Backend gathering process for a specific data source, incremental and archiving mechanisms. Client complexities to query the data source (e.g., pagination, OAuth access) handles connection problems. CommandLine set up parameters to execute a backend. Optional arguments such as help and debug.
- 13. /perceval As a program $ pip3 install perceval $ perceval github chaoss grimoirelab-perceval --from-date=2017-03-01 --api-token=5… [2018-08-01 12:31:10] - Sir Perceval is on his quest. [2018-08-01 12:31:11] - Getting info for https://api.github.com/users/jgbarah [2018-08-01 12:31:12] - Getting info for https://api.github.com/users/sduenas … producing JSON documents … [2018-08-01 12:34:22] - Sir Perceval completed his quest
- 14. /perceval As a Python3 library from perceval.backends.core.github import GitHub from_date = datetime.datetime(2017, 3, 1) github = GitHub("chaoss", "grimoirelab-perceval", api_token="5e3d7...") for issue in github.fetch(from_date=from_date): print(issue['data'])
- 15. /perceval { "backend_name": "GitHub", "backend_version": "0.17.2", "category": "issue", "data": { "comments": 5, "comments_data": {...} "created_at": "2017-02-28T05:33:10Z", ...., "id": 210691361, "state": "closed", "title": "..", "updated_at": "2017-03-02T09:51:49Z" }, "origin": "https://github.com/chaoss/grimoirelab-perceval", "perceval_version": "0.11.7", "tag": "https://github.com/chaoss/grimoirelab-perceval", "timestamp": 1517479766.878609, "updated_on": 1488448309.0, "uuid": "77a42463d5d0a34b1d58006263b85909a9788b52" } Data source data } ]Perceval data
- 16. /graal No code-based analysis with Perceval: - Code complexity - License evolution - ...
- 17. /graal Graal does it! It leverages on the incremental functionalities of Perceval and enhances the logic to handle Git repositories to process source code valeriocos/graal
- 18. /graal
- 19. /graal Filter of commits based on Git JSON documents. For selected commits, checkouts it on the working tree. Analysis executes analysis tools in the working tree . Results of the analysis are automatically embedded in the JSON document. Post-process alters the attributes of inflated JSON document, thus granting the user complete control over the output.
- 20. /graal Code Complexity Dependencies Quality checks Security Backends
- 21. /graal As a program $ pip install graal $ sudo apt-install cloc $ graal cocom https://github.com/chaoss/grimoirelab-perceval --git-path /tmp/graal-cocom [2018-05-30 18:22:35,643] - Starting the quest for the Graal. [2018-05-30 18:22:39,958] - Git worktree /tmp/... created! [2018-05-30 18:22:39,959] - Fetching commits … producing JSON documents … [2018-05-31 04:51:56,112] - Quest completed.
- 22. /graal As a Python3 library from graal.backends.core.cocom import CoCom repo_uri = "https://github.com/chaoss/grimoirelab-perceval" repo_dir = "/tmp/graal-cocom" cc = CoCom(uri=repo_uri, gitpath=repo_dir) commits = [commit for commit in cc.fetch()]
- 23. /graal { "backend_name": "CoCom", "backend_version": "0.2.1", "category": “code_complexity", "data": { "AuthorDate": “Mon May 28 ...”, "CommitDate": “Mon May 28 ...”, "commit": "dc78c25….", "message": "Increase coverage ...", "analysis": [{ "ccn": 80, “num_funs”: 33, …, "comments": "153", “loc”: 341, "file_path": "perceval/backend.py", }], ... }, "origin": "https://github.com/chaoss/grimoirelab-perceval", "graal_version": "0.1.0", "tag": "https://github.com/chaoss/grimoirelab-perceval", "timestamp": 1534782753.878609, "updated_on": 1392000436.0, "uuid": "77a42463d5d0a34b1d58006263b85909a9788b52" } Git metadata } ]Graal data } Code data
- 24. /arthur Arthur allows to schedule and run Perceval (and Graal) executions at scale through distributed Redis queues. chaoss/grimoirelab-kingarthur
- 25. /explotation
- 26. /explotation
- 27. /explotation