Compound ZIP parser plugin for Google Takeout #2296

stefanomart · 2019-01-17T13:43:11Z

One line description of pull request

This is a PR about the Compound ZIP parser plugin for Google Takeout dump files.

Description:

This plugin parses the Google Takeout dump.
It parses these services:

GMail
Google Maps (location history)
Google Chrome History
My Activity
Hangouts
Purchases

I made a slight change to the czip logic: I added a method CheckZipFile to czip_plugins\interface.py (this can be overrided by every czip plugin) to check the plugin: it checks if the zip file contains a file 'index.html' and the existence of the keyword 'Google Takeout' or 'Google Takeaway' in the same file.

Like we discussed here, I added the logic of mailbox into plaso.

I hope that all the code styling rules are respected.

Notes:

Checklist:

Automated checks (Travis, Codecov, Codefactor )pass
No new new dependencies are required or l2tdevtools has been updated
Reviewer assigned

Onager

Hey @stefanomart - I'm guessing this is still a work in progress. Please request a review from me once the test are passing - thanks!

codecov · 2019-01-23T16:47:44Z

Codecov Report

Merging #2296 into master will decrease coverage by 1.08%.
The diff coverage is 25.9%.

@@            Coverage Diff            @@
##           master   #2296      +/-   ##
=========================================
- Coverage   85.88%   84.8%   -1.09%     
=========================================
  Files         455     454       -1     
  Lines       32071   32745     +674     
=========================================
+ Hits        27544   27769     +225     
- Misses       4527    4976     +449

Impacted Files	Coverage Δ
plaso/parsers/czip_plugins/oxml.py	`92.06% <ø> (ø)`	⬆️
plaso/formatters/google_takeout.py	`100% <100%> (ø)`
plaso/parsers/czip_plugins/interface.py	`100% <100%> (+11.76%)`	⬆️
plaso/lib/definitions.py	`100% <100%> (ø)`	⬆️
plaso/parsers/czip_plugins/google_takeout.py	`19.3% <19.3%> (ø)`
plaso/parsers/presets.py	`85.91% <0%> (-14.09%)`	⬇️
plaso/filters/interface.py	`73.91% <0%> (-6.74%)`	⬇️
plaso/containers/event_sources.py	`94.11% <0%> (-5.89%)`	⬇️
plaso/parsers/text_parser.py	`81.4% <0%> (-4.62%)`	⬇️
plaso/engine/tagging_file.py	`94.44% <0%> (-3.6%)`	⬇️
... and 145 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bf94d93...13c2499. Read the comment docs.

stefanomart · 2019-01-27T10:06:52Z

Hey @stefanomart - I'm guessing this is still a work in progress. Please request a review from me once the test are passing - thanks!

Hi @Onager - I noticed that CodeFactor doesn't check the updated code, but always check the initial code (10 days ago).

Onager · 2019-01-27T21:30:31Z

OK @stefanomart - pylint is passing in Travis, so I'll take a look.

Onager · 2019-01-27T21:33:12Z

Hey @stefanomart - there's no tests in this PR, could you please add a test for parser?

If you need test data, I noticed there's a public data set here: https://aboutdfir.com/dfir-research/ in the Android nougat image.

stefanomart · 2019-02-16T13:17:18Z

Hello again @Onager, I noticed that Travis could not complete due to an error: "Unable to locate package python3-tox". I also did not understand why CodeFactor still checks old code (from 17 Jan).
However I added a test-data.
Thanks.

Onager

@stefanomart - checking in on this, it looks like there's still no tests in this PR, just the test data. Are you still working on this?

As to the tox issues, these should be resolved now if you rebase or merge to HEAD. There's currently an issue with dfwinreg, but that's being fixed.

stefanomart · 2019-04-02T13:58:44Z

Even if I change the code, CodeFactor keeps to analyze the code uploaded on January 17th.
Am I doing something wrong?

Onager · 2019-04-08T11:02:21Z

Hey @stefanomart - don't worry too much about codefactor in this case, the extra pylint check will do most of the checks we'll need to work about.

Stefano Martinelli and others added 7 commits Jan 16, 2019

Add Google Takeout parser/formatter

5d3a881

Add Google Takeout to init

368c00d

Code updated

f3cb9b5

Delete __init__.pyc

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

99f9b1a

Delete google_takeout.pyc

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

3373568

Delete interface.pyc

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

d23a7fc

Delete oxml.pyc

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

Loading status checks…

17afc13

Onager self-requested a review Jan 17, 2019

Onager self-assigned this Jan 17, 2019

Stefano Martinelli added 3 commits Jan 17, 2019

Fixed some errors

b91ecfc

Update

Loading status checks…

c0e52fa

Fixed CodeFactor issues

Loading status checks…

2c811a9

Onager requested changes Jan 21, 2019

View changes

Stefano Martinelli added 2 commits Jan 23, 2019

Add check for a file

Loading status checks…

8c667f6

Add check for a file

Loading status checks…

aaa6515

Stefano Martinelli added 5 commits Jan 24, 2019

Comment-out import hachoir

Loading status checks…

670b1a6

Remove import no more used (archoir)

26cedef

Remove if-else blocks

Loading status checks…

a899ec9

Remove pylint warning

Loading status checks…

a12e237

Remove some definitions and remove some if-else blocks

Loading status checks…

96a1dd9

Onager self-requested a review Jan 27, 2019

Stefano Martinelli added 5 commits Jan 27, 2019

Add test data

Loading status checks…

974505b

Remove some functions

Loading status checks…

d31d98e

Change some textual pylint and update if-blocks

Loading status checks…

421d071

Remove some pylint warning

Loading status checks…

82f456b

Rename variables

Loading status checks…

74afa89

Remove print

Loading status checks…

e0049ae

Onager requested changes Mar 11, 2019

View changes

Stefano Martinelli and others added 4 commits Mar 20, 2019

Add tests

Loading status checks…

db6f49c

Removed Pylint message

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

Loading status checks…

79bacf8

Fixed Pylint message

ed51d52

Merge branch 'master' of https://github.com/stefanomart/plaso

Loading status checks…

13c2499

Onager self-requested a review Apr 8, 2019

joachimmetz force-pushed the log2timeline:master branch from 8213f1c to 5157939 Jun 28, 2019

log2timeline / plaso

Compound ZIP parser plugin for Google Takeout #2296

Compound ZIP parser plugin for Google Takeout #2296

stefanomart commented Jan 17, 2019 •

edited

Onager left a comment

codecov bot commented Jan 23, 2019 •

edited

stefanomart commented Jan 27, 2019 •

edited

Onager commented Jan 27, 2019

Onager commented Jan 27, 2019

stefanomart commented Feb 16, 2019 •

edited

Onager left a comment

stefanomart commented Apr 2, 2019

Onager commented Apr 8, 2019

log2timeline / plaso

Compound ZIP parser plugin for Google Takeout #2296

Compound ZIP parser plugin for Google Takeout #2296

Conversation

stefanomart commented Jan 17, 2019 • edited

One line description of pull request

Description:

Notes:

Checklist:

Onager left a comment

codecov bot commented Jan 23, 2019 • edited

Codecov Report

stefanomart commented Jan 27, 2019 • edited

Onager commented Jan 27, 2019

Onager commented Jan 27, 2019

stefanomart commented Feb 16, 2019 • edited

Onager left a comment

stefanomart commented Apr 2, 2019

Onager commented Apr 8, 2019

stefanomart commented Jan 17, 2019 •

edited

codecov bot commented Jan 23, 2019 •

edited

stefanomart commented Jan 27, 2019 •

edited

stefanomart commented Feb 16, 2019 •

edited