Skip to content

/ carbondata

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation #4072

Open
wants to merge 2 commits into
base: master
from
Open

[CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation #4072

wants to merge 2 commits into from
+391 −62

Conversation

@vikramahuja1001
Copy link
Contributor

@vikramahuja1001 vikramahuja1001 commented Jan 7, 2021 •
edited

Why is this PR needed?

Currently in the clean files operation the user does not know how much space will be freed. The idea is the add support for dry run in clean files which can tell the user how much space will be freed in the clean files operation without cleaning the actual data.

What changes were proposed in this PR?

This PR has the following changes:

  1. Support dry run in clean files: It will show the user how much space will be freed by the clean files operation and how much space left (which can be released after expiration time) after the clean files operation.
  2. Clean files output:
    a. Total size released(expired segments deleted outside trash and inside trash)
    b. Remaining size which can be released after expiration time.
  3. Clean files log: To enhance the clean files log to print the name of every file that is being deleted in the info log.

Does this PR introduce any user interface change?

  • No
  • Yes. (please explain the change and update document)

Is any new testcase added?

  • No
  • Yes
@vikramahuja1001
Added logs while deleting anything during clean files
5df3423
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 7, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3529/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 7, 2021

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5289/
@QiangCai
Copy link
Contributor

@QiangCai QiangCai commented Jan 12, 2021

please perfect the PR title
@vikramahuja1001
Copy link
Contributor Author

@vikramahuja1001 vikramahuja1001 commented Jan 12, 2021

It's not yet finished, i will change the title and add jira after completing the code. Please review after that
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 12, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3539/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 12, 2021

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5299/
@vikramahuja1001 vikramahuja1001 force-pushed the vikramahuja1001:cleanFilesPhase2 branch from 85aac5e to e57c162 Jan 12, 2021
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 12, 2021

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5300/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 12, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3540/
@vikramahuja1001 vikramahuja1001 force-pushed the vikramahuja1001:cleanFilesPhase2 branch from e57c162 to 75457c4 Jan 25, 2021
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 25, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3579/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 25, 2021

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5339/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 25, 2021

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5052/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 25, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3294/
@vikramahuja1001
Copy link
Contributor Author

@vikramahuja1001 vikramahuja1001 commented Jan 25, 2021

retest this please
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 25, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3308/
@vikramahuja1001
Copy link
Contributor Author

@vikramahuja1001 vikramahuja1001 commented Jan 25, 2021

retest this please
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 25, 2021

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5343/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 25, 2021

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3582/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 25, 2021

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3585/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 25, 2021

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5346/
@vikramahuja1001 vikramahuja1001 changed the title [WIP] Clean files phase2 [WIP] Support clean files dry run operation and show statistics after clean files operation Jan 25, 2021
@vikramahuja1001 vikramahuja1001 changed the title [WIP] Support clean files dry run operation and show statistics after clean files operation [CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation Jan 25, 2021
@vikramahuja1001 vikramahuja1001 force-pushed the vikramahuja1001:cleanFilesPhase2 branch from 75457c4 to 9af03c7 Jan 27, 2021
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 27, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3597/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 27, 2021

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5357/
@vikramahuja1001
Copy link
Contributor Author

@vikramahuja1001 vikramahuja1001 commented Jan 27, 2021

retest this please
@vikramahuja1001
Support dry run
cfa94df
@vikramahuja1001 vikramahuja1001 force-pushed the vikramahuja1001:cleanFilesPhase2 branch from 9af03c7 to cfa94df Jan 27, 2021
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 27, 2021

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5360/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 27, 2021

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3600/
@ydvpankaj99
Copy link
Contributor

@ydvpankaj99 ydvpankaj99 commented Jan 28, 2021

retest this please
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 28, 2021

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5370/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Jan 28, 2021

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3609/
@vikramahuja1001
Copy link
Contributor Author

@vikramahuja1001 vikramahuja1001 commented Feb 3, 2021

@QiangCai @ajantha-bhat @akashrn5 please review
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Feb 11, 2021

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3317/
@CarbonDataQA2
Copy link

@CarbonDataQA2 CarbonDataQA2 commented Feb 11, 2021

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5075/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
@vikramahuja1001 @CarbonDataQA2 @QiangCai @ydvpankaj99