Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BR should provide the total size of one backup #95

Open
DanielZhangQD opened this issue Dec 9, 2019 · 9 comments
Open

BR should provide the total size of one backup #95

DanielZhangQD opened this issue Dec 9, 2019 · 9 comments

Comments

@DanielZhangQD
Copy link
Contributor

@DanielZhangQD DanielZhangQD commented Dec 9, 2019

BR should provide the total size of one backup considering following cases:

  • Users want to know the size of each backup.
  • The platforms that integrate with BR, e.g. TEP, DBaaS, etc. need to show the size of each backup.

Maybe TiKV can return the size of the files it created for each backup to BR and BR calculates the total size from all the TiKVs and writes the size to backup meta.

@kennytm @overvenus @tennix

@kennytm
Copy link
Member

@kennytm kennytm commented Dec 9, 2019

Include this as part of checksum calculation?

@DanielZhangQD
Copy link
Contributor Author

@DanielZhangQD DanielZhangQD commented Dec 9, 2019

You mean platforms integrating with BR want to include this as part of checksum? Not necessarily.
They may just need to know the total size of each backup.

@overvenus
Copy link
Member

@overvenus overvenus commented Dec 11, 2019

BR should provide the total size of one backup considering following cases:
...

That's a good feature request! And it would be easy to implement. What we need to do is,

  1. Add a new field uint64 size = 10 to backup.File https://github.com/pingcap/kvproto/blob/5dbce7e7b868a1a19be3a368f9ea6db120ea8926/proto/backup.proto#L42-L54
  2. Adjust TiKV backup writer, attach file size to the File. https://github.com/tikv/tikv/blob/134c5663340c8e868f23bc2f3273b84168aed862/components/backup/src/writer.rs#L60-L86
  3. Sum up all files at BR.

Include this as part of checksum calculation?

The total_bytes in checksum calculation is a logical size, and the actual file content is compressed, so it does not match the file size.

@fredchenbj
Copy link

@fredchenbj fredchenbj commented Dec 12, 2019

I would like to try it ~ If I have some questions, who can I ask for help ? @overvenus

@overvenus
Copy link
Member

@overvenus overvenus commented Dec 16, 2019

That's so cool! You can always ask me! @fredchenbj

@fredchenbj
Copy link

@fredchenbj fredchenbj commented Dec 16, 2019

ok, this week I would do some work about this.

@kennytm
Copy link
Member

@kennytm kennytm commented May 5, 2020

The total size is already computed in #249. It just needs to printed somewhere to close this issue.

@liangyuanpeng
Copy link

@liangyuanpeng liangyuanpeng commented Aug 23, 2020

The total size is already computed in #249. It just needs to printed somewhere to close this issue.

In my test , it's already print total size.

[2020/08/23 18:46:31.324 +08:00] [INFO] [collector.go:61] ["Full backup Success summary: total backup ranges: 2, total success: 2, total failed: 0, total take(s): 0.01, total kv: 2, total size(Byte): 66, avg speed(Byte/s): 10990.76"] ["backup checksum"=1.485799ms] ["backup fast checksum"=335.652µs] ["backup total regions"=2] [BackupTS=418826134061580290] [Size=4486]

total size(Byte): 66, .

@kennytm
Copy link
Member

@kennytm kennytm commented Aug 23, 2020

@liangyuanpeng that "total size" is the total uncompressed KV size, not the total (compressed) archive size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.