EloqData snapshot and branching feature MVP proposal

[githubzilla]

The MVP of the snapshot and branching feature

This is the first step to implement the full-fledged snapshot and branching feature comparable to Neon.

What Neon has for its snapshot and branching feature?

• Branching from Current Data
• Branching from Past data (PIT enable)
• Schema only branching
• Anonymized data branching
• Branching with TTL
• Restore from history (PIT enable)
• Restore from another branch (PIT enable)
• Time travel assist

What this MVP will include?

• Take snapshot or branching from Current Data of the main branch
• Delete snapshot or branching
• Purging the obsoleted data files
• For RocksDB-Cloud DSS only

Interface

API name	RPC host	Input parameters	Output parameters
CreateBackup	DB instance	(when take snapshot) 1. backup_name = {branchname}-snapshot	1. BackupStatus
		(when branching) 1. backup_name = {new_branch_name}
		2. ng_id
FetchBackup	DB instance	1. backup_name	1. backup_files (include snapshot_cookie or branch_cookie)
			2. backup_ts
			2. BackupStatus
RunPurger	Serverless	1. bucket_prefix	1. success
		2. bucket_name	2. errors
DeleteSnapshot	CP	1. bucket_prefix	1. success
		2. bucket_name	2. error
		3. snapshot_path
DeleteBranching	CP	1. bucket_prefix	1. success
		2. bucket_name	2. error
		3. branching_path
RestoreFromSnapshot	CP	1. snapshot_path	1. success
		2. new_branching_path	2. error
RestoreFromBranching	CP	1. new_branching_path	1. success
			2. error

• DSS configuration parameters when Start DB
  • bucket_prefix
  • bucket_name
  • object_path
  • cloudmanifest_postfix (pass branching_name in this case)

Design

• The snapshot and branching are same thing from the perspective of DSS

  • It is about recording the set of files consist of the data store at the moment
  • In context of RocksDB-Cloud, it is about rolling an new CLOUDMANIFEST file from the original CLOUDMANIFEST file

• The naming pattern for CLOUDMANIFEST file

  • Branching
    • CLOUDMANIFEST-${branch_name}-${UUID}-${dss_shard_id}-${increment_number}
    • The prebuilt branches are "production" and "development"
    • The default branch is "production"
    • The increment_number is purpose to prevent overwrites between DB nodes involved in fail over scenario
  • Snapshot
    • CLOUDMANIFEST-${branch_name}-snapshot-${UUID}-${dss_shard}-snapshot-${ckpt_ts}

• Create a snapshot or branching

  • Stop regular checkpoint
  • Trigger an exclusive checkpoint and wait util finish
  • Rolling a new CLOUDMANIFEST
  • Open regular checkpoint
  • For inactive DB instance, we take a snapshot before it becomes inactive.

• Delete a snapshot or branching

  • Delete the corresponding CLOUDMANIFEST file
  • Purger will clean the orphan files at next run

• Restore from snapshot or branching (CP works)

  • Stop DB instance
  • Start DB instance on snapshot or inactive branching
  • Create an new branch from snapshot or branch
  • Start DB instance with the new branch

Future works

• Impelment PIT enabled snapshot and branching with the combination of regular snapshot and redo logs
• TTL

[liunyl]

1. DeleteSnapshot/Braching is a direct s3 API call and can be done by CP directly.
2. Is there anything special to do when restoring from a snapshot? Isn't it just a regular open with a different cloud manifest postfix passed in?
3. We should use DSS shard id instead of ng_id in manifest name.
4. Does rocksdb support open multiple db instance on the same process? If not we need to implement our own "create new CloudManifest". If we can know how to "create new CloudManifest" and make it a CP function call, it will make a lot of things easier and avoid many unnecessary db start

[githubzilla]

Sync with @liunyl @zhangh43 ,

1. For inactive instance, we may need to take snapshot before inactive.
2. Use DSS shard id instead of ng_id in manifest name.
3. Considering calling RocksDB-Cloud API from CP, so CP can avoid starting DB instance when branching from snapshot or inactive branching.
4. DeleteSnapshot/Braching and Restore are totally CP works.

[weidaolee]

https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html

[liunyl]

1. branch name format in CreateBackUpRequest
2. Filter out s3 invalid characters