Skip to content

Instantly share code, notes, and snippets.

@javisantana
Last active February 8, 2021 13:28
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save javisantana/de40a5d9bf9ea43b3d6a176f6638984a to your computer and use it in GitHub Desktop.
Save javisantana/de40a5d9bf9ea43b3d6a176f6638984a to your computer and use it in GitHub Desktop.

Sometimes you have to move data from one table to a different one. You usually use

insert into target select * from source

This works but have several problems:

  1. materialized columns are not properly copied
  2. it's slow
  3. the new table needs to be merged again

Clickhouse has a way to do it really fast: copying partitions. Unfortunately there is no command to copy all the partitions from one table to another but it's pretty easy to run a SQL to generate an script to do that:

SELECT concat('ALTER TABLE target REPLACE PARTITION ID \'', partition_id, '\' FROM source;')
FROM system.parts
WHERE (table = 'source') AND active
INTO OUTFILE '/tmp/move_data.sql'
FORMAT TabSeparatedRaw

Then

cat /tmp/move_data.sql | clickhouse-client -mn

You have to be careful because if something changes in between you generate and run the script, the data will be lost in the new table. To avoid that you could setup a materialized view from source to target.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment