datapipeline 1
๋ฐ์ดํ„ฐ ๋ถ„์„ ํ™˜๊ฒฝ ๊ตฌ์ถ• - 05. Clickhouse ์‹ค์‹œ๊ฐ„ ๋ฐ์ดํ„ฐ ๋ถ„์„ ํ…Œ์ด๋ธ” ์ƒ์„ฑํ•˜๊ธฐ

๐Ÿš€ ์ด ๊ธ€์—์„œ ๋‹ค๋ฃฐ ๋‚ด์šฉ1๏ธโƒฃ Kafka์™€ ์—ฐ๊ฒฐ๋œ ClickHouse ํ…Œ์ด๋ธ” ์ƒ์„ฑ (ํ™˜๊ฒฝ์„ค์ • ํฌํ•จ)2๏ธโƒฃ ์‹ค์Šต์šฉ ๋ฐ์ดํ„ฐ์…‹ kafka produce3๏ธโƒฃ Kafka ํ…Œ์ด๋ธ”์—์„œ ๋ฐ์ดํ„ฐ ์…‹ ํ™•์ธ ๋ฐฉ๋ฒ•1. Kafka์™€ ์—ฐ๊ฒฐ๋œ ClickHouse ํ…Œ์ด๋ธ” ์ƒ์„ฑ๐Ÿ”น Kafka ๋ฐ์ดํ„ฐ๋ฅผ ClickHouse์—์„œ ์ฒ˜๋ฆฌํ•˜๋Š” ๋ฐฉ์‹ClickHouse๋Š” Kafka์™€ ์ง์ ‘ ์—ฐ๊ฒฐํ•˜์—ฌ ์‹ค์‹œ๊ฐ„์œผ๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ๋Š” Kafka ์—”์ง„์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.๊ธฐ๋ณธ์ ์œผ๋กœ Kafka → Buffer ํ…Œ์ด๋ธ” → MergeTree ํ…Œ์ด๋ธ”์˜ ๊ตฌ์กฐ๋กœ ์šด์˜๋ฉ๋‹ˆ๋‹ค.๐Ÿ”น ClickHouse ์„ค์ • ๋ณ€๊ฒฝ (Kafka ์‚ฌ์šฉ์„ ์œ„ํ•œ ์„ค์ •)๋จผ์ €, clickhouse-server์˜ ํ™˜๊ฒฝ์„ค์ •์„ ์ˆ˜์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.ClickHouse ์„ค์ • ํŒŒ์ผ (config.xml)์—์„œ Ka..
๋ฐ์ดํ„ฐ ์—”์ง€๋‹ˆ์–ด๋ง 2025.03.30
1
๋”๋ณด๊ธฐ

ํ‹ฐ์Šคํ† ๋ฆฌํˆด๋ฐ”