Skip to main content

How can I convert topic data into a CSV file?

  • Updated

Occasionally users want to see data that's in a topic in a familiar tool like Excel, Google Sheets, or any tool compatible with the CSV format. A quick way to convert data from a topic into a CSV file is with the following bash script, which makes use of rpk:

#!/bin/bash

# Configuration
TOPIC="name-of-topic"
CSV_FILE="exported_data.csv"

# Write CSV Header
echo "Timestamp,Key,Value" > "$CSV_FILE"

# Consume messages and append to CSV
rpk topic consume log1 -o :end | jq -sr '
  ["topic","key","value","timestamp","partition","offset"],
  (.[] | [.topic, .key, .value, .timestamp, .partition, .offset])
  | @csv' >> "$CSV_FILE"

echo "Export complete. Data saved to $CSV_FILE"

This script creates 6 columns: "topic","key","value","timestamp","partition", and "offset". Feel free to change which of these columns are created based on what you need out of the topic data.