Programming

pipe2firehose

I was in need of a way to feed large amounts JSON of data into Kinesis Firehose, so I could work out a bunch of other downstream logic. While the AWS CLI does allow me to push data, I have to call out each line I want to write as arguments to the command, and it has issues with some characters, meaning the data in didn’t always match the data out.

Though this seems like a pretty generic need, I couldn’t find any portable tools out there to push data from a file or pipe from stdin/stdout to a Firehose, so I decided to write my own, called pipe2firehose.

To install, download from Github release page or run:

1
go get github.com/Brayyy/pipe2firehose

https://github.com/Brayyy/pipe2firehose
A simple tool to pipe data from stdin and push into AWS Kinesis Firehose or Kinesis Stream.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Usage:
  cat data.json | pipe2firehose [options] [firehose-name]
or
  pipe2firehose [options] [firehose-name] < data.json

Options:
  -batch-size int
    	Modify the number of records included per PutRecordBatch (default 500)
  -delay string
    	Time in ms to wait between each line read
  -region string
    	AWS Region of Firehose
  -version
    	Print the version

This is far superior for my use case compared to the AWS CLI method, requiring a lot of string replacing to even get the records to be postable:

1
2
3
4
5
# Not really sustainable when you need to push 500,000 lines
aws firehose put-record-batch \
  --delivery-stream-name my-firehose \
  --region us-east-1 \
  --records '\{"test":"line1"\}' '\{"test":"line2"\}' '\{"test":"line3"\}'