Amazon Kinesis Data Firehose allows you to compress your data before delivering it to Amazon S3. The service currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Amazon Redshift.
Q: What compression format can I use? Amazon Kinesis Data Firehose allows you to compress your data before delivering it to Amazon S3. The service currently supports GZIP, ZIP, and SNAPPY compression formats.
Choosing The Json Deserializer
Choose the OpenX JSON SerDeif your input JSON contains time stamps in the following formats: The OpenX JSON SerDe can convert periods (.) to underscores (_).
It can also convert JSON keys to lowercase before deserializing them.
For more information about the options that are available with this deserializer through Kinesis Data Firehose, see OpenXJ.
,
Choosing The Serializer
The serializer that you choose depends on your business needs.
To learn more about the two serializer options, see ORC SerDe and Parquet SerDe.
,
Converting Input Record Format
You can enable data format conversion on the console when you create or update a Kinesis delivery stream.
With data format conversion enabled, Amazon S3 is the only destination that you can configure for the delivery stream.
Also, Amazon S3 compression gets disabled when you enable format conversion.
However, Snappy compression happens automaticall.
,
Record Format Conversion Error Handling
When Kinesis Data Firehose can't parse or deserialize a record (for example, when the data doesn't match the schema), it writes it to Amazon S3 with an error prefix.
If this write fails, Kinesis Data Firehose retries it forever, blocking further delivery.
For each failed record, Kinesis Data Firehose writes a JSON document with the following schema.
,
Record Format Conversion Requirements
Kinesis Data Firehose requires the following three elements to convert the format of your record data: You can convert the format of your data even if you aggregate your records before sending them to Kinesis Data Firehose.
Record Format Conversion Requirements
Kinesis Data Firehose requires the following three elements to convert the format of your record data: You can convert the format of your data even if you Choosing The Json Deserializer
Choose the OpenX JSON SerDeif your input JSON contains time stamps in the following formats: The OpenX JSON SerDe can convert periods ( Choosing The Serializer
The serializer that you choose depends on your business needs. To learn more about the two serializer options, see ORC SerDe and Parquet SerDe Converting Input Record Format
You can enable data format conversion on the console when you create or update a Kinesis delivery stream. With data format conversion enabled Record Format Conversion Error Handling
When Kinesis Data Firehose can't parse or deserialize a record (for example, when the data doesn't match the schema), it writes it to Amazon S3 with an error prefix Record Format Conversion Example
For an example of how to set up record format conversion with AWS CloudFormation Q: What compression format can I use? Amazon Kinesis Data Firehose allows you to compress your data before delivering it to Amazon S3. The service currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Amazon Redshift.
