Enable Amazon Kinesis

Logpush supports Amazon Kinesis ↗ as a destination for all datasets. Each Kinesis record that Logpush sends will contain a batch of GZIP-compressed data in newline-delimited JSON format (by default), or in the format specified in the output_options parameter when the job was created.

Configure Kinesis using STS Assume Role (recommended)

1. Create an IAM Role for Cloudflare Logpush to Assume with the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::391854517948:user/cloudflare-logpush"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

2. Ensure that the IAM role has permissions to perform the PutRecord action on your Kinesis stream. Replace <AWS_REGION>, <YOUR_AWS_ACCOUNT_ID> and <STREAM_NAME> with your own values:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kinesis:PutRecord",
            "Resource": "arn:aws:kinesis:<AWS_REGION>:<YOUR_AWS_ACCOUNT_ID>:stream/<STREAM_NAME>"
        }
    ]
}

3. Create a Logpush job, using the following format for the destination_conf field:

kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>

4. (optional) When using STS Assume Role, you can include sts-external-id as a destination_conf parameter so it is included in your Logpush job's requests to Kinesis. Refer to Securely Using External ID for Accessing AWS Accounts Owned by Others ↗ for more information.

kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>&sts-external-id=<EXTERNAL_ID>

STS Assume Role example

$ curl https://api.cloudflare.com/client/v4/zones/$ZONE_TAG/logpush/jobs \
-H 'Authorization: Bearer <API_TOKEN>' \
-H 'Content-Type: application/json' -d '{
  "name": "kinesis",
  "destination_conf": "kinesis://<STEAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<IAM_ROLE_NAME>",
  "dataset": "http_requests",
  "enabled": true
}'

Configure Kinesis using IAM Access Keys

When configuring your Logpush job using IAM Access Keys, ensure that the IAM user has permission to perform the PutRecord action on your Kinesis stream:

kinesis://<STREAM_NAME>?region=<AWS_REGION>&access-key-id=<AWS_ACCESS_KEY_ID>&secret-access-key=<AWS_SECRET_ACCESS_KEY>

IAM Access Key example

$ curl https://api.cloudflare.com/client/v4/zones/$ZONE_TAG/logpush/jobs \
-H 'Authorization: Bearer <API_TOKEN>' \
-H 'Content-Type: application/json' -d '{
  "name": "kinesis",
  "destination_conf": "kinesis://<STREAM_NAME>?region=<AWS_REGION>&access-key-id=<AWS_ACCESS_KEY_ID>&secret-access-key=<AWS_SECRET_ACCESS_KEY>",
  "dataset": "http_requests",
  "enabled": true
}'