Scribd
Upload
33 views

Lecture 18 (Aws Kinesis Configuration)

This document provides steps to configure the AWS Kinesis agent on an EC2 instance to stream log files to a Kinesis data stream. It involves installing the Kinesis agent, configuring the agent.json and sysconfig files with IAM credentials and stream details, starting the agent service, and verifying logs are sent by checking the agent log file. It also mentions creating the Kinesis stream and delivery stream.

Uploaded by

vishuredhu1234512345
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
33 views

Lecture 18 (Aws Kinesis Configuration)

This document provides steps to configure the AWS Kinesis agent on an EC2 instance to stream log files to a Kinesis data stream. It involves installing the Kinesis agent, configuring the agent.json and sysconfig files with IAM credentials and stream details, starting the agent service, and verifying logs are sent by checking the agent log file. It also mentions creating the Kinesis stream and delivery stream.

Uploaded by

vishuredhu1234512345
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 7

AWS kinesis configuration: ----

1. Launch EC2 with a programmatic access of the following policies - AmazonS3FullAccess,


AmazonKinesisFullAccess, AdministratorAccess, AmazonEC2FullAccess

2. Download the access key and .pem file if you don't have.

Once EC2 is up using sudo access - install Kinesis agent on the machine

sudo yum install –y aws-kinesis-agent

Writing to Amazon Kinesis Data Streams Using Kinesis Agent - Amazon Kinesis Data Streams

STEP1:-- INSTALL AWS-KINESIS-AGENT ON EC2-INSTANCE AND DO


SOME CONFIGURATION:--
2 sudo yum install –y aws-kinesis-agent

sudo yum install https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-


latest.amzn2.noarch.rpm –y

or

sudo yum install https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-


latest.amzn2.noarch.rpm

3 vim /etc/aws-kinesis/agent.json

"cloudwatch.emitMetrics": true,

"kinesis.endpoint": "kinesis.us-east-1.amazonaws.com",

"firehose.endpoint": "firehose.us-east-1.amazonaws.com",
"awsAccessKeyId": "AKIA5QYYHX2KNYAL2UOO",

"awsSecretAccessKey": "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",

"flows": [

"filePattern": "/tmp/*.log",

"kinesisStream": "kds-demo",

"partitionKeyOption": "RANDOM"

4 vim /etc/sysconfig/aws-kinesis-agent

# Set AWS credentials for accessing Amazon Kinesis Stream and Amazon Kinesis Firehose

# AWS_ACCESS_KEY_ID="AKIA5QYYHX2KNYAL2UOO"

# AWS_SECRET_ACCESS_KEY="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"

# AWS_DEFAULT_REGION="us-east-1"

# AGENT_ARGS=""

# AGENT_LOG_LEVEL="INFO"

5 sudo service aws-kinesis-agent start

6 sudo service aws-kinesis-agent restart

9 sudo service aws-kinesis-agent restart

10 exit
STEP2:--NOW CREATE KINESIS STREAM:--
STEP 3:--NOW CREATE KINESIS DELIVERY STREAM:--
2 ls -lrt
3 vi temp.log

4 head /var/log/messages

5 sudo head /var/log/messages

6 vi temp1.log

7 cd /tmp

8 ls -lrt

9 cd

10 ls

11 cp temp1.log /tmp/temp1.log

12 cp temp1.log /tmp/temp2.log

13 cp temp1.log /tmp/temp3.log

14 cp temp1.log /tmp/temp4.log

15 cp temp1.log /tmp/temp5.log

16 tail -40 /var/log/aws-kinesis-agent/aws-kinesis-agent.log

17 history

STEP 4:--NOW GO TO S3 BUCKET FOR LOG FILE OF KINESIS:--

You might also like