Scribd
Upload
3 views

Kinesis Data Stream

The document outlines the steps to configure Kinesis Data Stream, including creating user roles, configuring EC2 instances, and installing the Kinesis agent. It provides detailed commands for logging into the EC2 instance, installing the agent, and configuring the data stream with JSON settings. Additionally, it includes instructions for copying data to the EC2 instance and managing permissions for the weather directory.

Uploaded by

Pushpjeet Cholkar
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
3 views

Kinesis Data Stream

The document outlines the steps to configure Kinesis Data Stream, including creating user roles, configuring EC2 instances, and installing the Kinesis agent. It provides detailed commands for logging into the EC2 instance, installing the agent, and configuring the data stream with JSON settings. Additionally, it includes instructions for copying data to the EC2 instance and managing permissions for the weather directory.

Uploaded by

Pushpjeet Cholkar
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

Kinesis Data Stream

—Manish

Real time streaming flow


Steps to configure Kinesis Data Stream

1- Create User role and give access to Data Stream , S3 and EC2.
2- Configure EC2 Instance.
3- Create Data Stream.
4-Download data test data from my GitHub repository.
5- Login to EC2 instance from your local terminal.
6- install Agent in the EC2 instance.
7- configure stream data file in EC2.
Login to EC2 instance.

1. Change the permission of .pem file.

Eg . Chmod 400 File name

2. ssh -i "Kinesis_stream.pem" [email protected]

3. Elevate your permission to root.

Sudo su
Install the Kinesis agent

Latest version.

sudo yum install –y https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-


latest.amzn1.noarch.rpm

Install Specific version:

sudo yum install –y https://streaming-data-agent.s3.amazonaws.com/aws-kinesis-


agent-1.1.4-1.amzn1.noarch.rpm
Configure Data stream json

Go to the logs inside /var and create the folder called weather. mkdir weather

{
"cloudwatch.emitMetrics": true,
"kinesis.endpoint": "",
"firehose.endpoint": “”,

"awsAccessKeyId": “AKIATPF6UVKKPZJGNHHD”,
"awsSecretAccessKey": “TBidJNOGxTqfLtUhDn5O5ysnFJMhphtDFsYPEDUq",

"flows": [
{
"filePattern": “/var/log/weather/.log*",
"kinesisStream": "Kinesis-Data-Stream-City-Temp",
"partitionKeyOption": “RANDOM”,
“dataProcessingOptions": [
{
“optionName” : “CSVTOJSON”
“customFieldNames” : [ “Region”,”Country”,”City”,”Month”,”Day”,”Year”,”AvgTemperature”]
}
]
}
]

sudo service aws-kinesis-agent start


sudo chkconfig aws-kinesis-agent on

sudo service aws-kinesis-agent stop

sudo service aws-kinesis-agent restart


Copy data into EC2 from local

scp -i “kinesis-demo.pem” /Users/msingh/AWS/KINESIS/city_temperature_first.log [email protected]: /var/log/weather

1- Add the www group to your EC2 instance with the following command:
[ec2-user ~]$ sudo groupadd weather

2- Add the ec2-user user to the weather group:


[ec2-user ~]$ sudo usermod -a -G weather ec2-user

3- To refresh your permissions and include the new www group, log out:
[ec2-user ~]$ exit

4- Log back in again and verify that the www group exists with the groups:
[ec2-user ~]$ groups
> ec2-user wheel weather

5- Change the group ownership of the /var/weather directory and its contents to the www group:
[ec2-user ~]$ sudo chown -R root:weather /var/log/weather

6- Change the directory permissions of /var/www and its subdirectories to add group write permissions and set the group ID on subdirectories created in the
future:
[ec2-user ~]$ sudo chmod 2775 /var/log/weather
[ec2-user ~]$ find /var/log/weather -type d -exec sudo chmod 2775 {} +
Thank you

You might also like