11.2.2 Introduction (AWS Lambda linkage tool (for JP1 event registration command))

(a) Register the application log on CloudWatch Logs(EC2 as a JP1 event.)

Install and set up CloudWatch Logs agent on the target EC2 to forward application logs that you want to register as JP1 events to CloudWatch Logs.

For information about detail, see Amazon CloudWatch documentation.

■QuickStart: Install and setup CloudWatch Logs agent on EC2 Linux instance during execution

(b) To register a CloudTail(CloudTrail trail as a JP1 event.)

When registering a trail recorded by CloudTrail as an JP1 event, setup is executed to forward CloudTail trail to CloudWatch Logs.

For more information about detail, see AWS Cloud Trail documentation.

■Sending Events to CloudWatch Logs

(c) Preparing S3

For Lambda function to work, create a bucket to store configuration file of Lambda function.

For information about detail, see Amazon Simple Storage Service documentation.

■Creating a Bucket

Important

It must be setup that can be downloaded from Lambda of AWS Lambda linkage tool (JP1 event registration command compatible version).

(d) Preparing SQS

To run AWS Lambda linkage tool (JP1 event registration command compatible version), create a SQS to store JP1 event data.

For information about detail, see Amazon Simple Queue Service documentation.

■Getting started with Amazon SQS (Amazon Simple Queue Service)

Select "FIFO" in 4 of "Step 1: Create a Queue". Add ".fifo" to the suffix of the queue name as described.

Important

• AWS Lambda linkage tool (for JP1 event registration command) must be setup that can add the queue from Lambda function.

• AWS Lambda linkage tool (for JP1 event registration command) must be set to setup so that the queues can be retrieved and delete from JP1 event sending function.

(a) JP1 Event-Send Function Arrangement

This function allocates JP1 event-sending function.

1. Login to the execution server of the JP1 event-sending function.

2. Copy the "sendevent_aws_lambda" directory to any directory.

3. Grant the required permissions to the directories and files that you copied in step 2.

   Table 11‒5: Directory-structure of the JP1 event-sending facility

   No.	Directory structure		Description	Required permissions
   1	sendevent_aws_lambda		AWS Lambda linkage tool directory	Write permission#1
   2		sendevent_aws.py	AWS Lambda linkage	Execute permission#2
   3		logging.conf	Logging configuration file	Read permission#3

   #1

   Write permission: Grant permissions that can be written to by anyone who wants to execute JP1 event-sending functionality.

   #2

   Execute permission: If you want to execute JP1 event-sending facility, grant the privilege to execute.

   #3

   Read permission: Grant permissions that can be read by anyone who wants to execute JP1 event-sending functionality.

(b) Configuration file Deployment for Lambda functions

Place configuration file for Lambda functionI will.

1. Login AWS Management Console from a browser.

2. Select S3 from services.

3. Select the bucket you created in 11.2.2(1)(c) Preparing S3.

4. Upload the following files in the sendevent_sqs directory:

   • eventconf.csv

   • eventignore.csv

   • logging.conf

(c) Creating Lambda functions

Create a Lambda functionI will.

1. Login AWS Management Console from a browser.

2. Select Lambda-functions from Services.

3. Click Create a Function.

4. Select Create from scratch and enter the following information: Click Create a Function.

   Table 11‒6: Basic Info about Lambda functions

   No.	Item	Value
   1	Function name	sendevent_sqs
   2	Run time	Python 3.7
   3	Execute role	Specify a role with the following permissions: Permissions that can be written to CloudWatch Logs#1 Permissions that can read S3's files#2 Permissions that can be written to SQS#3

   #1

   The following policies are required for accessing CloudWatch Logs:

   - CreateLogGroup

   - CreateLogStream

   - PutLogEvents

   #2

   The following policies are required for accessing S3:

   - GetObject

   #3

   The following policies are required for accessing SQS:

   - SendMessage

5. Select "Lambda"-"Functions"-"sendevent_sqs"-"Setup".

6. For Function Code, enter the information shown in the following table.

   Table 11‒7: Function codes for Lambda functions

   No.	Item	Value
   1	Code Entry Type	Upload.zip file In the sendevent_sqs directory, select sendevent_sqs.zip.
   2	Run time	Python3.7
   3	Handler	sendevent_sqs.lambda_handler

7. Click Save.

(a) JP1 Event Send Function (sendevent_aws_lambda.py)

This section explains setup steps related to AWS CLI, JP1/Base of JP1 event-sending function.

Setup field

• AWS CLI installation directory

• JP1/Base installation directory

• AWS Access Key ID/Secret Access Key

• AWS region

• Proxy setup

• SQS URL

1. Login to the execution server of the AWS Lambda linkage tool (JP1 event registration command compatible version) JP1 event-sending function.

2. Open "sendevent_aws_lambda".py under the "sendevent_aws_lambda" directory that was copied by 11.2.2(2)(a) JP1 Event-Send Function Arrangement in a text editor.

3. Setup the user-variable in lines 18 through 25.

   Table 11‒8: User-variable number of the JP1 event-sending facility (sendevent_aws_lambda.py)

   No.	Parameters	Required/Optional	Description
   1	AWS_CLI_DIR	Required	Specify AWS CLI install directory.
   2	BASE_DIR	Required	Specify JP1/BASE install directory.
   3	ACCESS_KEY	Optional	Setup the access key ID of AWS. However, setup is not required if the access key ID is setup in AWS CLI(aws configure). #1
   4	SECRET_ACCESS_KEY	Optional	Setup the secret access key for AWS. However, setup is not required if the secret access key is Setup in AWS CLI(aws configure). #1
   5	DEFAULT_REGION	Required	Setup AWS region.
   6	HTTP_PROXY	Optional	If AWS CLI is accessed through a proxy, specify the proxy URL, user name and password. #2 If not specified, OS uses value of HTTP_PROXY/HTTPS_PROXY that is Setup.
   7	HTTPS_PROXY	Optional
   8	SQS_QUEUE_URL	Required	Specify the URL of the SQS that contains the JP1 event information created in 11.2.2(1)(d) Preparing SQS.

   #1

   The specified account must have the following policy:

   - ReceiveMessage

   - DeleteMessage

   #2

   For information about setup values, see Amazon Command Line Interface documentation.

   ■Use HTTP Proxy-authentication the proxy

   The following shows a sample setup:

   15: ###############################################################
   16: # User-variable (Value enclosed in double quotes)
   17: ###############################################################
   18: AWS_CLI_DIR = r"C:\Program Files\Amazon\AWSCLI\bin"
   19: BASE_DIR = r"C:\Program Files (x86)\Hitachi\JP1Base"
   20: ACCESS_KEY = "AAAAAAAAAA"
   21: SECRET_ACCESS_KEY = "BBBBBBBBBB"
   22: DEFAULT_REGION = "ap-northeast-1"
   23: HTTP_PROXY = "http://username:password@proxy.example.com:1234"
   24: HTTPS_PROXY = "http://username:password@proxy.example.com:1234"
   25: SQS_QUEUE_URL = "https://sqs.ap-northeast-1.amazonaws.com/999999999999/queue.fifo"
   26: ###############################################################

4. Setup the user-defined variable value in step 2, and then save it with the following character code/line feed code.

   Table 11‒9: Character code/line feed code of the JP1 event send function (sendevent_aws_lambda.py)

   No.	Item	Value
   1	Character code	UTF-8 (without BOM)
   2	Line feed code	LF (\n) or CRLF (\r\n)

(b) Log configuration file (logging.conf) of JP1 event send function

This section explains setup steps for logging JP1 event sending functionality.

Setup fields

• Log level

• Logging file

• Log rotation size

• Number of log generations

1. Login to the executing server of the JP1 event sending function.

2. Open "logging.conf" under "sendevent_aws_lambda" which was copied in 11.2.2(2)(a) JP1 Event-Send Function Arrangement in a text editor.

3. Modify the following lines:

   11th line

   11: level=INFO#1

   17th line

   17: args=('sendevent_aws_lambda.log', 'a', 2097152, 3, 'utf-8')#2#3#4

   #1

   Specifies the logging level. You can specify DEBUG,INFO,WARNING,ERROR, or CRITICAL.

   #2

   Specify the log file path. You can specify an absolute or relative path.

   #3

   Specifies the log rotation size in bytes.

   #4

   Specifies the number of log generations.

   For information about detail, see Python documentation.

   ■Environmental configuration file format

4. Setup the user-defined variable value in step 2, and then save it with the following character code/line feed code.

   Table 11‒10: Character code/end-of-line code of the log configuration file (logging.conf) of the JP1 event sending function

   No.	Item	Value
   1	Character code	UTF-8
   2	End-of-line code	LF (\n) or CRLF (\r\n)

(c) Event-definition file (eventconf.csv)

This section explains setup step for JP1 event that is registered for AWS Lambda linkage tool (JP1 event registration command supported).

Setup fields

• JP1 Events registration requirements

• JP1 event-information

1. Login AWS Management Console from a browser.

2. Select S3 from services.

3. Select the bucket you created in 11.2.2(1)(c) Preparing S3.

4. Download eventconf.csv from the bucket you selected in step 3 to the appropriate directory.

   If File does not exist in the bucket, please copy the above files under sendevent_sqs to the appropriate directory.

5. Open eventconf.csv you downloaded or copy in step 4 in a text editor and setup the event-definition.

   • Write in one entry per line.

   • A line with the first character "#" is a comment line and is ignored.

   • Write in CSV format.

   • JP1 event definition is searched from the top, and JP1 event with the condition that matches the condition is fired first.

   The columns in CSV are listed in the following tables.

   Table 11‒11: Event definition columns

   Column number	Setup values	Required/Optional	Scope of values
   1	Specifies a regular expression that identifies the log that you want to register as a JP1 event.	Required	Range supported by regular expressions
   2	Specifies the log group name of the log that you want to register as a JP1 event. If this option is omitted, all log groups are subject to processing.	Optional	Range supported by AWS log group name
   3	Specifies event ID of JP1 event that you want to register. #1	Optional	Scope supported by jevsend command-argument-i
   4	Specifies message of JP1 event to register. If this option is omitted, the target log is used. #2	Optional	Scope supported by jevsend command-argument-m
   5	Specifies the destination event server name for JP1 events that you want to register. #1	Optional	Scope supported by jevsend command-argument-d
   6	Specifies the source event server name for JP1 events that you want to register. #1	Optional	Scope supported by jevsend command-argument-s
   7 or later	Specifies the extended properties of JP1 events that you want to register. #1#3 To specify multiple extended attributes, specify column numbers 7 or later.	Optional	The scope supported by jevsend command-argument-e. Note that you do not need to specify " even if the extended attribute value contains spaces. (" is also recognized as an extended attribute value.)

   #1

   If omitted, this is the default behavior for the corresponding jevsend command-option.

   #2

   For message exceeding 1,023 bytes, the range is up to 1,023 bytes.

   Also, if message contains a carriage return, the carriage return code is converted to a single-byte space.

   #3

   In addition to the specified extended attributes, there are extended attributes that are added by AWS Lambda linkage tool (JP1 event register command support version). The extended properties to add are as follows:

   Table 11‒12: Extended attributes that are added by AWS Lambda linkage tool (JP1 event register command support version)

   No.	Attribute name	Value
   1	AWS_LAMBDA_LOG_TIMESTAMP	UNIX period during which logging was added to CloudWatch Logs
   2	AWS_LAMBDA_LOG_GROUP	Target log group name
   3	AWS_LAMBDA_LOG_STREAM	Target log stream name
   4	AWS_LAMBDA_SUBSCRIPTION_FILTERS	A comma-separated setup of subscription filter names for value (No comma if setup value is one)
   5	AWS_LAMBDA_LOG_DATETIME	Character string in which item 1 is converted to "YYYY/mm/dd HH:MM:SS"
   6	AWS_LAMBDA_MSGID	Target JP1 event-specific ID# #: If the same value is specified, it can be determined that JP1 events are the same.

   The following shows a sample setup:

   #Condition,LogGroupId,EventID,Message,Destination,Source,ExtAttr
   .*Error.*,LOGGROUP1,1111,message1,DEST_HOST,SRC_HOST,EXATTR1=value1,EXATTR2=value2
   .*Warning.*,,2222,message2,DEST_HOST,SRC_HOST,EXATTR1=aaa bbb,EXATTR2=value

6. After changing setup in step 5, save it with the following character code/end-of-line code.

   Table 11‒13: Event-definition file (eventconf.csv) character code/end-of-line code

   No.	Item	Value
   1	Character code	UTF-8
   2	End-of-line code	LF (\n) or CRLF (\r\n)

7. Re-select the bucket in step 3 and upload setup modified eventconf.csv.

(d) Exclude Events File (eventignore.csv)

This section explains how to setup JP1 events that are excluded from AWS Lambda linkage tool (JP1 event register command support version).

Setup fields

• JP1 events exclusion conditions

1. Login AWS Management Console from a browser.

2. Select S3 from services.

3. Select the bucket you created in 11.2.2(1)(c) Preparing S3.

4. Download eventignore.csv from the bucket you selected in step 3 to the appropriate directory.

   If file does not exist in the bucket, please copy the above file under sendevent_sqs to the appropriate directory.

5. Open eventignore.csv that you downloaded or copy in step 4 in a text editor and setup the event-definition.

   • Write in one entry per line.

   • A line with the first character "#" is a comment line and is ignored.

   • Write in CSV format.

   • JP1 event exclusion definitions are judged in order from the top, and if they match, they are excluded from JP1 event.

   The columns in CSV are listed in the following tables.

   Table 11‒14: Event exclusion definition columns

   Column number	Setup values	Required/Optional	Scope of value
   1	Specifies a regular expression that identifies the log that you want to exclude as JP1 events.	Required	Range supported by regular expressions
   2	Specifies the log group of the logs that you want to exclude as JP1 events. If this option is omitted, all log groups are subject to processing.	Optional	Range supported by AWS log group name

   The following shows a sample setup:

   #Condition,LogGroupId
   .*INFO.*,LOGGROUP2
   .*DEBUG.*,

6. After changing setup in step 2, save it with the following character code/end-of-line code.

   Table 11‒15: Event-excluded file (eventignore.csv) character code/line feed code

   No.	Item	Value
   1	Character code	UTF-8
   2	End-of-line code	LF (\n) or CRLF (\r\n)

7. Re-select the bucket in step 3 and upload setup modified eventignore.csv.

(e) Lambda (sendevent_sqs.py)

Describes setup step for Lambda function.

Setup fields

• SQS URL

• S3 bucket name

• Execute triggering (JP1 event registration target logging)

1. Login AWS Management Console from a browser.

2. Select "Lambda"-"Functions"-"sendevent_sqs"-"Setup" from services.

3. Setup the following items in "Environment-variable".

   Table 11‒16: Environmental variables for Lambda functions

   No.	Key	Values
   1	SQS_QUEUE_URL	Specifies 11.2.2(1)(d) Preparing SQS. JP1 event-information is stored in URL.
   2	S3_BUCKET_NAME	Specifies AWS Lambda bucket for storing JP1 event registration command-compatible Lambda data configuration file which created in 11.2.2(1)(c) Preparing S3.

4. In Designer, press Add Trigger to trigger the log you want to add as a JP1 event. You can add several triggers.

   Table 11‒17: Triggering for Lambda functions

   No.	Item	Values	Remarks
   1	Trigger	CloudWatch Logs
   2	Log group	Specify the log group that contains the logs you want to register as JP1 events.	Do not specify a log group that contains logs that are output by Lambda function of AWS Lambda linkage tool (for JP1 event registration command).
   3	Name of filter	Specifies the name of filter
   4	Filter patterning	Specifies filter pattern for the log that you want to register as a JP1 event.	It does not matter if you do not specify it.

5. Click Save.

(f) Logging configuration file of Lambda (logging.conf)

This section explains setup procedure for logging Lambda functions for AWS Lambda linkage tool (JP1 event register command support version).

Setup fields

• Log level

1. Login AWS Management Console from a browser.

2. Select S3 from services.

3. Select the bucket you created in11.2.2(1)(c) Preparing S3.

4. Download logging.conf from the bucket you selected in step 3 to the appropriate directory.

   If file does not exist in the bucket, please copy the above file under sendevent_sqs to the appropriate directory.

5. Open logging.conf that you downloaded or copy in step 4 in a text editor and change the following line:

   11th line

   11: level=INFO#

   #

   Specifies the logging level. You can specify DEBUG,INFO,WARNING,ERROR, or CRITICAL.

   For information about detail, see Python documentation.

   ■Environmental configuration file format

6. After changing setup in step 2, save it with the following character code/end-of-line code.

   Table 11‒18: Character code/end-of-line code of log configuration file (logging.conf) of Lambda function

   No.	Item	Values
   1	Character code	UTF-8
   2	End-of-line code	LF (\n) or CRLF (\r\n)

7. Re-select the bucket in step 3 and upload setup modified logging.conf.

8. After you upload configuration file, you can Lambda the function to restart (for example, save Lambda function).

(a) Uninstalling JP1 event-sending function

The following describes how to uninstall JP1 event-sending functionality.

1. Login to the executing server of the JP1 event-sending function.

2. Delete the "sendevent_aws_lambda" directory that was copied in 11.2.2(2)(a) JP1 Event-Send Function Arrangement.

(b) Uninstalling Lambda functions

The following describes how to uninstall Lambda.

1. Login AWS Management Console from a browser.

2. Select Lambda-functions from services.

3. Select "sendevent_sqs" and then execute "Action"-"Delete".

4. Select S3 from services.

5. Select the bucket you created in 11.2.2(1)(c) Preparing S3.

6. Select the following file, and then execute "Action"-"Delete".

   • entconf.csv

   • eventignore.csv

   • logging.conf