Custodian demo 1 & 2

2023年08月14日

% source custodian/bin/activate

% touch custodian.yml

File content:

policies:
  - name: my-first-policy
    resource: aws.ec2
    filters:
      - "tag:Custodian": present

说明：
name是自定义的。执行custodian run命令后，在参数"--output-dir"所指定的路径下会创建一个同名的文件夹。

Special Values

These meta-values can be used to test whether or not a resource contains a specific value, and if the value is empty.
present: matches when a key does exist

% custodian -h

usage: custodian [-h] {run,schema,report,logs,metrics,version,validate} ...

Cloud Custodian - Cloud fleet management

optional arguments:
  -h, --help            show this help message and exit

commands:
  {run,schema,report,logs,metrics,version,validate}
    run                 Execute the policies in a config file
    schema              Interactive cli docs for policy authors
    report              Tabular report on policy matched resources
    version             Display installed version of custodian
    validate            Validate config files against the json schema

Create an EC2 instance, and tag it with the key Custodian with any value.

AWSTemplateFormatVersion: '2010-09-09'
Description: An EC2 instance with tag key Custodian
Parameters:
  ImageId:
    Type: AWS::EC2::Image::Id
    Description: The AMI ID for the EC2 instance
  InstanceType:
    Type: String
    Description: The instance type for the EC2 instance
  SecurityGroupIds:
    Type: List<AWS::EC2::SecurityGroup::Id>
    Description: The security group IDs for the EC2 instance
  SubnetId:
    Type: AWS::EC2::Subnet::Id
    Description: The subnet ID for the EC2 instance
Resources:
  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      SubnetId: !Ref SubnetId
      SecurityGroupIds: !Ref SecurityGroupIds
      Tags:
        - Key: Custodian
          Value: blablabla # You can change the value as needed
Outputs:
  InstanceId:
    Description: The instance ID
    Value: !Ref EC2Instance

ImageId: ami-0e****14
InstanceType: t4g.micro
SecurityGroupIds: sg-02****f2
For the Sandbox environment, refer to OneNote page.

% aws configure set region eu-central-1

% custodian run --output-dir=. custodian.yml

2023-08-14 16:16:06,821: custodian.policy:INFO policy:my-first-policy resource:aws.ec2 region:us-west-2 count:0 time:6.03

PS:
其中，参数flag "-s"等效于"--output-dir"，用来指定OUTPUT_DIR，即directory or S3 URL for policy output。

Stop all EC2 instances that are tagged with Custodian.

policies:
  - name: my-first-policy
    resource: aws.ec2
    filters:
      - "tag:Custodian": present
    actions:
      - stop

% custodian run --output-dir=. custodian.yml

2023-08-14 16:31:36,590: custodian.policy:INFO policy:my-first-policy resource:aws.ec2 region:us-west-2 count:1 time:7.43
2023-08-14 16:31:39,055: custodian.policy:INFO policy:my-first-policy action:stop resources:1 execution_time:2.46

You should find the instance stopping or stopped in your AWS console.

You should also find a new my-first-policy directory with a log and other files (subsequent runs will append to the log by default rather than overwriting it).

% tree my-first-policy

my-first-policy
├── action-stop
├── custodian-run.log
├── metadata.json
└── resources.json

0 directories, 4 files

PS: 如果系统找不到这个命令，执行yum install tree命令安装。

-bash: tree: command not found

% cat my-first-policy/custodian-run.log

2023-08-14 16:16:06,821 - custodian.policy - INFO - policy:my-first-policy resource:aws.ec2 region:us-west-2 count:0 time:6.03
2023-08-14 16:31:36,590 - custodian.policy - INFO - policy:my-first-policy resource:aws.ec2 region:us-west-2 count:1 time:7.43
2023-08-14 16:31:39,055 - custodian.policy - INFO - policy:my-first-policy action:stop resources:1 execution_time:2.46

Create an IAM role CloudCustodian-QuickStart with below inline policy.

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AllowCustodianToActOnEc2Instances",
        "Action": [
          "ec2:CreateTags",
          "ec2:DeleteTags",
          "ec2:DescribeInstances",
          "ec2:TerminateInstances",
          "ec2:DescribeVolumes"
        ],
        "Effect": "Allow",
        "Resource": "*"
      },
      {
        "Sid": "AllowCustodianToLogLambdaActions",
        "Action": [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ],
        "Effect": "Allow",
        "Resource": "*"
      }
    ]
  }

This role need to be assumed by Lambda, so I use below trust relationship.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Create another file ec2_volume_unencrypted.yaml.

policies:
- name: ec2-require-non-public-and-encrypted-volumes
  resource: aws.ec2
  conditions:
    - region: us-west-1
  description: |
    Provision a Lambda and CloudWatch event target
    that looks at all NEW EC2 instances and terminates those with
    unencrypted volumes.
  mode:
    type: cloudtrail
    role: CloudCustodian-QuickStart
    events:
      - RunInstances
  filters:
    - type: ebs
      key: Encrypted
      value: false

这里的condition要和aws profile对应上。

...
  conditions:
    - region: us-west-1
...

profile里的region设置为us-west-1。
Get you the current region at that point in your script.
% aws configure get region

us-west-1

Validate the configuration (note this happens by default on run)
% custodian validate ec2_volume_unencrypted.yaml

2023-08-15 06:18:06,003: custodian.commands:INFO Configuration valid: ec2_volume_unencrypted.yaml

Dryrun on the policies (no actions executed) to see what resources match each policy.
% custodian run --dryrun -s out ec2_volume_unencrypted.yaml

2023-08-15 06:30:23,482: custodian.policy:INFO policy:ec2-require-non-public-and-encrypted-volumes resource:aws.ec2 region:us-west-1 count:1 time:0.85

Run the policy
% custodian run -s out ec2_volume_unencrypted.yaml

2023-08-15 06:27:50,286: custodian.policy:INFO Provisioning policy lambda: ec2-require-non-public-and-encrypted-volumes region: us-west-1
2023-08-15 06:27:51,104: custodian.serverless:INFO Publishing custodian policy lambda function custodian-ec2-require-non-public-and-encrypted-volumes

增加actions。请谨慎操作！

policies:
- name: ec2-require-non-public-and-encrypted-volumes
  resource: aws.ec2
  conditions:
    - region: us-west-1
  description: |
    Provision a Lambda and CloudWatch event target
    that looks at all new instances and terminates those with
    unencrypted volumes.
  mode:
    type: cloudtrail
    role: CloudCustodian-QuickStart
    events:
      - RunInstances
  filters:
    - type: ebs
      key: Encrypted
      value: false
  actions:
    - terminate

% custodian validate ec2_volume_unencrypted.yaml

2023-08-15 06:33:47,039: custodian.commands:INFO Configuration valid: ec2_volume_unencrypted.yaml

% custodian run --dryrun -s out ec2_volume_unencrypted.yaml

2023-08-15 06:34:21,126: custodian.policy:INFO policy:ec2-require-non-public-and-encrypted-volumes resource:aws.ec2 region:us-west-1 count:1 time:1.29

Now I create an EC2 instance with EBS volume unencrypted.

It will be terminated immediately.

Check the CloudWatch logs generated by the Lambda function.

[Cleanup]
Delete Lambda function "custodian-ec2-require-non-public-and-encrypted-volumes".
Delete IAM role "CloudCustodian-QuickStart".
Delete Log group "/aws/lambda/custodian-ec2-require-non-public-and-encrypted-volumes".

# vim s3_access_log.yml

policies:
  - name: check-bucket-logging
    resource: s3
    filters:
      - type: bucket-logging
        op: disabled

这个policy的目的是列出所有没有启用S3 server access logging的S3 buckets。

# custodian run --output-dir=. s3_access_log.yml

2024-04-15 05:37:15,675: custodian.policy:INFO policy:check-bucket-logging resource:s3 region:eu-central-1 count:5 time:2.52

There will be a new check-bucket-logging directory with a log and other files (subsequent runs will append to the log by default rather than overwriting it).
# tree check-bucket-logging/

check-bucket-logging/
├── custodian-run.log
├── metadata.json
└── resources.json

0 directories, 3 files

# cat check-bucket-logging/custodian-run.log

2024-04-15 05:37:15,675 - custodian.policy - INFO - policy:check-bucket-logging resource:s3 region:eu-central-1 count:5 time:2.52

# cat check-bucket-logging/resources.json

[
  {
    "Name": "aws-cloudtrail-logs-211125714651-ea036076",
    "CreationDate": "2024-03-04T06:24:30+00:00",
    "Location": {
      "LocationConstraint": "eu-west-1"
    },
    "Tags": [],
    "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"AWSCloudTrailAclCheck20150319\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"cloudtrail.amazonaws.com\"},\"Action\":\"s3:GetBucketAcl\",\"Resource\":\"arn:aws:s3:::aws-cloudtrail-logs-211125714651-ea036076\",\"Condition\":{\"StringEquals\":{\"aws:SourceArn\":\"arn:aws:cloudtrail:eu-west-1:211125714651:trail/verify-cost-mgmt-scp\"}}},{\"Sid\":\"AWSCloudTrailWrite20150319\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"cloudtrail.amazonaws.com\"},\"Action\":\"s3:PutObject\",\"Resource\":\"arn:aws:s3:::aws-cloudtrail-logs-211125714651-ea036076/AWSLogs/211125714651/*\",\"Condition\":{\"StringEquals\":{\"aws:SourceArn\":\"arn:aws:cloudtrail:eu-west-1:211125714651:trail/verify-cost-mgmt-scp\",\"s3:x-amz-acl\":\"bucket-owner-full-control\"}}}]}",
    "Acl": {
      "Owner": {
        "DisplayName": "aws-root-a3789554-7da4-4abe-845d-a970de89e836",
        "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53"
      },
      "Grants": [
        {
          "Grantee": {
            "DisplayName": "aws-root-a3789554-7da4-4abe-845d-a970de89e836",
            "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53",
            "Type": "CanonicalUser"
          },
          "Permission": "FULL_CONTROL"
        }
      ]
    },
    "Replication": null,
    "Versioning": {},
    "Website": null,
    "Logging": {},
    "Notification": {},
    "Lifecycle": null
  },
  {
    "Name": "cf-templates-ror5mangohw4-eu-central-1",
    "CreationDate": "2024-04-15T01:10:44+00:00",
    "Location": {
      "LocationConstraint": "eu-central-1"
    },
    "Tags": [],
    "Policy": null,
    "Acl": {
      "Owner": {
        "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53"
      },
      "Grants": [
        {
          "Grantee": {
            "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53",
            "Type": "CanonicalUser"
          },
          "Permission": "FULL_CONTROL"
        }
      ]
    },
    "Replication": null,
    "Versioning": {},
    "Website": null,
    "Logging": {},
    "Notification": {},
    "Lifecycle": null
  },
  {
    "Name": "cf-templates-ror5mangohw4-eu-west-1",
    "CreationDate": "2024-03-05T03:29:16+00:00",
    "Location": {
      "LocationConstraint": "eu-west-1"
    },
    "Tags": [],
    "Policy": null,
    "Acl": {
      "Owner": {
        "DisplayName": "aws-root-a3789554-7da4-4abe-845d-a970de89e836",
        "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53"
      },
      "Grants": [
        {
          "Grantee": {
            "DisplayName": "aws-root-a3789554-7da4-4abe-845d-a970de89e836",
            "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53",
            "Type": "CanonicalUser"
          },
          "Permission": "FULL_CONTROL"
        }
      ]
    },
    "Replication": null,
    "Versioning": {},
    "Website": null,
    "Logging": {},
    "Notification": {},
    "Lifecycle": null
  },
  {
    "Name": "cov-macie-repo-result",
    "CreationDate": "2024-04-07T07:49:58+00:00",
    "Location": {
      "LocationConstraint": "eu-central-1"
    },
    "Tags": [],
    "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"Deny non-HTTPS access\",\"Effect\":\"Deny\",\"Principal\":\"*\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::cov-macie-repo-result/*\",\"Condition\":{\"Bool\":{\"aws:SecureTransport\":\"false\"}}},{\"Sid\":\"Deny incorrect encryption header. This is optional\",\"Effect\":\"Deny\",\"Principal\":{\"Service\":\"macie.amazonaws.com\"},\"Action\":\"s3:PutObject\",\"Resource\":\"arn:aws:s3:::cov-macie-repo-result/*\",\"Condition\":{\"StringNotEquals\":{\"s3:x-amz-server-side-encryption-aws-kms-key-id\":\"arn:aws:kms:eu-central-1:211125714651:key/e1254929-b5dd-4bf1-b04d-589ceeee682f\"}}},{\"Sid\":\"Deny unencrypted object uploads. This is optional\",\"Effect\":\"Deny\",\"Principal\":{\"Service\":\"macie.amazonaws.com\"},\"Action\":\"s3:PutObject\",\"Resource\":\"arn:aws:s3:::cov-macie-repo-result/*\",\"Condition\":{\"StringNotEquals\":{\"s3:x-amz-server-side-encryption\":\"aws:kms\"}}},{\"Sid\":\"Allow Macie to upload objects to the bucket\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"macie.amazonaws.com\"},\"Action\":\"s3:PutObject\",\"Resource\":\"arn:aws:s3:::cov-macie-repo-result/*\",\"Condition\":{\"StringEquals\":{\"aws:SourceAccount\":\"211125714651\"},\"ArnLike\":{\"aws:SourceArn\":[\"arn:aws:macie2:eu-central-1:211125714651:export-configuration:*\",\"arn:aws:macie2:eu-central-1:211125714651:classification-job/*\"]}}},{\"Sid\":\"Allow Macie to use the getBucketLocation operation\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"macie.amazonaws.com\"},\"Action\":\"s3:GetBucketLocation\",\"Resource\":\"arn:aws:s3:::cov-macie-repo-result\",\"Condition\":{\"StringEquals\":{\"aws:SourceAccount\":\"211125714651\"},\"ArnLike\":{\"aws:SourceArn\":[\"arn:aws:macie2:eu-central-1:211125714651:export-configuration:*\",\"arn:aws:macie2:eu-central-1:211125714651:classification-job/*\"]}}},{\"Sid\":\"S3PolicyStmt-DO-NOT-MODIFY-1713148053100\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"logging.s3.amazonaws.com\"},\"Action\":\"s3:PutObject\",\"Resource\":\"arn:aws:s3:::cov-macie-repo-result/*\",\"Condition\":{\"StringEquals\":{\"aws:SourceAccount\":\"211125714651\"}}}]}",
    "Acl": {
      "Owner": {
        "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53"
      },
      "Grants": [
        {
          "Grantee": {
            "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53",
            "Type": "CanonicalUser"
          },
          "Permission": "FULL_CONTROL"
        }
      ]
    },
    "Replication": null,
    "Versioning": {},
    "Website": null,
    "Logging": {},
    "Notification": {},
    "Lifecycle": null
  },
  {
    "Name": "cov-test-drift-s3-lify-cycle-rule",
    "CreationDate": "2024-03-05T05:58:29+00:00",
    "Location": {
      "LocationConstraint": "eu-west-1"
    },
    "Tags": [
      {
        "Key": "aws:cloudformation:stack-id",
        "Value": "arn:aws:cloudformation:eu-west-1:211125714651:stack/drift-test/c1102f70-daa0-11ee-8946-06d3e2093329"
      },
      {
        "Key": "aws:cloudformation:stack-name",
        "Value": "drift-test"
      },
      {
        "Key": "aws:cloudformation:logical-id",
        "Value": "S3Bucket"
      }
    ],
    "Policy": null,
    "Acl": {
      "Owner": {
        "DisplayName": "aws-root-a3789554-7da4-4abe-845d-a970de89e836",
        "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53"
      },
      "Grants": [
        {
          "Grantee": {
            "DisplayName": "aws-root-a3789554-7da4-4abe-845d-a970de89e836",
            "ID": "45d782d7c7923fa7ef543e326633560394568227282ca9a10a8abe20a2762a53",
            "Type": "CanonicalUser"
          },
          "Permission": "FULL_CONTROL"
        }
      ]
    },
    "Replication": null,
    "Versioning": {},
    "Website": null,
    "Logging": {},
    "Notification": {},
    "Lifecycle": {
      "Rules": [
        {
          "Expiration": {
            "Days": 365
          },
          "ID": "ExampleRule",
          "Filter": {
            "Prefix": ""
          },
          "Status": "Enabled",
          "Transitions": [
            {
              "Days": 90,
              "StorageClass": "STANDARD_IA"
            }
          ]
        },
        {
          "Expiration": {
            "Days": 365
          },
          "ID": "Rule3IaC",
          "Filter": {
            "Prefix": ""
          },
          "Status": "Enabled",
          "Transitions": [
            {
              "Days": 30,
              "StorageClass": "STANDARD_IA"
            }
          ]
        }
      ]
    }
  }
]

里面包含5个元素。
列出了所有没有启用S3 server access logging的S3 buckets。

policies:
  - name: check-bucket-logging
    resource: s3
    filters:
      - type: bucket-logging
        op: disabled
      - type: value
        key: tag:cov:access-logging
        value: not-required
        op: ne
      - type: value
        key: cov:exception:C21.0004
        op: not-in
        value_regex: '^(RITM.*)$'
        value_from:
          url: "s3://{{ exception_bucket }}/whitelist-C21.0004.json"
          format: "json"
          expr: "[?account=='{account_id}'].reference"

policies:
  - name: check-bucket-logging
    resource: s3
    filters:
      - type: bucket-logging
        op: disabled
      - type: value
        key: tag:cov:access-logging
        value: not-required
        op: ne

policies:
  - name: check-bucket-logging
    resource: s3
    filters:
      - type: bucket-logging
        op: disabled
      - type: value
        key: tag:cov:access-logging
        value: not-required
        op: ne
      - type: value
        key: cov:exception:C21.0004
        op: not-in
        value_regex: '^(RITM.*)$'
        value_from:
          url: "s3://{{ exception_bucket }}/whitelist-C21.0004.json"
          format: "json"
          expr: "[?account=='{account_id}'].reference"

-

References

Getting Started

Category: AWS Tags: Custodian public

Sky Cone

Custodian demo 1 & 2

Special Values