AWS - Amazon Web Services

From Noah.org
Jump to navigationJump to search

ERROR: "client error (AuthFailure) ... AWS was not able to validate the provided access credentials"

If you see an error like this and you have confirmed that your credentials are good then the next thing to check is your clock. Even AWS instances somehow seem to get clocks askew.

$ aws ec2 --region=us-west-1 describe-security-groups --query 'SecurityGroups[*].GroupId' --output table

A client error (AuthFailure) occurred when calling the DescribeSecurityGroups operation: AWS was not able to validate the provided access credentials

Try running this on the instance to correct clock drift:

sudo ntpdate time.nist.gov

clips

# For safety this will only print the command you would use.
function aws-terminate () {
    local IFS=","
    echo aws ec2 terminate-instances --instance-ids "$*"
}

function aws-console () {
    aws ec2 get-console-output --output text --latest --instance-id $1
}

function aws-instanceid-by-name () {
    local IFS=","
    aws ec2 describe-instances --output text --query "Reservations[*].Instances[*].[InstanceId]" --filters "Name=tag:Name,Values=$*"
}

# Very slow.
function aws-instanceid-by-private-ip () {
    local IFS=","
    #aws ec2 describe-instances --output text --query "Reservations[*].Instances[*].[InstanceId]" --filters "Name=private-ip-address,Values=$*"
    aws ec2 describe-network-interfaces --filters "Name=addresses.private-ip-address,Values=$*"
}

The minimum Route53 stuff that I actually use or care about

Route53 stuff

Notes

  1. The Hosted Zone ID for example.com is "Z35CNAMBBVZ957".
  2. All names are specified with the FQDN even though the domain name is implied with the Hosted Zone ID.
  3. Route53 automatically creates Reverses (PTR records) for A records. This is a beautiful thing.
# This lists the zones and Hosted Zone IDs necessary for later commands.
$ aws route53 list-hosted-zones

# This lists all DNS info for a given Hosted Zone ID.
# This lists every DNS record.
$ aws route53 list-resource-record-sets --hosted-zone-id Z35CNAMBBVZ957

# This will give you a JSON outline that you will use later.
# To do anything useful with records you have to use JSON because
# there are no CLI options for most DNS operations.
# Note that this JSON contains fields that need to be deleted
# or are not necessary for most operations. The examples after this
# one will probably be more useful. This is best just for reference.
$ aws route53 change-resource-record-sets --generate-cli-skeleton
{
    "HostedZoneId": "", 
    "ChangeBatch": {
        "Comment": "", 
        "Changes": [
            {
                "Action": "", 
                "ResourceRecordSet": {
                    "Name": "", 
                    "Type": "", 
                    "SetIdentifier": "", 
                    "Weight": 0, 
                    "Region": "", 
                    "GeoLocation": {
                        "ContinentCode": "", 
                        "CountryCode": "", 
                        "SubdivisionCode": ""
                    }, 
                    "Failover": "", 
                    "TTL": 0, 
                    "ResourceRecords": [
                        {
                            "Value": ""
                        }
                    ], 
                    "AliasTarget": {
                        "HostedZoneId": "", 
                        "DNSName": "", 
                        "EvaluateTargetHealth": true
                    }, 
                    "HealthCheckId": ""
                }
            }
        ]
    }
}

# Now for the useful, practical operations...

# This is an example that creates an A record.
$ aws route53 change-resourc-record-sets --cli-input-json '{
    "HostedZoneId": "Z35CNAMBBVZ957",
    "ChangeBatch": {
        "Comment": "This is a test and may be deleted.",
        "Changes": [
            {
                "Action": "CREATE",
                "ResourceRecordSet": {
                    "Name": "noah-test.example.com",
                    "Type": "A",
                    "TTL": 600,
                  "ResourceRecords": [
                    {
                      "Value": "192.168.0.1"
                    }
                  ]
                }
            }
        ]
    }
}'

# You may want to test your new A record using `host`:
$ host noah-test.example.com
noah-test.example.com has address 192.168.0.1

# This updates an existin A record.
$ aws route53 change-resource-record-sets --cli-input-json '{
    "HostedZoneId": "Z35CNAMBBVZ957",
    "ChangeBatch": {
        "Comment": "This is a test A and may be deleted.",
        "Changes": [
            {
                "Action": "UPSERT",
                "ResourceRecordSet": {
                    "Name": "noah-test.example.com",
                    "Type": "A",
                    "TTL": 600,
                  "ResourceRecords": [
                    {
                      "Value": "192.168.1.2"
                    }
                  ]
                }
            }
        ]
    }
}'

# This creates an alias (CNAME) to an A record.
$ aws route53 change-resource-record-sets --cli-input-json '{
    "HostedZoneId": "Z35CNAMBBVZ957",
    "ChangeBatch": {
        "Comment": "This is a test CNAME and may be deleted.",
        "Changes": [
            {
                "Action": "CREATE",
                "ResourceRecordSet": {
                    "Name": "noah-test-cname.example.com",
                    "Type": "CNAME",
                    "TTL": 600,
                  "ResourceRecords": [
                    {
                      "Value": "noah-test.example.com"
                    }
                  ]
                }
            }
        ]
    }
}'

# This tests if a resource record set exists or not in DNS.
# This is kind of lame, but it's the only way I've been able to figure out
# how to do it. I suppose you could assume it does not exist and do a CREATE
# and check if that fails due to the record already existing then fall-back
# to doing an update (UPSERT) if it does.
# At any rate, this command will not generate output, but it will return
# an exit code of 0 (exists) or 1 (does not exist). Beware of the
# weird quoting, but you shouldn't have to change anything except the
# variable, DOMAIN_TO_TEST..
$ DOMAIN_TO_TEST="noah-test.example.com"
$ aws route53 list-resource-record-sets --hosted-zone-id Z35CNAMBBVZ957 | grep -q '"Value": "'${DOMAIN_TO_TEST}'"'
$ echo $?
0

EC2 Instance Store -- Ephemeral Storage

EC2 Instance Store is ephemeral storage. When the EC2 instance is terminated this storage is lost. The EC2 guest operating system runs mostly off of instance storage. EC2 instance storage runs off a physical drive on the host physical machine. You have little control over this storage outside of the EC2 instance. You can choose some options to before an instance is started to modify performance, such as choosing Hard Disk, SSD, or NVMe SSD backing storage.

Many of the typical directories you may care about for storage are mounted on instance store devices, so beware; these are all lost if your EC2 instance is terminated or crashes. See #EBS -- Elastic Block Store or #EFS -- Elastic File System if you want to store data in a mounted file system that is persistent. Or consider S3 storage, but this is not intended for file systems.

# df -h / /tmp /etc /opt /var/log /var/lib /root
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  194G   34G  161G  18% /
/dev/nvme0n1p1  194G   34G  161G  18% /
/dev/nvme0n1p1  194G   34G  161G  18% /
/dev/nvme0n1p1  194G   34G  161G  18% /
/dev/nvme0n1p1  194G   34G  161G  18% /
/dev/nvme0n1p1  194G   34G  161G  18% /
/dev/nvme0n1p1  194G   34G  161G  18% /


EBS -- Elastic Block Store

EBS (Elastic Block Store) is intended as a drive for EC2 instances. This is in contrast to ephemeral storage called EC2 Instance Store, which is always lost when the EC2 instance is terminated.

EBS volumes behave as disks. They are not easy to scale once an instance is started (can't reprovision, change size, share between instances). If you need scaleable storage see #EFS - Elastic File System storage.

Persistent storage on ephemeral instances

EBS (Elastic Block Store) mounts for ephemeral instances are typically mounted to md[NNN] devices and are formatted as xfs. Example:

$ mount | grep xfs
/dev/md127 on /home type xfs (rw,noatime,attr2,inode64,sunit=1024,swidth=1024,noquota)

EFS -- Elastic File System

EFS (Elastic File System) is intended as a persistent file system storage for EC2 instances. It's not unlike #EBS -- Elastic Block Store, except that it is designed to scale (storage grows as needed, storage may be mounted between different instances).