Coder Legion - Recent posts tagged s3-accesspoints

Mastering Amazon S3 Replication: Same-Region vs. Cross-Region Replication -

Sun, 02 Feb 2025 09:34:30 +0000

Amazon Simple Storage Service (S3) is a powerhouse for cloud storage, offering unmatched scalability, durability, and security. Among its many features, S3 Replication plays a vital role in ensuring data availability, resilience, and compliance. AWS provides two types of replication mechanisms: Same-Region Replication (SRR) and Cross-Region Replication (CRR). While both serve to replicate objects, they cater to different business needs. Let’s dive deep into their differences, use cases, and best practices.

What is Amazon S3 Replication?

S3 Replication is an AWS feature that automatically replicates objects from one S3 bucket to another. The replication can occur within the same AWS region (SRR) or across different regions (CRR). This ensures redundancy, availability, and compliance with organizational policies.

AWS S3 Replication requires enabling Versioning on both the source and destination buckets. Once set up, it replicates newly uploaded objects asynchronously while preserving metadata, object tags, and Access Control Lists (ACLs), depending on your configurations.

Same-Region Replication (SRR)

✅ What is SRR?

SRR allows replication of S3 objects within the same AWS region. This is useful for organizations that need multiple copies of data in the same geography but within different buckets.

Use Cases

Compliance & Governance: Maintain multiple copies of data within a region to meet internal compliance regulations.

Data Tiering & Organization: Separate different teams' data while keeping it in the same region for low latency.

Backup & Disaster Recovery: Keep replicated copies in a different bucket to recover from accidental deletions or corruption.

Logging & Analytics: Send logs to a centralized bucket for analysis without moving them across regions.

⚡ Benefits

Low Latency: Since data is within the same region, replication is faster.

Cost-Effective: No inter-region data transfer costs.

Better Access Control: Different teams can access separate replicated buckets without modifying the source bucket’s policies.

⚠️ Considerations

SRR does not protect against regional failures; for true disaster recovery, CRR is needed.

There might be some added storage costs depending on data volume.

Cross-Region Replication (CRR)

✅ What is CRR?

CRR replicates objects across different AWS regions, providing geographical redundancy and enhanced disaster recovery.

Use Cases

Disaster Recovery: Store copies of critical data in another region to protect against region-wide outages.

Compliance & Regulations: Meet legal requirements for keeping data in multiple geographic locations.

Global Content Distribution: Improve performance by serving data closer to end users in different regions.

High Availability & Fault Tolerance: Maintain business continuity in case of failures in one region.

⚡ Benefits

Resilient Disaster Recovery: Data is available in another region in case of primary region failure.

Reduced Latency for Global Users: Store data closer to international customers to reduce access time.

Improved Business Continuity: Ensures uninterrupted service availability.

⚠️ Considerations

Higher Costs: Inter-region data transfer fees apply.

Increased Latency: Replication takes longer due to network distance.

Regulatory Compliance: Ensure cross-border data transfers align with local regulations.

️ Setting Up S3 Replication (SRR & CRR)

To enable replication:

Enable versioning for both source and destination buckets.

Set up IAM permissions for replication.

Create a Replication Rule specifying:

Source and destination buckets

Filter conditions (prefix, object tags, etc.)

Replication options (SRR or CRR)

Enable S3 Replication Time Control (RTC) for predictable replication times (optional).

Monitor replication progress using AWS CloudWatch and S3 Replication Metrics.

SRR vs. CRR: Which One Should You Choose?

FeatureSame-Region Replication (SRR)Cross-Region Replication (CRR)
Data Transfer CostLower (No inter-region cost)Higher (Inter-region transfer fees)
LatencyLow (Faster replication)Higher (Depends on region distance)
Disaster RecoveryNo protection from regional failuresHigh protection from region-wide failures
ComplianceUseful for internal regulationsRequired for international regulations
Global AccessLimited to one regionEnables worldwide data access

Best Practices for S3 Replication

✅ Use Lifecycle Policies: Delete older versions or move replicated data to S3 Glacier for cost efficiency.
✅ Monitor with CloudWatch: Keep track of replication success rates and failures.
✅ Use Replication Time Control (RTC): Ensure predictable replication times.
✅ Encrypt Replicated Data: Enable AWS KMS encryption to secure sensitive information.
✅ Optimize Costs: Use Intelligent-Tiering for replicated objects to reduce expenses.

Conclusion

Both Same-Region Replication (SRR) and Cross-Region Replication (CRR) are powerful features in AWS S3 that serve different business needs. SRR is ideal for compliance, backups, and data segregation, while CRR is essential for disaster recovery and global distribution. Choosing the right replication strategy depends on your use case, performance requirements, and budget.

By implementing these best practices, you can ensure optimal data availability, security, and resilience in your AWS cloud environment.

Ready to enhance your AWS storage strategy? Implement S3 replication today!

AWS S3 Access Points in 2025 using CLI

Fri, 10 Jan 2025 15:44:49 +0000

Hello, Machas!

Today, I’m here to demystify S3 Access Points. No promises of turning you into a superhero, but by the end of this blog, you’ll at least upgrade from a curious bystander to the protagonist of your own AWS journey. Let’s dive in and make you the star of this S3 saga!.

I hope you might have already known what are bucket polices. If you don’t know I just give you a quick recap: bucket policies are used to restrict/allow access to any user/group/role to access bucket and its objects.

Use of Access points

Lets consider a scenario today if you want to give read permissions to some objects for a user , you will use simply bucket policy to do that .

Tomorrow if you want to give some other permissions to a role you will again use the same bucket policy.

Other day when you want to give permissions to a group if you keep on using this bucket policy this will grow and it becomes hard to read .

That’s where access points comes in rescue , instead of giving permissions to every one in one bucket policy you will create individual access points
for each one and give permissions to these access points and all these access points are attached to the bucket.

In this way bucket policy wont grow .

Now its time to get our hands dirty .

Let’s consider a scenario where you need to allow access to the objects in your bucket only through access points ,
and also grant access to a specific user in another account . Sounds simple, right?"

NAAM CHOTA HAI LEKIN SOUND BADA HAI

1. Firstly let’s create access point.

 

    aws s3control create-access-point --account-id 9848032919 --name my-access-point \
    --bucket my-bucket

Now this will create access point its network origin is internet if you want to create VPC as network origin then No internet access.
Requests are made over a specified VPC only. This will create access point and is attached to the bucket.

Grab the access point arn.

2. Next let’s grant permissions to this access point

I will now grant user KGF to access the bucket and its objects its simple!!! only if you know already know how to write bucket policy.
That’s why have a quick glance at my above attached blog.

Access point policy is similar to bucket policy.

Lets grant KGF user, get object and list bucket permissions , in the access point policy Resource is arn of the access point .

// access-policy.json

    {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": { "AWS": "arn:aws:iam::9848032919:user/KGF" },
            "Action": [
              "s3:GetObject",
              "s3:ListBucket" 
            ],
            "Resource": [
              "arn:aws:s3:ap-south-1:9848032919:accesspoint/my-access-point",
              "arn:aws:s3:ap-south-1:9848032919:accesspoint/my-access-point/object/*"
            ]
          }
        ]
      }

Now KGF user will be having get object and list bucket permission on the bucket to which this access point was attached.

3. Attach the permissions to access point

Now attach this access point policy to the access point.


    aws s3control put-access-point-policy --account-id 9848032919 --name my-access-point \
     --policy file://access-policy.json

Now if KGF user tries to access the bucket using …


    aws s3 ls my-bucket

You will get error as KGF is fresh user this user doesn’t have any IAM permissions to access the object.

If KGF user has IAM permissions then this user will be able to . But we are interested in accessing the objects in the bucket using access point .

Now also we got same error .

4. Access Point-Based Bucket Permissions

Now we have to configure bucket policy such that this will receive permissions from access point policy on the bucket/objects.


    // bucket-policy.json
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": "*",
          "Action": "s3:*",
          "Resource": [
            "arn:aws:s3:::my-bucket",
            "arn:aws:s3:::my-bucket/*"
          ],
          "Condition": {
            "StringEquals": {
              "s3:DataAccessPointArn": "arn:aws:s3:ap-south-1:9848032919:accesspoint/my-access-point"
            }
          }
        }
      ]
    }

There is one condition called DataAccessPointArn which identifies that access is coming from which accesspoint.

Now this policy allows permissions from accesspoint.

Uploading this bucket policy ..

 

    aws s3api put-bucket-policy --bucket my-bucket --policy file://bucket-policy.json

Now If I try to access the objects again as a KGF user …

I got the result ..

Now that we have allowed KGF user using access point , now our mini goal is to allow users to access the bucket/objects only via access point.

5. Allowing traffic only through access point.

As you guessed we have to change bucket policy to do that.


    //bucket-policy.json
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Deny",
          "Principal": "*",
          "Action": "s3:*",
          "Resource": [
            "arn:aws:s3:::my-bucket",
            "arn:aws:s3:::my-bucket/*"
          ],
          "Condition": {
            "StringNotEquals": {
              "s3:DataAccessPointArn": "arn:aws:s3:ap-south-1:9848032919:accesspoint/my-access-point"
            }
          }
        },
        {
          "Effect": "Allow",
          "Principal": "*",
          "Action": "s3:*",
           "Resource": [
            "arn:aws:s3:::my-bucket",
            "arn:aws:s3:::my-bucket/*"
          ],
          "Condition": {
            "StringEquals": {
              "s3:DataAccessPointArn": "arn:aws:s3:ap-south-1:9848032919:accesspoint/my-access-point"
            }
          }
        }
      ]
    }

Don’t get panic I am here to explain this.

1st rule is to deny every traffic unless it is made through access point .

2nd rule is to allow traffic from access point .

Seems Confusing right ? Deny rules has more precedence than allow rules, so deny rule here in bucket policy ,denies every one even access point permissions and over writes the access point policy , and we are again allowing permissions to flow from access point .

Uploading this bucket policy ..


     aws s3api put-bucket-policy --bucket my-bucket --policy file://bucket-policy.json

When I access as KGF user …

This worked ..

OK but lets try accessing the objects in the bucket using normal S3 URI instead of access point .

I got explicit deny error and that’s worked hurray!!!

6.Cross Account Access

Now we have almost completed our requirement, next step is we have to allow user from the other account to access objects in the bucket in our account .

If you give arn of the user in the other account as a Principal in the access point policy, only this is not enough to do so. The User in the other account must have IAM permissions to access the s3 bucket as well as this access point in our account..

IAM policy to access S3 in other account :


    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Statement1",
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket",
                    "s3:GetObject"
                ],
                "Resource": [
                    "arn:aws:s3:::my-bucket",
                    "arn:aws:s3:::my-bucket/*"
                ]
            }
        ]
    }

As the bucket names are unique that’s why no Account IDs in resource .

IAM policy to access the access point

 

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Sid": "Statement1",
    "Effect": "Allow",
    "Action": [
    "s3:*"
    ],
     "Resource": [
                  "arn:aws:s3:ap-south-1:9848032919:accesspoint/my-access-point",
                  "arn:aws:s3:ap-south-1:9848032919:accesspoint/my-access-point/object/*"    
                ]
    }
    ]
    }

Now if the user has these 2 IAM Policies ,then after adding user arn as the Principal in the access point policy.

I have created user named Ali in the other account and also Ali has 2 IAM policies set.


    {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": { "AWS": 
                         ["arn:aws:iam::9848032919:user/KGF",
                           "arn:aws:iam::9848022338:user/Ali" ] 
                             },
            "Action": [
              "s3:GetObject",
              "s3:ListBucket" 
            ],
            "Resource": [
              "arn:aws:s3:ap-south-1:9848032919:accesspoint/my-access-point",
              "arn:aws:s3:ap-south-1:9848032919:accesspoint/my-access-point/object/*"
            ]
          }
        ]
      }

When I try to access the bucket as Ali user…

I named Ali user as Goutham-Ali on my CLI . This works!!!

OK If I try to access using Normal S3 URI

This will not work..

This brings us to end of this article.

That’s it we have allowed access to the bucket/objects only using access point and also allowed cross account access using access point .

Thanks for reading my blog, leave a like and comment if you read something useful, have a great day.