minnogit

Posted on Oct 27 AI-assisted

Access control and cost monitoring for LLM models on AWS Bedrock

#aws #bedrock #llm #devops

I recently created an IAM user for an Android app that uses LLM models on Amazon Bedrock. I wanted two things: the app should be able to call only the models I picked, and I wanted to know at any time how much it was costing. This article describes how I set up both, and where the AWS documentation corrected a wrong idea I had.

The IAM user and the policy

The user has a single policy, which allows invoking four models in eu-west-1 and nothing else:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BedrockInvokeSpecificModels",
            "Effect": "Allow",
            "Action": [
                "bedrock:InvokeModel",
                "bedrock:InvokeModelWithResponseStream"
            ],
            "Resource": [
                "arn:aws:bedrock:eu-west-1::foundation-model/qwen.qwen3-vl-235b-a22b",
                "arn:aws:bedrock:eu-west-1::foundation-model/google.gemma-3-4b-it",
                "arn:aws:bedrock:eu-west-1::foundation-model/google.gemma-3-12b-it",
                "arn:aws:bedrock:eu-west-1::foundation-model/google.gemma-3-27b-it"
            ]
        },
        {
            "Sid": "BedrockGetModelInfo",
            "Effect": "Allow",
            "Action": [
                "bedrock:GetFoundationModel"
            ],
            "Resource": [
                "arn:aws:bedrock:eu-west-1::foundation-model/qwen.qwen3-vl-235b-a22b",
                "arn:aws:bedrock:eu-west-1::foundation-model/google.gemma-3-4b-it",
                "arn:aws:bedrock:eu-west-1::foundation-model/google.gemma-3-12b-it",
                "arn:aws:bedrock:eu-west-1::foundation-model/google.gemma-3-27b-it"
            ]
        }
    ]
}

Each model is listed by ARN, so if I want to add one in the future I have to do it explicitly. You could use a wildcard to allow all of them, but for an app installed on a device I prefer a closed list.

The tag on the user

For costs, I added a tag to the user: key bedrock, value tablet. You can do it from the IAM console or from the CLI:

aws iam tag-user --user-name <user-name> --tags Key=bedrock,Value=tablet

A second IAM user, which I use for something else, has the same bedrock key with the value inno. This way the same tag groups different users, and the values tell them apart.

My initial plan was a bit different. I thought AWS couldn't attribute costs to an IAM user, and that the tag was the way around that. That is no longer true: Bedrock records the identity behind each call (IAM user or role), and that data reaches Cost Explorer without changing the policy or the app's code. For CUR 2.0 reports you need a new export instead, created in Data Exports with the option "Include caller identity (IAM principal) allocation data", which adds the line_item_iam_principal column. For this case Cost Explorer is enough, and I didn't create the export. The tag is still useful for another reason: it adds a grouping dimension, for example by team, application or device, that you can use in filters and budgets.

Activating the tag as a cost allocation tag

A tag on an IAM user doesn't show up in billing data by itself. It has to be activated:

Open Billing and Cost Management, Cost allocation tags section.
Filter by type IAM principal.
Select the bedrock tag and press Activate.

Activation is done from the organization's management account, or from a standalone account. From a member account the console returns the error Cost allocation tags are managed at the payer account level.

In cost data the tag is called iamPrincipal/bedrock. There are three limits to know before expecting numbers:

the tag only appears after the user has made at least one call to Bedrock;
activation is not retroactive, so earlier costs are not tagged;
the tag can take up to 24 hours to appear in Cost allocation tags after the first call and up to another 24 to activate, and cost data can then arrive with a further delay of up to 24 hours.

Looking at costs in Cost Explorer

Once the data has arrived, in Cost Explorer I filter on the Amazon Bedrock service and group by the iamPrincipal/bedrock tag. The finest level of detail is a cost per usage type per day, so you can't see the cost of a single request. For that you need Bedrock's invocation logs, which are a different path and which I haven't enabled.

The budget

To avoid discovering an out-of-scale bill at the end of the month, I created a plan in AWS Budgets. In my case, spending on LLM calls is less predictable than spending on the other AWS resources I use, which is why I need the budget here more than elsewhere. I chose a custom cost budget with these parameters:

recurring monthly period, with a fixed amount;
scope by tag: the iamPrincipal/bedrock tag, with the values of the IAM users to watch (in my case tablet and inno);
service filter: Amazon Bedrock;
an alert at 80% of the amount, sent by email. It can also go to Slack, but I haven't set that up.

The tag filter also offers a value for untagged costs. That is what the service filter is for: without it, I expect that value to also count the untagged costs of the account's other services, though I haven't verified it. With both filters, the budget should count all Bedrock spending, tagged or not.

If the untagged value is left out, the budget only sees tagged spending. An IAM user calling Bedrock without the tag stays out of the count and doesn't trigger the alert. Either way, it's worth checking Cost Explorer from time to time for untagged Bedrock spending and, if there is any, finding out which user generates it and tagging it.

A budget only alerts: it doesn't stop the spending. If you need a real block, you have to set up an action tied to the threshold, or revoke the user's credentials.

Things to keep in mind

If calls go through a gateway (LiteLLM, API Gateway or a homemade service), the cost is attributed to the gateway's role and not to the end user. To tell users apart, you need to pass session tags when the gateway assumes the role.
An IAM user's credentials are long-lived keys. If the Android app contains them, anyone who extracts the APK can use them: consider temporary credentials, which offer the same cost attribution options.
Attribution by identity can be combined with application inference profiles if you also need a split by application, but it requires changing the policy: you have to add the profiles' ARNs, as well as those of the models in the regions involved.

Summary of the steps

Create an IAM user with a policy limited to the chosen models.
Add a tag to the user (here bedrock=tablet).
Activate the tag in Cost allocation tags, type IAM principal.
Wait for the first call and up to a couple of days to see the data in Cost Explorer.
Create a monthly budget filtered on the tag.

DEV Community