KETTLE Hopefully you haven't had reason to notice yet, but there's a rising problem with AI services on Google Cloud, AWS, and other platforms sticking their customers with bills in the tens of thousands of dollars. This week's episode of the Kettle focuses on two such stories that The Register published this week, one concerning Google and another involving AWS. In both cases, cloud customers using AI incurred massive bills without any prior notification from their provider and not a lot of help to resolve the matter with any sense of urgency. Tune in to this week's episode to hear host Brandon Vigliarolo chat with O'Ryan Johnson and Richard Speed about their stories, what's causing these massive bills, and how you can avoid a similar situation at your own organization. You can listen to The Kettle here, as well as on Spotify and Apple Music, or read the transcript of the latest episode below. It's been lightly edited for clarity. Brandon (00:01) Hello everyone and welcome back to another episode of The Register's Kettle podcast. I'm Reg reporter Brandon Vigliarolo, and this week I'm joined by my colleagues Richard Speed and Kettle newcomer O'Ryan Johnson to talk about a recent spike in cloud AI API abuse that's sticking customers with some massive charges. We're talking tens of thousands of dollars that Google is seeming to...try hard not to refund. Guys, thanks for coming on. O'Ryan Johnson (00:29) Great to be here. Brandon (00:30) And O' Ryan, welcome again to your first Kettle episode. Glad to have you here. So in this case, this one is primarily based on an exclusive you published this week about compromised Google Cloud API keys. And from what I read, it seems like cyber criminals are using those keys to run all the AI inference they want on most expensive models that Google has without paying a dime. So walk me through what exactly this story's about. O'Ryan Johnson (00:33) So there were a couple parts of this. One is the API abuse. But then there was this policy by Google that kind of threw gasoline on the fire. So if you're a developer and you've created an API key for your projects, if your project uses Maps, you'll create an API key. And for years, the advice from Google was put that API key on the front end of that, make it public so that when users are using your site, it links back to your project. The problem was a couple years ago, they allowed those API keys, if they were configured correctly, to also access Gemini. And a lot of folks who were early adopters of AI went in and said, okay, I want to use Gemini with my project. And not really connecting the dots that their API key on the front end that was publicly available would now also allow anybody to inference Google's Gemini platform. And it wasn't a big deal, I think, for a lot of years because I don't think the platform was really that amazing. Brandon (02:01) Yeah, because you said this is a three year old change, right? O'Ryan Johnson (02:22) But recently...Nano Banana and the Veo 3 models came out. And that's when I think we started to see a lot of this. This great security company named Truffle wrote something about this in February saying, look, be careful because if you've put your API key out according to Google's instructions, and if you've also been working with Gemini models, there's a chance that you may have inadvertently opened up your API key to anybody to be able to inference [Veo] and NanoBanana to their heart's content. Brandon (02:40) And specifically a Maps API key, right? Okay, O'Ryan Johnson (02:51) Correct. Which again was, was Google had told everybody for quite a while was safe. And so, what happened kind of inevitably is folks were bad actors were in fact using that for for those purposes So you'd have these you know sort of like horror stories of waking up in the morning and seeing your Google account Which you maybe you never spent more than fifty dollars a month, all of a sudden you have a $3,000 bill, $5,000 bill. I talked to a guy who got a notification from his credit card company that "Hey, we're basically we're shutting off your account because you spent too much," and he's like "What the hell is going on?" And as he's in there trying to figure it out he sees the bill keeps going up.... Brandon (03:26) ...I think you mentioned basically that where this is, how you figure this out, is kind of buried, right? It's hard to find, right? So as he's looking, trying to frantically figure out what's happening, more charges are being added. I couldn't imagine waking up in the morning to that kind of scenario. O'Ryan Johnson (03:36) It's a rough, rough way to start the day. It's really tough. Brandon (03:50) So that's the first part, right? So what's the second part then? O'Ryan Johnson (03:52) So, right, that's the first part. The second part is that, you know, this happened to people who had spending caps in place. And Google has only recently put spending caps in place, but they're really loose caps. I talked to a developer in Australia who said, "Look, I put a $250 spending cap in place. And when I woke up, I had a $10,000 bill." ...And he said, "When I was going to going through afterwards, I looked and I said my spending tier was at the $100,000 limit. And I said, how does it happen?" Well, if you look like Google was actually very upfront about this. In March, they put out a blog and said, "Hey, we're going to help you out. If you've only got a $250 spending cap, if you spent $1,000 in the lifetime of your account and you've been a Google member for 30 days or more, a Google Cloud developer for 30 days or more, you can spend $100,000." Brandon (04:47) And there's no notification to the user accounts that this is being done? O'Ryan Johnson (04:50) Except for the emails that say this is how much you owe us, which is all after the fact. Brandon (04:57) And if you're less than 30 days, right, it's moving to tier two is, I think, what's the cap on that? O'Ryan Johnson (05:01) Two thousand dollars. Brandon (05:04) But even then, it's spend a hundred bucks in the lifetime of your account? O'Ryan Johnson (05:06) A hundred dollars and be three days old, and Google will give you a $2,000 cap to spend. Those are the most generous terms – you guys have been around IT for years, what distributor would ever give you terms like that like if you went to TD Synnex or if you went to Ingram Micro and said, "Hey, I'm 30 days old and I've spent $1,000. I would like $100,000 in credit with you." They would laugh . Brandon (05:13) They would laugh you out of the office and then maybe close your account. O'Ryan Johnson (05:37) Yeah, even the best distributor in the world is not going to give you those terms, but Google's opened that up. And then of course the problem is trying to get that money ... trying to get your account restored. like in two of the cases, the money had already been spent. So the credit cards, one was $17,000, one was $10,000. The money was already out of their account and they have this project. If they charge it back, they're afraid that Google's going to shut down their project and delete it. If they stick with the bill, then they're stuck with this debt that is obviously outside the bounds of any budget that they had set for their Google Cloud project. Brandon (06:12) Yeah, for a small developer that can be devastating. O'Ryan Johnson (06:15) Right. Right. So Google, though, we do have an update coming today. Google has refunded the two people we talked about and looked in their account. It looks like they're kind of going after this with more accounts too, based on what I've talked to with Google, they're going to look at a lot more of these issues...This didn't come to me in a vacuum. I mean, this this was on these posts have been kind of flooding Reddit. If you go to the Google Cloud subreddit there, you you pretty much don't go, there's two or three a day that are popping up saying, "Hey, my gosh, I've got $10,000. I got $7,000 in bills. Like I only ever spent, you know, $50 with these folks. How am I getting these bills?" Brandon (06:57) Right, so it's kind of a two-part story here. The automatic tier upgrades are obviously a problem, but are all these cases that you're seeing, are they tied back to the Truffle notice? I mean, these are all Maps API keys? O'Ryan Johnson (07:02) Not all of them. Some people say like, "Look, I never put my API key out publicly." And I talked to a guy yesterday who said, "Look, my API key has been hidden from everybody. I think I got brute forced." ....I don't possibility or the probability of being able to brute force an API key, they're huge, long chains of numbers and texts. Probably not impossible...But this guy, his bill was $127,000, which is just a huge, huge amount. Brandon (07:40) God that is so that is so much money that's ridiculous. Ten thousand dollars is bad enough add another zero to that and oh my God. O'Ryan Johnson (07:51) That's rough. Fortunately, he caught it before...That bill only exists with Google. Fortunately, the good side is, it's not in his credit card. So he doesn't have to try to pay that back. The bad news is, his Google project is looking at a possible deletion if he can't convince Google that, "Look, this wasn't me, this was really somebody else who brute-forced my API." Brandon (08:15) I'm guessing proving that is pretty difficult. O'Ryan Johnson (08:17) Well it's difficult, what makes it difficult is he no longer has access to the logs because he hasn't paid the account, so now he has to rely on somebody at Google to go through those logs and make his case for him. Brandon (08:34) When there's $100,000 on the line. O'Ryan Johnson (08:36) When there's 127 on the line. That's a gamble. That's a gamble. Brandon (08:43) So this is bad enough, but as I understand, Richard, Google's not the only company being a bit shifty with their AI billing. You wrote a story this week about an AWS customer who was billed $30,000 despite supposedly having a setting enabled to prevent this. So what's this all about? Richard Speed (08:50) It's kind of almost a cautionary tale in some ways. Again, we've talked about Google, there's also, this is AWS. And this is a user who was using AWS Bedrock. He wanted to take Claude Opus out for a spin, try it out. He had some startup credits fired by Activate. All great. Now he was using a tool called the AWS Cost Anomaly Detection Tool. What that does, that actually sends you alerts if you're doing some odd things and your account is incurring additional costs, and as well as using AI machine learning, you can also set some custom thresholds... "If I spend more than this then stop or shout at me or whatever Brandon (09:39) Yeah, cut me off. Yeah. Richard Speed (09:45) So he thought, "Great, I've got that, what could possibly go wrong?" And so he began to use his AWS Bedrock and no alerts were fired, all was good until about a month after he began using it he got a bill for $30,000 or $38,000 through where he was expecting hundreds. And the reason being was that AWS Bedrock apparently bills through AWS Marketplace, and that is not compatible with the cost anomaly detection. Brandon (10:06) So Marketplace is where you can pick up third party integrations for AWS, right? Richard Speed (10:17) Right, and that's where AWS Bedrock was being billed, was basically invoiced through. And to be completely fair to AWS, that is documented. It is in the documentation, "This will happen." So, hence the cautionary tale aspect. But again, I've had a few people say, actually it's pretty unintuitive, this. You kind of would assume it's being caught and it wasn't caught. And so this is gone through. Now, unfortunately, at the moment, I don't think there is the happy ending about a refund. If and when I get more information, I willupdate. But the cautionary tale aspect is, I've heard from somebody else who said, yeah, similar sorts of things can happen. So I tend to go through directly through the AI provider. In this case, it's Anthropic. And there again, you can put limits in place. And those limits did save this particular person from a $50,000 mistake. And he only ended up paying $50 because he'd accidentally turned on a thing which enabled a lot more invoicing to happen, and of course it was stopped before it got out control. Brandon (11:25) I'm assuming a lot of customers, with the way they have their architectures and their infrastructure set up and their various providers, I mean, is it going to be simple for a lot of businesses to say, I'm going to skip AWS and go straight to the AI company itself? I mean, that seems like it might work in some cases, right? But a lot of people are going to be trying to integrate these. And so they're going to have to go through things. So does Cost Anomaly Detection function only with first-party Amazon products then basically? Anything in the Marketplace that you're pulling from a third-party provider doesn't get included in this? Richard Speed (11:59) Yeah, I believe so. Yeah, it's just through AWS services except for Marketplace stuff. But there are other checks and things in place in AWS. It's just in this instance, the expectation was if I'm using Cost Anomaly Detection, it should stop me running up a massive invoice or running up a massive bill using AWS Bedrock. In this case, it didn't. It was completely silent as the thousands and thousands and thousands began to rack up on the account. Brandon (12:05) And even, I think you wrote, even when his credits ran out. Like, he ran out of credits and switched to cash billing and there was no notice. Richard Speed (12:29) Exactly. It suddenly went from from credits to cash billing again with no notification or warning. It just happened. And so again, his account began to incur these charges. And so he didn't realize until the invoice came through. "Oh, my goodness me. How terrifying is this?" As as Ryan said, it's quite a shock when when you're used to a small amount per month and then suddenly a massive invoice comes through. O'Ryan Johnson (12:53) One thing that is kind of universal across this that one of these users pointed out, is that the most frustrating part is that they have the information. They can see what you're doing in your account and they don't stop it. All this information that we're talking about, whether it's your usage, whether it's your billing, all that stuff is within the four walls of, whether it's Google or AWS and they, whether it's intentionally or unintentionally – we live in this era where everybody talks about immaculate orchestration across all their environments, right? Like, I mean, if you're in SaaS, that's all you hear is about how amazing and perfect their SaaS products are. And we just don't see that in practice. You don't see that orchestration, and you certainly don't see it if it can ever give the user an advantage, or if it can ever give the user the ability to control how much they spend. Like if a user could shut off – if there was a notification that came in and said, "Hey, did you know that you're on Veo right now and you're generating videos? Would you like to shut that off?" Think about your credit card company. If I go one county over and I spend $10 at a Target, I'll get an alert from my card company. "Hey, are you sure?" Are you telling me, Google and AWS, that you can't do that? Like, don't give me that. I mean, this reminds me like when the banks in the US had overdraft fees, they used to – they could see how much money you had in your account. They would gladly let you spend much more than that so that they could fine you for every transaction. And so it was very similar. You'd open up your bank account and see like, I'm $800 in debt. So that was eventually determined to be, hey, that's an aggressive, that's not a good policy. We shouldn't allow people to do that. And I just wonder if, I wonder if there's gonna be some sort of trade regulation that kicks in on this. Brandon (14:26) I mean, it almost feels like there has to be. What we have in these two stories this week is multiple cloud platforms making their AI billing usage or usage billing so convoluted that a non-trivial number of customers are seeing their bill skyrocket, whether both due to cybercrime or simply the fact that Cost Anomaly Detection on AWS isn't very well-defined on the Marketplace, right? You're seeing multiple companies this is happening to, right? Again, O'Ryan, you kind of went right to the, the conspiracy theory, but that's where my mind goes too, this seems really convenient. Google's move in March. All these kinds of things are very well timed to ensure that companies that are adopting AI are being left with this ambiguous billing situation. Richard Speed (15:35) I mean, if only there was a tool that could spot strange patterns in data and frames. I mean, what would that look like? [Laughter.] Brandon (15:43) Yeah, I don't know. ⁓ There's no way, there's no way that ⁓ Google and AWS don't see this usage or can't monitor it. Can't pop a large language model on there to keep an eye out for ⁓ unusual billing and notify people. Like you said, if you never use [Veo] or never use NanoBanana and all of a sudden your account's racking up thousands of dollars of charges on it, Google should probably say, "Hey, is this you?" Right? Like, you know, that would be, I would hope that would happen. Right? You know, it's like you said, right? Your bank, Target will know, or your credit card company will notify if you spend things a county over. Right? If I try to log into a video game online from a different IP address, it locks me out and makes me me approve it. Right? Like this is not a complicated technology here. O'Ryan Johnson (16:32) No, think about the user agreements that we have like with all of our subscriptions like you know like Netflix. If my kid tries to log into my Netflix from where they live, they can't, and I get these notifications from Netflix, "Hey do you want to add somebody on your account?" Like don't tell me that you can't do that, Google. And Google actually says that they hat between the usage and the spend, they're better than AWS when it comes to being able to spot this. But it's like, it's still something like 28 days to be able to reconcile usage with spend. And that just does not make any amount of sense. Brandon (17:16) It takes Google 28 days? O'Ryan Johnson (17:18) They're pushing people into these products. They're pushing, they want you to use these products. They want developers to, they want to be able to say, we have X number of developers who are using this. We have X number of spend. All of those hijacked API keys are inevitably helping marketing for Gemini. Just through sheer usage numbers, through sheer revenue and dollar spend, that drives a narrative that they can then, you on the quarterly earnings call say, "Hey, look at all these people using our product. Look at all the spend on [Veo]. Look at all the spend on Banana." come on, you guys, you got to make it fair for the rest of us, man. Brandon (17:59) I'm just gonna toss it allegedly in there before Google comes after us, right? You know. We don't know for sure that this is what they're planning, but it sure seems, the ducks do line up, right? So guys, are you familiar? Do you know, are any other cloud platforms...are there similar issues on Azure, on other platforms? Have you heard anything? Or does this seem to be mainly confined right now to Google and AWS? Richard Speed (18:11) There have been some issues on Azure. I read a piece, oh crikey, several weeks, maybe even months ago now, regarding a similar thing to what's happened with AWS with a user who had, he hadn't realized that his startup credits didn't count towards AI usage. And then he found himself hit with a massive invoice because again, Microsoft just quietly said, "Yeah, sure. You want that service? No problem. Here you go. Use it." And so he used it and then the huge invoice came through. I think... I think it's important to point out that these companies, they're not doing anything wrong legally. Ethically, I'm with O'Ryan, they should be warning you to say, "Hey, you know, you're spending way more now than you ever used to before. These services that you've never used before, are you sure you want to be doing that? Are you sure about that?" Brandon (18:51) I was talking to my wife about Google before we started the podcast, right? Because when we were talking about the topic for this week, and I think Matt, our editor in chief said, "AI overage charges." I was like, "What? This is going to be a boring episode." And then I got to actually reading these stories and I'm like, "Oh my God, this is really interesting." My wife's like, "Surely this is illegal." I'm like, "I don't know, if it's in the terms of service, right? You know? Yeah." O'Ryan Johnson (19:23) It's like the South Park episode. Richard Speed (xx:xx) I think another aspect of this is there's a perception that AI services are inexpensive and you won't run up these massive costs. One thing I've come across a few times are companies saying, "Hey, we can increase the productivity of our staff enormously because we can roll out these AI tools and our employees can use them and they'll be massively more productive and it'll be great." They're forgetting that of course there is a cost to that. And I think what we're seeing here are people hitting these costs. So I think that the message has got to be, you need to be – I mean, until these companies actually put in warnings to say, know, perhaps make it very clear how much this stuff is really costing, I think you need to be aware that this isn't a free service, you know, it's going to be paid for somehow. Brandon (xx:xx) I guess that's kind of the big warning to businesses, right? Or AI users, anyone who's using AI in the cloud in general, It's like these things are not free. Yeah, sure, you can use ChatGPT for free if you're, you know, some random person logging into the website. But if you want to go enterprise with this or use it in any kind of business capacity, it's going to cost you money and potentially a lot of it. So Richard, you said that it looks like the AWS user might be a little bit hosed on getting a refund. Do you know is Amazon – did you talk to Amazon for the story? Do they have any intention to change the marketplace versus non-marketplace CAD policy? Richard Speed (xx:xx) They did respond, and at the moment there's no plans to change it. O'Ryan Johnson (xx:xx) Google is also, they're sticking by their automatic tier upgrades. They like the flexibility that it gives to developers. Flexibility, of course, meaning that developers can spend a lot more than they initially wanted to, or agreed to. Brandon (xx:xx) It's a very one-sided flexibility, really, when you think about it. O'Ryan Johnson (xx:xx) In fairness, we are kind of helping at least notify people that this could happen. This is something that is really happening to people and their bills really do become five-figure, in some cases six-figure bills at the end of the month through no intention of their own. Brandon (xx:xx) Yeah, so I guess basically the big, yeah, like we said, the big takeaway for business AI customers is to just really watch that billing, be sure that whatever system you have in place to prevent overages is actually doing its job, and hide those API keys. Well, like we said, guess this is just a cautionary tale, you know, to watch that billing. So if this keeps happening, we are definitely going to be talking about it and writing about it again. And we hope that you will tune in on a future episode of The Kettle to find out more. ®
