hckrnws

In Google AI Studio, Google documentation encourages to deploy vibecoded apps with an open proxy that allow equivalent AI billing abuse - giving the impression that the API key were secure because it is behind a proxy. Even an app with 0 AI features exposes dollars-per-query video models unless the key is manually scoped. Vulnerable apps (all apps deployed from AI Studio) are easily found by searching Google, Twitter or Hacker News. https://github.com/qudent/qudent.github.io/blob/master/_post...

[flagged]

I think the fact that it is not possible to put hard spending caps on API keys might be ruled illegal by some EU court soon enough, at least when they sell to consumers (given the explosion of vibecoding end-users making some apps). When I use OpenAI, Openrouter etc., I can put 10 $ on my API key, and when the key leaks, someone can use these 10 $ and that's it. With Google, there is no way to do that - there are extremely complicated "billing alerts" https://firebase.google.com/docs/projects/billing/advanced-b... , but these are time-delayed e-mails and there is no out of the box way to do the straightforward thing, which is to actually turn off the tap automatically once a budget is spent. The only native way to set a limit enforced immediately is by rate limiting - but I didn't see params which made it safe while usable in my case.

(a legal angle might be the Unfair Contract Terms Directive in the EU, though plenty of individual countries have their own laws that may apply to my understanding. A quite equivalent situation were the "bill shock" situations for mobile phone users, where people went on vacation and arrived home to an outrageously high roaming bill that they didn't understand they incurred. This is also limited today in the EU; by law, the service must be stopped after a certain charge is incurred)

I don't know if its still like this but around 1 year ago I set a spending limit for an OpenAI api key but it turns out its not a true limit. I spent 80$ on a 20$ limited key in the matter of minutes due to some bad code I wrote causing a looped loop.

I still had to pay it or else I wouldn't have been able to use my account.

> or else I wouldn't have been able to use my account.

Would that have been so bad? The world might be a better place if people stopping pouring money into that cesspit.

By continue to use their services, you're encouraging the anti-consumer tactics you're complaining about.

> When I use OpenAI, Openrouter etc., I can put 10 $ on my API key, and when the key leaks, someone can use these 10 $ and that's it.

On that note, I'll just mention that I had discovered over the last while that when you prepay $10 into your Anthropic account, either directly, or via the newer "Extra usage" in subscription plans, and then use Claude Code, they will repeatedly overbill you, putting you into a negative balance. I actually complained and they told me that they allow the "final query" to complete rather than cutting it off mid-process, which is of course silly, because Claude Code is typically used for long sessions, where the benefit of being cut off 52% into the task rather than 51% into it is essentially meaningless.

I ended up paying for these so far, but would hope that someone with more free time sues them on it.

I'm spitballing here, but I suspect that (same with AWS) google uses post processing for billing, they run a job that scrapes the states THEN bills you for that. instead of the major AI companies are checking billing every API request coming in.

Yes, you are on the money. A cloud service provider needs to maintain reliability first and foremost, which means they won't have a runtime dependency on their billing system.

This means that billing happens asynchronously. You may use queues, you may do batching, etc. But you won't have a realtime view of the costs

>they won't have a runtime dependency on their billing system

Well, that makes sense in principle, but they obviously do have some billing check that prevents me from making additional requests after that "final query". And they definitely have some check to prevent me from overutilizing my quota when I have an active monthly subscription. So whatever it is that they need to do, when I prepay $x, I'm not ok with them charging me more than that (or I would have prepaid more). It's up to them to figure this out and/or absorb the costs.

> they obviously do have some billing check that prevents me from making additional requests after that "final query"

No they don't actually! They try to get close, but it's not guaranteed (for example, make that "final query" to two different regions concurrently).

Now, they could stand up a separate system with a guaranteed fixed cost, but few people want that and the cost would be higher, so it wouldn't make the money back.

You can do it on your end though: run every request sequentially through a service and track your own usage, stopping when reaching your limit.

They do have a billing check, but that check is looking at "eventually consistent" billing data which could have arbitrary delays or be checked out-of-order compared to how it occurred IRL. This is a strategy that's typically fine when the margin of over-billing is small, maybe 1% or less. I take it from your description that the actual over-billing is more like dozens of dollars, potentially more than single-digit percentages on top of the subscription price. Here's hoping they tighten up metering <> billing.

Then the right thing to do from a consumer standpoint is to factor that overbilling into their upfront pricing, rather than surprising people with bills that they were led not to expect.

let's hope it happens soon, I'm pretty sick of this reality where companies get to charge you whatever they want and it's designed to always be your fault

You're configuring something that costs money (electricity, hardware, real estate) to provide. Either it's "pay as you go" or you have a flat rate and a cap.

If you have a cap and then your thing hits the front page and suddenly has 10000% more legitimate traffic than usual, and you want the legitimate traffic, they're going to get an error page instead of what you want. If there is no cap, you're going to get a large bill. People hate both of those things and will complain regardless of which one actually happens.

The main thing Google is screwing up here is not giving you the choice between them.

Imagine the outrage here, when a company credit card expires and the cloud provider terminates all their instances, deletes all your storage and blob backups?

That does happen, it's just usually not when the card expires, but when the follow up billing emails get ignored for some period.

This is one of the reasons people have suggested using a different provider for backups.

it's not an either or, they can easily let me configure any kind of behavior that I want. No cap, a hard cap, a soft cap, a cap that I program with a python script, a cap where I throttle, a cap where I opt in to deleting certain machines to save money. It can all be done. People are complaining because obvious features are not provided. People would not be complaining if they had all the options that we needed to control how to scale resources in response to load, not just technical load but also financial load.

You can already do any of those things in your own code when making the API requests. The issue here is, if you unintentionally try to make a billion expensive requests or allow someone else to do it against your account, do you want them to automatically turn off your stuff or do you want the bill that comes if they don't?

You're oversimplifying the problem in the other direction. Fine-grained scriptability of hard limits would bump up against all of the thorny distributed systems problems. But I do agree that fixing the simple cases is straightforward - maximum spend rates per instant and per unit of time (eg per minute, hour, day, month). Providers would shoulder the small costs from the slightly-leaky assumptions they have to make to implement those limits, and users can then operate within that framework to optimize what they want on a best-effort basis (eg a script that responds within a minute to explicitly scale resources, or a human-in-the-loop notification cycle over the course of hours so that you have the possibility to say "actually this is popularity traffic that I really do want to pay for, etc).

The main thing Google is screwing up is that if my API key somehow leaks and I end up with extremely out of line billing at Microsoft, I will be on the phone with a customer representative as soon as we or they notice something weird happening and a solution will be found.

Google will probably have me go through five bots and if, by some kind of miracle, I manage to have a human on the phone, they will probably explain to me that I should have read the third paragraph of the fourth page of the self service doc and it's obviously my fault.

> I'm pretty sick of this reality where companies get to charge you whatever they want and it's designed to always be your fault

But have you considered it from the companies POV? Charging whatever you like and its always the customers fault is a pretty sweet deal. Up next in the innovation pipeline is charging customers extra fees for something or other. It'll be great!

This is just the utility model. It's nothing particularly nefarious. Consider what your electric utility, your water utility, etc. do. If you use more, you pay more. If someone comes around and hooks up a garden hose to your outside faucet and steals your water, or plugs an extension cord into your outside outlet and steals your electricity, you still pay. Unless you can catch the thief and make him pay.

Funny enough, the utility business broadly wants to move away from this model to more of a cap-based prepaid model. Where I live, to get on the standard payment system may require a quite hefty deposit up front, but the prepaid payment option does not. I get the impression that, if not for customer sentiment and inertia, this would be the default option.

Why should I care about the companies POV? The company always wants to rat fuck everyone to make money. The company should be legally compelled to care about the customer because that's the only way these things change.

Healthy, even.

I think the term you are looking for is "negligence".

But not in the causal sense of the word but in the legal "the company didn't folly the legal required base line of acting with due diligence".

In general companies are required to act with diligence, this is also e.g. where punitive damages come in to produce a insensitive to companies to act with diligence or they might need to pay far above the actual damages done.

This is also why in some countries for negligence the executives related to the negligent decisions up to the CEO can be hold _personally_ liable. (Through mostly wrt. cases of negligence where people got physically harmed/died; And mostly as an alternative approach to keeping companies diligent, i.e. instead of punitive damages.).

The main problem is that in many cases companies do wriggle their way out of it with a mixture of "make pretend" diligence, lawyer nonsense dragging thing out and early settlements.

Upvoted.

Not illegal, but it should make enforcing payment illegal.

> Downvoted for asking an honest question.

If you put in "surely" and people think it's quite wrong then they might downvote. It's not personal.

Not illegal enough to worry about. nothing a peace board donation can’t fix.

Sure, after 6 years in court you may get a settlement, 95% of which will go towards paying your legal fees.

> 95% of which will go towards paying your legal fees

laughs in European

I laughed. No in europe when you win a case like this the judge usually forces the losing party to pay the legal expenses of the winner. Especially if the losing party is a big corporation.

It's the same in the US

It is not. Legal fees are rarely awarded in the U.S.

I should have said if you recover it in your damages, which every competent attorney will push for.

Legal fees are not something you are usually legally entitled to.

Your attorney can push for whatever illegal thing they can think of, it doesn't mean you will get it.

> Your attorney can push for whatever illegal thing they can think of, it doesn't mean you will get it.

It is not illegal to include legal fees in damages.

Under what statute is it illegal to request legal fees?

Requesting and being granted legal fees are two different things.

The default "American rule" is that each party pays their own legal fees, unless there is a relevant fee shifting rule.

It’s possibly civil, but I don’t see how this type of negligence would be breaking a law. If it was illegal, a massive number of independent consultants would be serving prison sentences. I’m not sure how that makes anything better though I guess a lot of people think rage is fun.

Civil law is law, and breaking it is illegal. You seem to be misunderstanding it.

How dare you question a corporation's ability to make unlimited money?

Will someone think of the shareholders? /s

The headline really undersells the point and reads like clickbait. "Things were fine, then she turned the tables. Watch what happens next." I avoided even opening this article several times out of distaste for the headline. It should be something like "Google leaves your Gemini data vulnerable to non-secret API key exploit."

The headline states a plain fact that is critically important. It's not the writer's fault that the fact is outrageous.

I accused the headline of underselling, not overselling. So unsure why you read me to have blamed the writer for making outrageous claims...

> Leaked key blocking. They are defaulting to blocking API keys that are discovered as leaked and used with the Gemini API.

There are no "leaked" keys if google hasn't been calling them a secret.

They should ideally prevent all keys created before Gemini from accessing Gemini. It would be funny(though not surprising) if their leaked key "discovery" has false positives and starts blocking keys from Gemini.

Yeah its tremendously unclear how they can even recover from this. I think the most selective would be: they have to at minimum remove the Generative Language API grant from every API key that was created before it was released. But even that isn't a full fix, because there's definitely keys that were created after that API was released which accidentally got it. They might have to just blanket remove the Generative Language API grant from every API key ever issued.

This is going to break so many applications. No wonder they don't want to admit this is a problem. This is, like, whole-number percentage of Gemini traffic, level of fuck-up.

Jesus, and the keys leak cached context and Gemini uploads. This might be the worst security vulnerability Google has ever pushed to prod.

The Gemini API is not enabled by default, it has to be explicitly enabled for each project.

The problem here is that people create an API key for use X, then enable Gemini on the same project to do something else, not realizing that the old key now allows access to Gemini as well.

Takeaway: GCP projects are free and provide strong security boundaries, so use them liberally and never reuse them for anything public-facing.

Imagine enabling Maps, deploying it on your website, and then enabling Google Drive API and that key immediately providing the ability to store or read files. It didn't work like that for any other service, why should it work that way for Gemini.

Also, for APIs with quotas you have to be careful not to use multiple GCP projects for a single logical application, since those quotas are tracked per application, not per account. It is definitely not Google's intent that you should have one GCP project per service within a single logical application.

Really? I make multiple GCP projects per app. One project for the (eg) Maps API, one for Drive, one for Mail, one for $THING. Internal corp-services might have one project with a few APIs enabled - but for the client-app that we sell, there are many projects with one or two APIs enabled only.

If you ever have to enable public OAuth on such a project, you'll need to provide a list of all the API projects in use with the application, and Google Trust and Safety will pressure you to merge them together into a single GCP project. I've been through it.

You can do what you're describing but it's not the model Google is expecting you to use, and you shouldn't have to do that.

It seems what happened here is that some extremely overzealous PM, probably fueled by Google's insane push to maximize Gemini's usage, decided that the Gemini API on GCP should be default enabled to make it easier for people to deploy, either being unaware or intentionally overlooking the obvious security implications of doing so. It's a huge mistake.

> decided that the Gemini API on GCP should be default enabled to make it easier for people to deploy

Like deciding ATM cabinets should be default open to make it easier for people to withdraw cash.

No, there must be more behind this than overzealotry.

> It didn't work like that for any other service, why should it work that way for Gemini.

Artifical Intelligence service design and lack of human intelligence are highly correlated. Who'd have guessed??

Isn't there a limit to the number of projects you can make and then you have to ask support to increase it?

There is, yes. The rumor mill suggests that the default limit is 30.

At $DAYJOB, we had a (not very special) special arrangement with GCP, and I never heard of anyone who was unable to create a project in our company's orgs [0].

Given how Google never, ever wants to have a human do customer support, I expect a robot will quickly auto-approve requests for "number of projects" quota increases. I know that's how it worked at work.

[0] ...with the exception of errors caused by GCP flakiness and other malfunction, of course.

Can vouch, I put in a request for 20 projects extra which was approved in hours.

Many products using the Cloud APIs auto-create projects. I know of AI Studio and Google Script (including scripts embedded in Docs, Sheets, etc)

So many organizations have the IAM "Project creator" role assigned to everyone at the org level. I think it's even a default.

As long as you are over a certain spend. I started something for my own project and went to apply the recommended architecture, which does not work without a quota increase. As it was from a fresh account, the email was we won't look at this until you spend or pre spend so much money. Frankly, for a trail period when evaluating at prior enterprises, that would have made me just say no to their cloud. One expects that the recommended architecture can be deployed in the trial run without hoops.

I’m usually client side dev, and am an ex googler and very curious how this happened.

I can somewhat follow this line of thinking, it’s pretty intentional and clear what you’re doing when you flip on APIs in the Google cloud site.

But I can’t wrap my mind around what is an API key. All the Google cloud stuff I’ve done the last couple years involves a lot of security stuff and permissions (namely, using Gemini, of all things. The irony…).

Somewhat infamously, there’s a separate Gemini API specifically to get the easy API key based experience. I don’t understand how the concept of an easy API key leaked into Google Cloud, especially if it is coupled to Gemini access. Why not use that to make the easy dev experience? This must be some sort of overlooked fuckup. You’d either ship this and API keys for Gemini, or neither. Doing it and not using it for an easier dev experience is a head scratcher.

They started off behind, and have been scrambling to catch up. This means they didn't get the extra year of design-doc hell before shipping, so mistakes were made.

I was trying to test the gemini-cli using code assist standard.

To this day I am unable to access the models they say I should be able to.

I still get 2.5 only, despite enabling previews in the google cloud config etc etc.

The access seems to randomly turn on and off and swaps depending on the auth used (Oauth, api-key, etc)

The entire gemini-cli repo looks like it is full of slop with 1000 devs trying to be the first to pump every issue into claude and claim some sort of clout.

It is an absolute shit show and not a good a look.

Everytime someone proposes protobuf as an rpc format, I respond “Hell no! There’s no support for protocol versioning.”

Of course, I bring this up because they could just version their API keys, completely solving this problem and preventing future ones like it.

Versioning data formats is wrongthink over there, so I’m guessing they just… won’t.

Does JSON have support for protocol versioning?

I started replying with a clever approach to layer scopes onto keys… but nope. Doesn’t work.

How did this get past any kind of security review at all? It’s like using usernames as passwords.

Maliciously thinking allowing this increase billable. Thus it increases the bottom line and make stock go up... Which is good for vesting...

Sheesh. We're in a world where a global Big Tech security team lacks comptetance to run even one high-street locksmith.

I hope Google has a database with the creation timestamp for every API key they issued.

You can see the creation date even on the GCloud dashboard. But this information isn't helpful in recovering from this issue, if they're interested in recovering correctly, because there's no guarantee that even keys created before the launch of Gemini didn't have Gemini access added to the keys intentionally. There are also likely public keys created after the launch of Gemini that also erroneously received the Gemini grant. The key creation date is ultimately useless; what it comes down to is whether the key's usage is intentional or malicious, which is impossible for Google to determine without involving the customer.

If there's one thing Google is good at, it's logging.

I think Google has a database with everything. EVERYTHING.

Ohh so that's how that happened. I had noticed (purely for research purposes of course) that some of Google's own keys hardcoded into older Android images were useable for Gemini (some instantly ratelimited so presumably used by many other people already but some still usable) until they all got disabled as leaked like two months ago. They also had over time disabled Gemini API access on some of them over them beforehand.

I also noticed lots of Github projects expose their gemini key and was confused. This explains a lot.

What's frustrating is that a lot of these keys were generated a long time ago with a small amount of GCP services that they could connect to. (Ex. Firebase remote config, firestore, etc.)

When Gemini came around, rather than that service being disabled by default for those keys, Gemini was enabled, allowing exploiters to easily utilize these keys (Ex. a "public" key stored in an APK file)

Gemini API is not enabled by default, a project owner has to go explicitly enable it.

The problem described here is that developer X creates an API key intended for Maps or something, developer Y turns on Gemini, and now X's key can access Gemini without either X or Y realizing that this is the case.

The solution is to not reuse GCP projects for multiple purposes, especially in prod.

Please see my response to your pasted comment in another thread: for many APIs that you can enable on a GCP project, you are intended to use the same GCP project across the whole application for quota tracking. Google even makes you assert that you are only using one GCP project (or at least list out all GCP projects, which APIs are enabled on them and what their purpose is and why you have more than one) when seeking approval for public facing OAuth.

You are wrong that increasing projects have no cost; many services have project based costs (Cloud Armour rules cannot be used cross project at the base tier), many services (mostly observeability) degrade significantly cross project, the Google Cloud Console _sucks_ cross project.

You are also wrong in saying there are no projects that could reasonably have a safe api key made unsafe by this exploit.

One example, a service that has firebase auth must publish the key (Google's docs recommend). Later, you add gen ai to that service, managing access using IAM/service accounts (the proper way). You've now elevated the Firebase Auth Key to be a Gemini key. Really undeniably poor from Google.

You may have responded to one comment here. The comment you responded to is actually in agreement with you.

[Edit: It's likely that you intended to reply to this comment: https://news.ycombinator.com/item?id=47163147 ]

The problem is Google explicitly stating that those API keys are not secret and should be public, which indeed was true until Gemini came around.

The problem is that developer X did not properly scope the API key when he created it. Yes, separate projects would also stop this, but keys have been capable for ever and creating unrestricted keys is strongly discouraged. Pretty sure you can even set an org policy to prevent someone from doing so…

It's not enabled by default on projects but it's enabled by default on keys.

It shouldn't be enabled by default on either one.

Or usecase: developer X stopped using Maps/etc N years ago, and is long gone, and then developer Y stumbles into the company's google api console.

Of course, Google is full of smart anti-fraud experts, they just handle 80% of this shit on the back-end, so they don't care about the front-end pain.

[dead]

This seems so… obvious? How can a company of this size, with its talent and expertise, not have standardized tests or specs preventing such a blatant flaw?

First of all, Google is a shell of the company it used to be.

That said, I’d actually argue there’s an evolutionary explanation behind this where at a certain size, and more importantly complexity, an oversight like this becomes even more likely, not less.

Another takeaway: if Google can become a shell of what it once was (in terms of institutional competence, I assume you mean; Alphabet market cap seems to be doing just fine), so can your organization. As such: making something that isn't supposed to be part of your security strategy, look like it could be, is actually a long-term security risk. Sooner or later a new team will not read your own documentation, and jump to wrong conclusions. Also, it probably trains a bad security posture into your users. How many inexperienced devs saw that it was safe and expected (and apparently even required) to leave these keys out in the open, and concluded that the same logic might apply to someone else's API keys?

I think this was much less likely to happen without the needless obfuscation. If the only purpose is to identify what project the data is for, and you're trusting the client to report that value, and counseling the client to use that value in a way that trivially exposes it to everyone... what is the point of making it look like cryptic garbage? Just use the account signup name or something, and don't call it a "key" in your query parameters. Keys are supposed to unlock stuff. A name tag is not a key.

A thing I’ve learned about market cap in tech recently is that actually very little needs to get done on the core product. The momentum behind the brand is what carries the stock through time. The brand becomes its own compounding monetary instrument. Google had built a very very strong brand over the last 25 years or so. Only now is that starting to shift away from them. Because of that, I think we’ll start seeing them take more bold risks or they’ll be crushed by the weight of their own bureaucracy. This also tends to be the same reason startups can disrupt so swiftly.

An oversimplified version is this: So there are two core very critical components to the mid/late-phase tech megacorp strategy, you need to protect the core money printing product at all cost first and sustain that fiercely over a long period of time (decade+), then use any and all profits to find/fund the next cash cow, looking for optionality. While doing that, grow the market or consume a larger share of market. Google benefited from mainly the latter two and all while the internet blew up globally, funneling even more money into the machine.

It’s no secret that nearly every Google product that wasn’t search, lost them money. They were searching for the next big thing. They likely were some of the first to see AI as exactly that but moved too slowly to commercialize. Likely because of bureaucracy risk and also perhaps some sense of altruism in knowing the cataclysmic impacts AI could have. There have been plenty of former Google employees confirming this.

They also used to do things just to be cool, but those days have been long gone since Larry Page tapped out (and probably a few years before that, about a decade). Since then they’ve almost completely lost sight of what made them so successful that nobody even knows their vision or identity as a company today. These don’t correlate to market cap but they do silently lead to stagnation.

Their brand protects them from quite a lot but it’s not invincible.

YouTube and AdSense are both extraordinarily profitable. Google Play and Cloud are great revenue drivers as well. Search Ads are still the king of the hill, but it's definitely not their only profitable project, and many of the "unprofitable" projects funnel traffic to profitable ones--Chrome gives Search uplift, Google Play can't exist without Android, etc.

Definitely, they found another cash printer in Youtube but to be fair, they bought that. My comment wasn’t intended to be definitive. Yes, a lot of their other products are really just funnels to their search ads though. Strategic plays have value too of course.

Any major incumbent like Google can survive for a very long time as a shadow of its former self, like Oracle/IBM/HP. Perhaps no longer in the top-20 of the S&P500, but still going as a reliable, boring, ever-stumbling blue chip.

Seems like they ought to be dedicated security teams monitoring for exactly this: does a key to X give users access to not-X. Even more bizarre is their VDP team not immediately understanding the severity of the issue.

And slow down the time to ship things? The shareholders wouldn't like that.

Those poor poor institutional shareholders…

They do have dedicated teams for exactly these sorts of concerns. They are also swamped with projects and so they can't review big new changes overnight. Google is very likely shipping first and asking questions later.

"Don't worry, we have Gemini looking at this very issue right now for all teams"

"I know, I'm reading along!"

That's how you slow down development to a crawl

Yeah, lets just start building a house and don't wait for architects to finish the blueprints :) They just slowing us down with all that thinking things through stuff.

I don't see a problem with this. The problem with "move fast and break things" isn't the moving fast part, it's the trail of broken things that no one bothers to fix. When those broken things affect people's wallets, that's when we have problems.

That's fine. Right is better than now.

I'll riff off this and say that even Google in its heyday was strangely uneven from product to product. Some products were amazing, still pretty dang good. Some products were released in a mess, abandoned nearly from the start, or driven into the ground with seemingly very little competence driving them. It always felt like Google had a bit of a darker side lurking as far as just getting basics wrong product to product / team to team.

Ah yes, I too remember Google Wave.

I don’t see it.

Imagine for a moment the there is no oversight. Every intern can ship prod code with their own homemade crypto.

How do you, in a retail business, agree to accept credentials that anyone can mint for free?

I mean obviously it happened. But… this doesn’t even seem like a compliance mistake. It’s a business-level mistake.

If you've never worked in a large corporate environment you don't know how stupid things become. In a perfect bureaucracy nobody thinks.

> In a perfect bureaucracy nobody thinks.

This resonates so well and I love it. I'm stealing this

I work at a Fortune 10.

Things get stupid for sure. But I have never once seen “hey let’s do away with access controls for high-COGS services”.

It's never that explicit, it's more the things that nobody takes care of, because it's nobody's job. The bigger the company, the more jobs fall through the cracks, that should be taken care of, but lack an explicit role in the hierarchy.

There's usually a small handful of people that care more than they should, keeping the company afloat, but it's despite the company's policies, not because of them.

I feel it in a smaller but forced growing organization as the combination of atomised responsibilities and confused/overloaded coordination. For - a certian kind of - efficiency people are isolated into their responsibility area that they are able to oversee/comprehend - with accountability - that a manegement layer is supposed to coordinate. If the mangemenet layer is now overloaded or poorly executed - confused in case of evolution and growth and any kind of restructuring - but the atomic responsibility areas are having basically no (other than anecdotic employee chatter) oversight then troubles, even obvious ones, go undetected.

> First of all, Google is a shell of the company it used to be.

Isn't that squarely at odds with Google's supposed AI prowess? Is the rot really so severe that their advances in AI (including things they've yet to make public) are insufficient to overcome it? Or are the capabilities of Gemini and AI systems in general being oversold?

> Or are the capabilities of Gemini and AI systems in general being oversold?

I pretty much sure that if anyone asked Gemini "Is it good idea to retroactively opt-in new services into for old API keys?" it would suggest it's bad idea. Problem is that no one asked.

I've learned repeatedly that LLMs are very susceptible to helpfully giving you the wrong answer when you're asking the wrong question, or asking it in the wrong way.

… Of course they are being oversold.

But also, I don’t think even Google would claim that their LLM stuff can solve problems like this.

Stuff like this was proposed to be added to standard interviews, but they were too busy reversing binary trees

Google does have a security review process on literally everything it launches.

Which is what makes this so notable. Did the security review not catch this, or did they choose to launch anyways because it was too hard to fix and speed was of the essence?

I'd expect the security team to realize what the code is treating as a secret isn't actually secret.

But there's a second insight that seems tough for a security review to catch. You have to realize that even though you can't do anything obviously malicious with the API, there is a billing problem.

Have you been on these reviews? The idea that the review will catch a misuse of the key generation infrastructure is a bit over the top.

Maybe the experienced security reviewers were laid off.

Security. The final frontier. Where no developer has ever bothered before.

To boldly allow to go where many have gone before (but shouldn't have been able to)...

in a company of this size ... left hand don't know what right hand is doing

Their “talent and expertise” is mostly in selling ads.

They probably used the in house AI tools to build this.

"This seems fine"

Someone on the Google subreddit did report getting a 80k bill yesterday from a Gemini key.

I’m very careful with Google and co since they’re so intent on infinite scaling access to your wallet

This and problematic Gemini pro availability are why I pay for two other ai services and won’t pay google.

About 10 years ago I got $100 for free to use on AdSense. I used it for fun not realizing it keeps going and then billed me. Since then I basically don't use any Google paid products. Hope that $250 was worth it.

That's... pretty much how every free trial works? Try signing up for a free month of Amazon Prime or Netflix and see what happens. The entire point of the promotion is retention.

AdSense doesn't present itself as a permanent service you stay subscribed to.

(Or at least didn't at the time I've tried to use it. That may have changed, but we don't know when the GP tried it either.)

It wasn't a subscription. My expectation was that it would simply stop once it hit $0. Not really here to argue about it. The tldr is I don't trust Google with this stuff anymore.

I wonder what you thought putting your credit card on file was for.

Wait till you try aws!

The article mentions "Building software at Google's scale is extraordinarily difficult...", which I've seen many times before when one or another of these big corporations has a serious security flaw.

If a company like Google, with its ability to attract the best of the best, cannot handle the complexity of security and safety with SaaS/PaaS products, at what point do we say that perhaps this sector needs much more oversight?

Oversight by whom?

> Someone on the Google subreddit did report getting a 80k bill yesterday from a Gemini key.

Do you have a link?

https://www.reddit.com/r/googlecloud/comments/1reqtvi/82000_...

It’s pretty much a daily occurrence in all three of the big cloud subs that people still learning get wiped out because the clouds refuse to provide appropriate safeguards

Why spend resources on enterprise customers who have millions to spend on infra when they can just rely on obfuscation to make newbies pay hundreds if not thousands?

(/s, of course)

I think GCP has a setting for max monthly spend and is enabled by default

Other comments in this discussion disagree.

This is mind-blowing, and it defies all security common sense. Changing global API keys permissions? Come on! We’re accustomed to seeing issues like this from Redmond but didn’t expect it from Google.

Out of all of the cloud providers, I find Microsoft's authentication stack to be the most legible and stable. Everything else really sucks though.

You know things are bad when Microsoft is the most stable...

As someone who has used very many "cloud providers" (including GCP, AWS, and Azure), it cannot be said that Azure is the most stable. GCP is far better for stability and reliability than Azure.

The extensive experience with Enterprise Authentication that the decades of use of Active Directory has given Microsoft may mean that their SSO and Enterprise Authentication stuff is the best out of those on offer. I wouldn't know about that... I just made (and destroyed) VMs and was often driven to frustration whenever Azure failed to reliably perform that simple task.

It shows their red teams were preoccupied with something else. Even the primary rejection of this issue by G themselves shows some serious ignorance.

[dead]

Is the implication at the end that Google has not actually fixed this issue yet? This is really bad; a massive oversight, very clearly caused by a rush to get Gemini in customers' hands, and the remediation is in all likelihood going to nuke customer workflows by forcing them to disable keys. Extremely bad look for Google.

As I was reading it I didn't realize I was reading a security report, so I was like, is it responsible for them to be sharing this?

Then I saw the disclosure at the end and didn't get the sense that the flaw was fixed, so then I was still thinking... Is it responsible for them to be sharing this?

I'm glad that they did, because I can audit my own projects, but a bad actor may also be glad that they did.

The fact that we're hearing this first from a third-party and not from Google themselves is extremely problematic.

I think this is making at least some waves in google. I literally just got an email from them with the subject "[Action Advised] Review Google Cloud credential security best practices"

A slew of recommendations, one of them being:

Disable Dormant Keys: Audit your active keys and decommission any that show no activity over the last 30 days.

(Although I don't think this even addresses the underlying issue)

This totally reminds me of SSN use, when initially they were just a number (not secret) to identify a person, and then suddenly people started to use them as a key for authorisation, because someone had a bright idea how to implement things fast/simple/cheap (cheap part comes at expense of others)

Rather than being about fast/simple/cheap, I think using SSN as a key was more about the fact that SSN is the only common identifier that almost all US citizens have.

I think you're using the word "key" differently than OP. You're talking about identifiers, and they're talking about security.

SSNs were a good potential identifier, until the people that needed security cheaped out and started using SSNs as a bad implementation of security. Now they're bad at both purposes!

Yes, designing and implementing a new common identifier almost all US citizens have would have been less cheap and fast.

Tax ids were never meant to be used as a form of global identification. If you go look in a real bank core, you'll find this field does not have any uniqueness constraints.

Why not? Two people with the same tax ID seems like a problem waiting to happen.

These schemas also support non-individuals with the same fields. EINs have a lot more edge cases than SSNs.

Many people wanted to be able to set a spending limit on google cloud account for many years but they were unable to implement anything, always suggesting a workaround by hosting a Cloud Run function which would remove billing from a project via API https://docs.cloud.google.com/billing/docs/how-to/disable-bi...

As someone who is new to the whole google cloud ecosystem, the amount of dark patterns they employ are absolutely shocking. Just off the top of my head:

1. You never know how much a single API request will cost or did cost for the gemini api

2. It takes anywhere between 12-24 hours to tell you how much they will charge you for past aggregate requests

3. No simple way to set limits on payment anywhere in google cloud

4. Either they are charging for the batch api before even returning a result, or their "minimal" thinking mode is burning through 15k tokens for a simple image description task with <200 output tokens. I have no way of knowing which of the two it is. The tokens in the UI are not adding up to the costs, so I can only assume its the first.

5. Incomplete batch requests can't be retrieved if they expire, despite being charged.

6. A truly labyrinthine ui experience that makes modern gacha game developers blush

All I have learned here is to never, ever use a google product.

At scale, distributed API routing shouldn't call accounting transactions, that expands the availability risk surface and adds latency to all valid requests for no reason (other than helping the minority of companies/users who want their product to stop working when it is popular).

Distributed “shared nothing” API handling should make usage available to accounting, and the API handling orchestrator should have a hook that allows accounting to revoke or flag a key.

This gets the accounting transactions and key availability management out of the request handling.

That is a nice excuse, do you work at Google? :) I get the idea of not slowing down requests or risking availability, but don’t tell me a company as big as Google can’t design an asynchronous accounting system robust enough to handle this. We’re not talking about penny-perfect precision - blocking at 110% or even 150% of the set cap would be enough. Right now, though, there’s nothing to prevent a $5k, 20k or even higher bill surprise due to API key leaks, misuse or wrong configuration. To me, this is unacceptable and one of the reason I try to avoid using gcloud (the other one is unbearably slow gogole cloud console "webapp").

That’s exactly what the cloud function does

Yes but each admin has to use their product (cloud function), configure IAM and do that for every project. This is clearly just a work-around.

I haven't used these budget alerts, maybe they are a pain to implement?

https://docs.cloud.google.com/billing/docs/how-to/budgets

They are still not a spending cap of course.

reminds me: Ever used Gemini API on Google Vertex Cloud API? The usage will show up like 24-48 hours later in the dashboard. So when you use Gemini's API on their Cloud me as Workspace admin cannot even track my own usage in near realtime there. Which makes me think that even Google cannot track it in realtime.

Unrestricted API keys were always secrets. They are created on a page called "Keys & Credentials". The fact that Google even allows unrestricted keys to be created has been a long standing security problem. The fact their docs encouraged it remains unforgivable.

I can maybe understand unrestricted keys (OK, I can't, to be honest).

But the fact that permissions are not hardened at time of creation is bonkers to me.

Public keys are a thing in computing, though?

Google Maps has one, even. And Stripe.

It's been a while since I've used stripe but don't their keys start with sk_ for secret and pk_ for public?

I like that. Easy to tell if you should keep the key a secret or not.

They do, yeah.

(Although `pk` always freaks me out. Public or private?! Oh, right, the other one's "secret".)

Or is `sk` shared key and `pk` private key...

I would like to restrict the term "Public keys" to refer to asymmetric encryption keys which can be made public without compromising security.

The only purpose of the keys Maps/Stripe encourage you to publicly put into your website is to guarantee it is talking to _your_ Google/Stripe account not someone else's. Obviously once you put them in your client they are of zero value in helping Google/Stripe identify you. The fact that Google allows you to use the same type of key they also use elsewhere to identify _you_ not _them_ was always incredibly bad design. Google already have the 'Project ID' which would have been the best thing to use.

Am i reading this right - it was like impossible to get an api key for gemini but actually i could have just grabbed an API key from someone's google maps site and gotten started right away?

> Retroactive Privilege Expansion. You created a Maps key three years ago and embedded it in your website's source code, exactly as Google instructed. Last month, a developer on your team enabled the Gemini API for an internal prototype. Your public Maps key is now a Gemini credential. Anyone who scrapes it can access your uploaded files, cached content, and rack up your AI bill. Nobody told you.

Malpractice/I can't believe they're just rolling forward

They should limit the new features to new API keys that explicitly opt-in instead of fucking over every user who trusted their previous documentation that these keys are public information.

Isn't it standard practice to harden permissions on API keys? Like, if I were a bootstrapped startup maybe I'd take shortcuts and let an API key have a * permission but not for anything that could rack up thousands of dollars in bills for the customer. But at googles scale that just seems irresponsible.

Maps keys should not be made public otherwise an attacker can steal them and drain your wallet and use it for their own sites.

Maps keys are always public in js on the website (but locked to use on certain domains). That’s how they work.

It is not actually locked to a site is just based off the host header. Which is public information an attacker can use to make the requests.

Sure, but the practical form of this attack is limited.

You can't maliciously embed it in a site you control to either steal map usage or run up their bill because other people's web browsers will send the correct host header.

That means you can use a botnet or similar to request it using a a script. But if you are botnetting Google will detect you very quickly.

> But if you are botnetting Google will detect you very quickly.

They don't do anything against that.

Is there a way to use Google maps apis on the web without exposing the key?

Re host header seems an odd way for Google to do it, surely they would have fixed that by now? I guess not a huge problem as attackers would have to proxy traffic or something to obscure the host headers sent by real clients? Any links on how people exploit this?

What is there to fix? It was designed this way.

Something that can be abused is if the key also has other Maps APIs enabled, like Places API, Routes API or Static APIs especially for scraping because those produce valuable info beyond just embedding a map.

The only suggestions I have are:

- If you want to totally hide the key, proxy all the requests through some server.

- Restrict the key to your website.

- Don't enable any API that you don't use, if you only use the Maps Javascript API to embed a map then don't enable any other Maps API for that key.

Google's own recommended practices https://developers.google.com/maps/api-security-best-practic...

It would be helpful if you answer the question about web api usage, most of that is not relevant.

The only suggestion I see there from a quick skim that would avoid the above is for customers to set up a google maps proxy server for every usage with adds security and hides the key. That is completely impractical suggestion for the majority of users of embedded google maps.

It’s been years but I thought I recalled having to use the key but then also setting what sites it’d work on.

If an attacker can figure out what sites it can be used on, they can use the API.

This is just embarrassing. It doesn’t even really qualify as a security vulnerability, more like a fatal flaw in the system’s design. I can see why the team pushed back on fixing it, seems like a massive pain.

It feels like something that would happen if you outsourced planning to an LLM.

While I completely appreciate the final sentence, the article clearly describes the probable escalation, exposure of confidential information, and other security issues created through the decisions of Google.

> This makes sense. These keys were designed as project identifiers for billing, and can be further restricted with (bypassable) controls like HTTP referer allow-listing. They were not designed as authentication credentials.

Can't you just run up a huge bill for a developer by spamming requests with their key? I don't see how this wasn't always an issue?

Keys could have certain restrictions [1] such as HTTP Referer, which meant you couldn't just embed a map on your website and charge a different website for the views.

Not perfect protection of course - an attacker could spam requests with all the right headers if they wanted to - but it removes one of the big motivations for copying someone else's API key.

[1] https://docs.cloud.google.com/api-keys/docs/add-restrictions...

I was thinking more maliciously targeting the developer and running up a huge bill than reusing their key for your use

I guess this was an issue all along - but the cost per request is most def way higher for LLM API calls than for e.g. a Maps API call.

with llms maybe you can reuse their api for your own benefit instead of just showing some maps, so the issue is even worse that only cost.

Woof. Impedance mismatch outcome from moving fast - the GCP auth model was never designed to work like oAI's API key model; this isn't the only pain point this year, but it's a nasty one. I'm sympathetic, except that dealing with GCP has always been a huge pain in the ass. So I'm a little less sympathetic.

API keys were always secrets. They control billing for heaven's sake. If you had any per-call billed APIs (like some of the voice processing APIs) enabled on the project then they're effectively keys to your pocket book. Otherwise they're a key tool to manage denial-of-service attacks.

Can’t wait til someone makes a Gemini prompt to find these public keys and launch a copy of itself using them.

Arguably, calling it a key while insisting it's a non-sensitive ID was a mistake to start with

Changing the semantics of existing non-key keys, making them actually keys is horrendous

I reported few instances last year, some companies fixed it, some other didn't even understand the problem (or ghosted me).

So even if they fix the issue, it sounds as though you can still shoot itself in the foot by essentially being at to arbitrarily change an existing key from “not a secret” to “is a secret”?

Even if you have a key that you use for maps (not secret) someone could add the generative AI scope to it and make it now necessarily secret (even though it’s probably already publicly available)?

the credential didn't change. the permissions changed underneath it. that's the worst kind of privilege escalation because nobody has a reason to go back and audit something they were told was safe a decade ago.

Who knew there were downsides to forcefeeding your product to an unwilling audience?

This whole Gemini roll-out has me reminded of the Google '+' days when they thought they were going to die if they didn't do social.

What are the chances this isn't intentional to some extent? This wouldn't be the first time we've traded downstream legal trouble for short term gains.

Making AI utilization appear to go up is the only thing that matters right now if you're in the boardroom at one of these companies. Whether or not that utilization was actually intended by the customer is entirely irrelevant. From here, the only remaining concern is mitigating legal issues which google seems to be immune to.

Does anyone really believe something like this?

There's a long stretch from over optimizing a UI to something that is very clearly an error like what has happened here.

I save $20/mo on my internet by having cable that I don’t watch. Why? So my telecom company can boast higher tv subscriber counts to shareholders and ad-networks.

It is entirely believable to me that a company like Google would do the same with AI use numbers. I suspect that all these AI use factors in corporate performance reviews are about the same thing.

This could be a standard oversight too, I find Google’s documentation on this stuff to be Byzantine.

They said they were going to disable it for leaked keys isn't better to just disable it for leaked keys. Isn't better to make the default behavior from now on to not have access to Gemini or I misunderstood?

I’ve been exploring this exact problem space from the angle of extreme constraints (single-digit MB memory, no cloud assumptions). I documented what broke first and why here, in case it’s useful: https://github.com/nullclaw/nullclaw

> Your public Maps key is now a Gemini credential. Anyone who scrapes it can access your uploaded files, cached content, and rack up your AI bill.

This destroys Google's right to pursue an unpaid "AI" bill as a debt.

This firm is doing great work, I still refer to this post ("Anyone can Access Deleted and Private Repository Data on GitHub"): https://trufflesecurity.com/blog/anyone-can-access-deleted-a...

Happened to me recently, I got a warning in Gemini Studio that a key leaked. I was perplexed initially and then realized what had happened. The proper fix is to limit the key to just Maps APIs. Of course even this is not so easy, as there's a long list of APIs with complicated names. It was at least limited to my domain.

This is true but also not as new as the author claims. There have been various ways to abuse Google API keys in the past (at least to abuse them financially) and it’s always been very confusing for developers.

Seems like the kind of bug caused by using Gemini to vibe code the GCP.

You must be right. Do you have inside info?

Uh what? Google maps API keys have always been separate and they have always adviced to lock it down to your domain such that others can abuse it.

> 2,863 Live Keys on the Public Internet

It will be more interesting if they scan GitHub code instead. The number terrified me. Though I am not sure how many of that are live.

2k feels very small considering the number of business sites that embed Google Maps. I guess a lot of those sites use other website building services that handle the Google API keys for them, and/or they're old and untouched enough that no one enabled Gemini on them.

I had the same thought. I guess a lot of those keys may belong to dormant/deleted accounts and only a % of people who have enabled Gemini (presumably it required user action)

[flagged]

I did. Specifically the part about "When you enable the Gemini API". This doesn't take into account that people may have had years old forgotten about other services they use.

Either way it requires action, there is nothing to presume about that

Fair enough. It's a reasonable expectation of someone that enabled Google maps 15 years ago that enables Gemini 6 months not to understand the fundamentals of how Google treats their keys. If it wasn't explained on the enabling Gemini screen, what do you expect the user to do.

Totally agreed. But it clearly requires user action. I have some old projects that only use Google Maps for websites and that wouldn't magically be impacted. Google needs to do better though

Thousands of engineers. Culture rot.

This is so weird, this feels like an incredibly stupid bug that any average developer would've noticed, but Google is so incredibly selective with their tech screen. What exactly is the point of those if they're going to fuck up in obvious ways?

Since I've never used them, how could API keys for Firebase or Maps be safe for embedding in client side code?

I mean, I get that authentication to the service is performed via other means, but what's the use of the key then?

I'm guessing it's just a matter of binding service invocations to the GCP Project to be billed, by first making sure that the authenticated principal has rights on that project, in order to protect from exfiltration. That would still be a strange use case for what gets called an "API key".

> That would still be a strange use case for what gets called an "API key".

The problem that you, and many people are having in this thread, is that you are typing "API key" but, in your head, you're thinking "private API key". API keys can be secret or public, and many services have matching pairs of secret and public keys (Stripe, Chargify, etc. etc. etc.)

They’re bound to http Referrer iirc

it's just firebase part 2

Wait, I can get such a key and perform gemini API requests with curl? (probably limited in some ways)

I'm a bit surprised by the timeline which seems to say that:

- 6 weeks ago Google said they would fix it

- 3 weeks ago Google said they were working on it

...but we're publishing the info anyway, so everyone can go nuts with it.

That's the nature of disclosure deadlines. Talk is cheap. If they didn't disclose when they said they would, Google wouldn't feel any pressure to fix the issue.

It seemed like this was already being exploited online so it is responsible to disclose so people can protect themselves by revoking their keys. Bills near $100,000 are showing up for people.

Great write-up. Hilarious situation where no one (except unwieldiness) is the villain.

Dang, another obvious reason (among many others) you shouldn't be uploading documents to any LLM client (or use them on anything important).

[dead]

[dead]

[dead]

[flagged]

[flagged]

The new code changes from not existing, to existing.

Indeed, the key doesn't change. The new capability comes from the new code.

It would not be a re-evaluation of risk, because this is a new project. The evaluation of risk is supposed to come at the moment when the new capability is implemented, and consciously tied to an existing key type, which was previously advertised as non-secret.

They're obviously talking about on the client's end, not Google

You're replying to an LLM.

It did read a bit LLM-ish, but I didn't expect that someone would prompt an LLM to present that particular point of view. Usually it seems like they don't try to argue for anything in particular (except perhaps to imply that the article was well written.)

ChatGPT writing a blog post attacking Gemini security flaws. It's their world now, we're just watching how it plays out.

How do you know that this blog post was written by ChatGPT?

It feels generated to me too. It’s this:

    When you enable the Gemini API (Generative Language API) on a Google Cloud project, existing API keys in that project (including the ones sitting in public JavaScript on your website) can silently gain access to sensitive Gemini endpoints. No warning. No confirmation dialog. No email notification.

If you scroll through tiktok or instagram you can see the same exact pattern in a lot of LLM generated descriptions.

I think there's a lot more than just that, but I think part of the problem is that you just get an uncanny valley feeling. All of the phrases and rhetorical tricks that these tools use are perfectly valid, but together they feel somehow thin?

That said, some specific things that feel very AI-y are the mostly short, equally-sized paragraphs with occasional punchy one-sentence paragraphs interspersed between them; the use of bold when listing things (and the number of two-element lists); there are a couple of "it's not X, it's Y"-style statements; one paragraph ends with an "they say it's X, but it's actually Y" construct; and even the phrasing of some of the headings.

None of these are necessarily individually tells of AI writing (and I suspect if you look through my own comments and blog posts on various sites, you'd find me using many of the same constructs, because they're all either effective rhetorically, or make the text clearer and easier to understand). But there's something about the concentration of them here that feels like AI - the uncanny valley feeling.

I would put money on this post at least having gone through AI review, if not having been generated by AI from human-written notes. I understand why people do that, but I also think it's a shame that some of the individual colour of people's writing is disappearing from these sorts of blog posts.

Using threes is common in English writing and speaking. It has an optimal balance of expressiveness (three marking a pattern or breadth; creating momentum) without being overwhelming.

It’s not uncommon, as basic writing advice, to use sets of three for emphasis. That isn’t a signifier of LLM generation, in my opinion.

It's also seemingly the only way ChatGPT knows how to write, while being very uncommon for blogposts beforehand. Of course it's not 100% proof, but it's the most likely explanation.

It has a name. The Rule of Threes. https://en.wikipedia.org/wiki/Rule_of_three_(writing)

“The rule of three is a writing principle which suggests that a trio of entities such as events or characters is more satisfying, effective, or humorous than other numbers, hence also more memorable, because it combines both brevity and rhythm with the smallest amount of information needed to create a pattern.”

It’s how I was taught to write, but I understand that my personal experience can’t be generalized to make sweeping statements.

Do you have data that suggests it’s uncommon in human-authored blog posts and more common in LLM-generated text?

> It has a name. The Rule of Threes. https://en.wikipedia.org/wiki/Rule_of_three_(writing)

I don't think that's exactly it.

Speaking of LLM-writing in general, it seems to greatly overuse certain types of constructions or use them in uncommon contexts. So that probably isn't so much using the rule of threes, but overusing the rule of threes in certain specific ways in certain specific contexts.

I don’t necessarily doubt you or the grand-parent comment, but if it’s ‘obvious to even the most casual of observers’ (as my father would say) then it should be easy to have hard data.

This excerpt is demonstrating the use of a literary technique to write non-literary prose. It's an almost sure sign that an LLM is generating the text.

Of course, how could a writer writing have writing chops and use writing techniques? It boggles the mind that anyone thinks that would ever happens. Must have been aliens.

A good writer knows when to use literary techniques.

They work just fine in this post.

Yeah, it's perfectly reasonable device that I often use. I love the circle reasoning being displayed:

  "this sounds like AI"
  "professional writers use this technique"
  "they can't be a professional writer, they're using AI"

No, it’s unpleasant to read. To be clear, it’s possible a person wrote this, and that would not change it being unpleasant.

I’m not a native speaker so my level of AI recognition is already low. I find it very interesting what patters people bring up to declare it’s AI. The 3 punchline one for instance is a pattern I use while speaking. Can’t say I would write like this though.

It's not so much the grouping of 3 or way it's supposed to be punchy specifically that's the problem, that is just one example of what gives the article the "LLM Generated" feeling since whatever cheap model people are using for this kind of spam has some common ticks.

I use groupings of 3 and try to make things punchy myself sometimes, especially when I'm writing something intended to sway others. I think the problem with this article is the way it feels like the perfect average of corporate writing. It's sort of like the "written by committee" feel that incredibly generic pop music often has.

When I write things, I often go back and edit and reword parts. Like the brushstrokes in an oil painting, the flow of thought varies between paragraphs and even sentences. LLMs only generate things from left to right (or vice versa in RTL languages, I presume). I think that gives LLM generated text a "smooth" texture that really stands out to anyone who reads a lot.

I completely agree with you. There's something conspicuous about this particular use of the "group of three" device. It's trying but it's goofy and conspicuous. I think it's not human, it's 52 trillion parameters in a trenchcoat.

I'm not a native speaker and my level of AI recognition is higher than 99.999% of native speakers - and I'd be happy to be tested on it for proof.

The biggest factor is simply how long you've been using LLMs to generate text, how often, how much. It's like how an experienced UI designer can instantly tell that something is off by a single pixel off upon first seeing a UI, whereas if you gave me $200 to find it within 10 minutes I might well fail.

Aside from particulars like the set of 3, LLMs add a lot of emotive language which doesn't mean anything or is a repetition of already established points. Since they can't add any actual substance beyond what was in the prompt, the only thing they do is pad the prompt with filler language.

OK I've seen many people make this point on this site over just the last few months, but where do you think LLMs pick up these patterns? How did this rule of threes https://en.wikipedia.org/wiki/Rule_of_three_(writing) get into the LLM so they are so damn recognizable as LLMs and not as humans?

HN Note: Yes the rule of threes is broader than just this particular pattern here, but in my opinion this common writing and communication pattern is a specific example of the rule of threes.

Punchy repetition in a set of 3. Yes. LLMs are able to capably mimic the common patterns that how to write books have suggested for the last 100 years as ways to make your writing more "impactful" and attention-grabbing. So are humans. They learned it from watching us.

I am a little bit worked up on this as I have felt insulted a couple times at having something I've written been accused of being by an LLM, in that case it was because I had written something from the viewpoint of a depressed and tired character and someone thought it had to be an LLM because they seemed detached from humanity! Success!

I too would like to be able to reliably detect when something has been written by an LLM so I can discount it out of hand, but frankly many of the attempts I see people make to detect these things seem poorly reasoned and actively detrimental.

People have learned in classes and from reading how to improve their writing. LLMs have learned from ingesting our output. If something matches a common writing 101 tip it is just as likely to be reasonably competent as it is to be non-human. The solution to escape being labelled an LLM is not to become less competent as a writer.

I have been overly verbose here, as I am somewhat worked up and angry and it is too late in the morning to go back to sleep but really too early to be awake. I know verbosity is also a symptom of being an LLM, but not giving a damn is a symptom of humanity.

>but where do you think LLMs pick up these patterns?

>LLMs are able to capably mimic the common patterns that how to write books have suggested for the last 100 years as ways to make your writing more "impactful" and attention-grabbing. So are humans. They learned it from watching us.

Don't forget that LLMs (at least the "instruct" versions) undergo substantial post-training to align them with the authors' objectives, so they are not a 100% pure reflection of the distribution seen on the internet. For example, it's common for LLMs to respond with "You're absolutely right!" to every second message, which isn't what humans usually do. It's a result of some kind of RLHF: human labelers liked to hear that they're right, so they preferred answers containing such phrases, and those responses became amplified. People recognize LLM-generated writing because LLMs' pattern distribution is different from the actual pattern distribution found in articles written by humans.

It's too well structured and the message is too clear. HN (and the whole internet) is allergic to proper writing. We praise human sloppiness now.

No, I'm not being sarcastic. People have given up em-dash, which is an official punctuation you use in proper writing. And it's all a downhill from there.

Strongly disagree. The post is really poorly structured and circles the drain a few times getting to the thesis.

The issues of style are annoying, but I find it much worse to wade through these 3000 word posts which are far longer than they need to be just because they're so damn cheap to compose.

> It's too well structured and the message is too clean. HN (and the whole internet) is allergic to proper writing. We praise human sloppiness now.

Yes. And it's only a matter of time that the model companies start to try to train in that "human sloppiness." After all, a lot of their customers want machines that can pass for humans.

> No, I'm not being sarcastic. People have given up em-dash, which is an official punctuation you use in proper writing. And it's all a downhill from there.

I wouldn't be surprised if the internet language of people devolves into a weird constantly-changing mish-mash of slang and linguistic fads. Basically an arms race where people constantly innovate in order to stay distinct from the latest models.

But the end result of that would be probably fragmentation, isolation, and a kind of dark ages. Different communities would have different slang, and that slang would change so fast that old text would quickly become hard to understand.

It's far longer than it needs to be because the writing process was too cheap.

> The Core Problem

> What You Should Do Right Now

> Bonus: Scan with TruffleHog.

> TruffleHog will verify whether discovered keys are live and have Gemini access, so you'll know exactly which keys are exposed and active, not just which ones match a regular expression.

I don't know exactly, but I'm sure. The cadence, the clarity, the bolding, the italics, it's all just crisp and clean structured and actionable in a way that a meandering human would not distill it down to.

Yup, it was actually an interesting article but there are a few telltale parts that sound like every AI spam post on /r/webdev and similar. "No warning. No confirmation dialog. No email notification." is another. The three negatives repeated is present in so many AI generated promotional posts.

I don't even have a problem with the content itself, I think frankly the smell is that it's too good. It's just fascinating in the sense that it's one LLM attacking another LLM.

I've reached the point where if any blog post has a subheading with some variant of "The Problem", I assume it's been edited with an LLM, because it co-locates with other indicators so strongly.

It's too structured and consistent. Imo. Has that AI smell to it, but I guess humans will eventually also start writing more like the AIs they learn from.

This is the first time I've seen people accuse AI text of being "too structured and consistent" compared to human text. Usually it's about specific patterns or tons of repetition or outright mistakes.

One example of being "too structured" is that LLMs love an explicit introduction and conclusion even when one that isn't really warranted. It's always telling you what it's going to say, and what it just said.

Patterns = consistent?

Patterns like heavy use of certain words or dashes or bullet points don't change how consistent the overall post is.

AI was trained on human writing.

> AI was trained on human writing.

AI output is not varied like real human writing. This is a very distinctive narrowing of style.

And now humans are trained on AI writing.

Like what happens to YouTube videos that go through the compression algorithm 20 times.

> guess humans will eventually also start writing more like the AIs they learn from.

With the AI feedback loop being so fast and tight for some tasks, the focus moves on to delivery than learning. There is no incentive, space or time for learning.

For me personally, both at work and in my free time, I spend _more_ time on writing things _that matter_ since I’ve freed up time by using LLM’s for boilerplate tasks.

My motto is - If it wasn’t worth writing, it won’t be worth reading.

A good example of writing where I’d recommend using LLM’s is product documentation. You pass the diff, the description of the task, and the context (existing documentation) with a prompt ”Update the documentation…”.

Documentation is important but it’s not prose. However, writing a comment on hacker news is.

Won't be well received here, but this is the truth.

The fact that according to this reply section most of HN can't tell means that predictably, all hope is lost and there's no point in writing anything by hand any more if you're in it for money/engagement.

While writing this I suddenly realized that marketers and writers probably do a better job at recognizing it than developers and engineers, so maybe all hope isn't.

For those who want to know the tells: overall cadence and frequency of patterns - especially infrequency of patterns - are the biggest ones. And that means that we can't actually give you the best tells, because they're more about what is absent than what is present. What's absent is a single sentence pattern that falls completely out of the LLM go-toes. Anything human written has at most a good mix of both. LLM-written text just entirely lacks it. Humans do use the LLM-preferred patterns, but not for every single sentence. But anyway, here we go.

> Transparently, the initial triage was frustrating; the report was dismissed as "Intended Behavior”. But after providing concrete evidence from Google's own infrastructure, the GCP VDP team took the issue seriously.

^ Fun fact - The ";" would've originally been an em-dash but was either rewritten or a rule was included for this.

> Then Gemini arrived.

^ Dramatic short sentences, a pattern with magnitudes higher LLM-frequency than human frequency, but hasn't reached the public conscious yet a la "not just X but Y".

> No warning. No confirmation dialog. No email notification.

^ Another such pattern. Not just because it's three of them, but also because of the content and repetition. Humans rarely write like that because it again sounds overly dramatic. It's something you see in fiction rather than a technical writeup. In a thriller.

> Retroactive Privilege Expansion. You created a Maps key three years ago and embedded it in your website's source code, exactly as Google instructed. Last month, a developer on your team enabled the Gemini API for an internal prototype. Your public Maps key is now a Gemini credential. Anyone who scrapes it can access your uploaded files, cached content, and rack up your AI bill. Nobody told you.

This style of scenario writing is another one.

> Nobody told you.

Absolute drama queen.

>The UI shows a warning about "unauthorized use," but the architectural default is wide open.

Again.

> The attacker never touches your infrastructure. They just scrape a key from a public webpage.

Again.

> These aren't just hobbyist side projects. The victims included major financial institutions, security companies, global recruiting firms, and, notably, Google itself.

..

> A key that was deployed years ago for a completely benign purpose had silently gained full access to a sensitive API without any developer intervention.

Surprised it hasn't gained consciousness by now. Maybe that's a future plot point.

Here's a great example to train your skills on, because it's rare in that the ratio of "human : straight from LLM" increased gradually as the article goes on: https://www.wallstreetraider.com/story.html

It started at heavy human editing (or just human-written), but less and less towards the end.

The author confirmed this upon pointing it out, FWIW [0].

[0] https://news.ycombinator.com/item?id=47013150

They don't. Many of these claims are due to illiteracy.

Someone is complaining that

> it's all just crisp and clean structured and actionable in a way that a meandering human would not distill it down to.

but this is a security report ... people intentionally write such things carefully and crisply with multiple edits and reviews.

They may have used ChatGPT or similar to help with the prose but the technical content (as discussed elsewhere on this page) is good, so does it really matter if they did?

The problem with AI slop (to me) is more that the technical content is not good or is entirely the product of the LLM. At that point, there's no point in me reading it, I can just prompt the question if I'm interested.

This is original research which wasn't public before, so the value is still there and I didn't think whichever combination of a human and LLM that generated it did a bad job.

Private data should not be allowed to be accessed using public keys. That is the core problem. It is not about Google API keys are secret or not.

It was intended for situations where the keyholder is a middleman between Google's API and the end user.

Explain It Like I'm Five.

From TFA:

> Last month, a developer on your team enabled the Gemini API for an internal prototype. > The result: thousands of API keys that were deployed as benign billing tokens are now live Gemini credentials sitting on the public internet.

Benign, deployed openly without any access restrictions whatsoever, billing tokens can be used to bill for a service under the account it is enabled for. That's the intended behavior, literally. Maps API keys are used to give your users access to Google Maps on your credit card.

What's the problem here? Yes, the defaults could have been stricter, but it's not like it costs anything to create a bunch of internal projects that do not have good-for-billing access keys floating around open internet. People moved fast, deployed LLM generated code, broke things and then blame everyone else but themselves?

The problem is that Maps API keys are now used to give your users access to Gemini, including sensitive content in that service

At least read the article in full before commenting. You don't need to deploy LLM generated code at all for the privilege escalation. The Gemini API merely needs to be enabled and there are no access restrictions by default.

Google guidelines say "API keys" (a huge misnomer for something that is more accurately described as a project ID) are not secrets. The idea of creating an internal project goes against what the guidelines suggest. The "API keys" are customer facing identifiers.