Access Token for internal GCP project “cxl-services” (with scopes
https://www.googleapis.com/auth/cloud-platform") can be leaked using a URL whitelist bypass on
https://cxl-services.appspot.com/proxy. This token could be used to elevate privileges in other internal projects, access some internal Compute instances and possibly take full control over
cxl-services.appspot.com. This App Engine app proxies demo API requests on Google Cloud product pages (e.g. Cloud Text-to-Speech Demo).
The domain whitelist check on
cxl-services.appspot.com can be bypassed with the
\@ trick. It is probably using the vulnerable regex from https://tools.ietf.org/html/rfc3986#appendix-B.
When validating the URL
https://[your_domain]\@jobs.googleapis.com, the regex thinks that the authority is
jobs.googleapis.com, but the library making the request actually parses
[your_domain] as the authority, and
/@jobs.googleapis.com as the path.
Steps to Reproduce:
If you already have an HTTPS server where you can view request headers, skip to
You can probably use plain HTTP and
netcat, but then the access token would be transported unencryped.
- Set up a VPS, and point a domain to it
- Generate an SSL certificate with LetsEncrypt
- Download my
httpsserver.pyPython script, and fill out the
- Create an empty directory,
cdinto it, and start the server with
sudo /path/to/httpsserver.py --verbose
- With a browser, open the URL
- On your VPS, see that the
cxl-services.appspot.comApp Engine app has sent a request with the header
authorization: Bearer [access-token]
I’ve sent some (in my viewpoint, harmless) read-only requests with the access-token to prove impact, here is what I found:
This access token has (some) access to these GCP projects:
Based on some other API requests I made, it feels like the token has “full” access to
cxl-services, and limited access on
On the project
docai-demo I have found 2 GCE instances:
Both of these instances have the metadata entry with key
kube-env witch exposes (to my limited Kubernetes knowledge) all of the node’s credentials:
cxl-services is hosting the App Engine app
https://cxl-services.appspot.com/, which proxies demo API requests on the Google Cloud product pages. It feels like this access token has full access to this App Engine app, but I wasn’t able to prove any impact here, since the API
appengine.googleapis.com is not enabled on the
cxl-services project, and I didn’t want to enable it.
I also saw some enabled
autopush APIs, so some non-public APIs/versions might be available using these projects.
Interestingly enough, I wasn’t able to hit the Metadata API using this domain bypass. Every attempt resulted in
500 Internal Server Error. But it looks like this access token has the same privileges as the service account I could have stolen from the Metadata API.
Documentation of my actions:
During testing I might have made
cxl-services.appspot.com send HTTP / HTTPS requests (with the access token) to these domains not under my control:
metadata.google.internal(for metadata testing without success)
I have called these API methods with the access tokens obtained:
An anonymous attacker could exploit the URL validation vulnerability, and steal the access token the
cxl-services.appspot.com App Engine app is sending.
Using this access token, the attacker could access resources on GCP projects
Some attacks I think could be performed by an attacker:
- Take over the App Engine app
https://cxl-services.appspot.com/, and steal the contents of all demo API requests and/or return malicious content to the users trying out the demos on the Cloud product pages.
- Take over the GKE cluster used by
docai-demo, (which is probably running the demo on
https://cloud.google.com/document-ai) and steal the contents of all demo API requests and/or return malicious content. This falls under the first attack’s impact, since this demo is also proxied by
- Try escalate privileges and maybe gain access to more internal Google Cloud resources.
Vendor - 2021-03-23 12:10
NOTE: This e-mail has been generated automatically.
Thanks for your report.
This email confirms we’ve received your message. We’ll investigate and get back to you once we’ve got an update. In the meantime, you might want to take a look at the list of frequently asked questions about Google VRP.
If you are reporting a security vulnerability and wish to appear in Google Security Hall of Fame, please create a profile.
You appear automatically in our Honorable Mentions if we decide to file a security vulnerability based on your report, and you will also show up in our Hall of Fame if we issue a reward.
Note that if you did not report a vulnerability, or a technical security problem in one of our products, we won’t be able to act on your report. This channel is not the right one if you wish to resolve a problem with your account, report non-security bugs, or suggest a new feature in our product.
Google Security Bot
Follow us on Twitter!
Vendor - 2021-03-23 18:58
NOTE: This e-mail has been generated automatically.
Just letting you know that your report was triaged and we’re currently looking into it.
You should receive a response in a couple of days, but it might take up to a week if we’re particularly busy. In the meantime, you might want to take a look at the list of frequently asked questions about Google VRP.
Google Security Bot
Me - 2021-03-28 21:29
I was able to demonstrate full access to the
cxl-services.appspot.com GAE app by deploying a
vrp-poc service: https://vrp-poc-dot-cxl-services.appspot.com/
I found some other sensitive things on project
- The bucket
2017-10-18up until today. These might contain sensitive user data.
- The bucket
staging.cxl-services.appspot.com(was, now I uploaded the 3 POC files) empty, so the source files were not available. (The
get-versionAppEngine API call listed all filenames, like
google3/cloud/ux/services/services/proxy.py) But, the
us.artifacts.cxl-services.appspot.combucket had some images, from which the source code could be extracted.
The requests I have made with the obtained access tokens:
- https://servicemanagement.googleapis.com/v1/services/appengine.googleapis.com - https://servicemanagement.googleapis.com/v1/services/issuetracker.corp.googleapis.com - https://serviceusage.googleapis.com/v1beta1/projects/cxl-services/services/appengine.googleapis.com:enable - https://cloudconsole-pa.clients6.google.com/graphql/SERVICE_USAGE_GRAPHQL - https://servicemanagement.googleapis.com/v1/services/appengine.googleapis.com:enable?consumer_id=project%3Acxl-services (success) - https://appengine.googleapis.com/v1/apps/cxl-services.appspot.com/services - https://appengine.googleapis.com/v1/apps/cxl-services/services - https://appengine.googleapis.com/v1/apps/cxl-services/services/default/versions - https://servicemanagement.googleapis.com/v1/services/issuetracker.corp.googleapis.com:enable?consumer_id=project%3Acxl-services (fail) - https://appengine.googleapis.com/v1/apps/cxl-services/services/default/versions/b347699687-dev-gokulr - https://appengine.googleapis.com/v1/apps/cxl-services/services/default/versions/b347699687-dev-gokulr?view=FULL - https://storage.googleapis.com/staging.cxl-services.appspot.com/ae-fingerprinted/5d8be43828d2c7161e68484222b6212529235137 (404) - https://storage.googleapis.com/staging.cxl-services.appspot.com/ae-fingerprinted/ (404) - https://storage.googleapis.com/staging.cxl-services.appspot.com/ (empty) - https://storage.googleapis.com/storage/v1/b?alt=json&fields=items%2Fid%2CnextPageToken&maxResults=1000&project=cxl-services&projection=noAcl - https://storage.googleapis.com/cxl-services.appspot.com/ (request log!) - https://storage.googleapis.com/artifacts.cxl-services.appspot.com/ (empty) - https://storage.googleapis.com/us.artifacts.cxl-services.appspot.com/ (images!) - https://storage.googleapis.com/storage/v1/b/cxl-services.appspot.com/o?alt=json&fields=items%2Fid%2CnextPageToken&maxResults=1000&project=cxl-services&projection=noAcl - https://storage.googleapis.com/storage/v1/b/cxl-services.appspot.com/o?alt=json&maxResults=1000&project=cxl-services&projection=noAcl (log sizez!) - https://storage.googleapis.com/upload/storage/v1/b/staging.cxl-services.appspot.com/o?alt=json&name=e8439a6b8f01e74d2dfc52eba0604d8c6885c834&uploadType=multipart - https://storage.googleapis.com/upload/storage/v1/b/staging.cxl-services.appspot.com/o?alt=json&name=28e492b17a63d596557c1c73ef4ac3e18447a21e&uploadType=multipart - https://storage.googleapis.com/upload/storage/v1/b/staging.cxl-services.appspot.com/o?alt=json&name=125e0b77828c18da7d6e2e698800b63996579658&uploadType=multipart - https://storage.googleapis.com/staging.cxl-services.appspot.com/ - https://storage.googleapis.com/staging.cxl-services.appspot.com/ - https://appengine.googleapis.com/v1/apps/cxl-services/services/vrp-poc/versions?alt=json - https://storage.googleapis.com/storage/v1/b/cxl-services.appspot.com/o - https://storage.googleapis.com/storage/v1/b/cxl-services.appspot.com/o?prefix=appengine.googleapis.com/request_log/2021
Vendor - 2021-03-29 08:53
🎉 Nice catch! I’ve filed a bug based on your report.
The panel will evaluate it at the next VRP panel meeting and we’ll update you once we’ve got more information. All you need to do now is wait. If you don’t hear back from us in 2-3 weeks or have additional information about the vulnerability, let us know!
[redacted], Google Security Team
Vendor - 2021-03-29 09:03
Thanks for the report, nice! great that you are not downloading files with potential PII, could you please confirm that you didn’t download them? If it was by mistake during testing please let us know which files, thanks!
Regards, [redacted], Google Security Team
Me - 2021-03-29 09:12
I only sent the requests documented above.
The files which could contain user-information I think (from my testing) are only the logs in
cxl-services.appspot.com. I didn’t download any of those objects, I just listed the contents of the bucket. (multiple times, even with a prefix to see if there are logs from 2021.)
Thank you, David
Me - 2021-03-29 09:22
There are still 3 projects for this access token that I didn’t really look into:
Can I play with these, or should I stop testing for now?
Vendor - 2021-03-29 10:01
Thanks for the info and quick followup.
Please don’t continue - the info we have is enough to fix and determine the reward, thanks!
Vendor - 2021-04-13 18:20
** NOTE: This is an automatically generated email **
Thank you for reporting this bug. As part of Google’s Vulnerability Reward Program, the panel has decided to issue a reward of $4133.70.
Rationale for this decision: 1k bonus for the well written report and documenting lateral movement. Thanks! :)
Important: if you aren’t registered with Google as a supplier, firstname.lastname@example.org will reach out to you. If you have registered in the past, no need to do it again - sit back and relax, and we will process the payment soon.
If you have any payment related requests, please direct them to email@example.com. Please remember to include the subject of this email and the email address that the report was sent from. Regards,
Google Security Bot
If you’d like your name added to our Hall of Fame:
Just create a profile here: https://bughunter.withgoogle.com/new_profile
In addition, we encourage you to signup for our Vulnerability Research Grants program, where we issue monetary payments to VRP researchers, even when no vulnerabilities are found. To read more about the program visit: https://www.google.com/about/appsecurity/research-grants/
– How did we do? Please fill out a short anonymous survey (https://goo.gl/IR3KRH).
Me - 2021-06-01 12:08
Just a heads-up, this issue will be disclosed in 20 days, on
I might make a video about this using screen recordings of finding the bug. These recordings include some internal information. Should I redact any particular details?
Also, my POC is still up and running: https://vrp-poc-dot-cxl-services.appspot.com/
Vendor - 2021-06-01 12:24
Thanks for the heads up about the disclosure of your report.
Please read our stance on coordinated disclosure. In essence, our pledge to you is to respond promptly and fix vulnerabilities in a sensible timeframe - and in exchange, we ask for a reasonable advance notice.
Some things to note:
- We might decide not to file a bug, or not to issue a VRP reward. Even if this is the case, please still coordinate your disclosure with us. Many of the reports we receive don’t describe vulnerabilities with an impact that qualifies for a reward, but we still want to make sure we are prepared for your vulnerability disclosure.
- We usually decide and pay the rewards before the vulnerability is fixed. The reward is not a payment for your silence, but we really want to have a chance to fix it on time for your disclosure, so please keep us up to date with the disclosure timeline, so we can update the engineering team of your plans.
- If you gave us enough time to fix the described issue (you can read https://www.google.com/about/appsecurity/ for inspiration), and we are anyway unable to fix it before your disclosure date, we’ll try to let you know in time (the disclosure date would still be your own decision).
At the end, the decision on what to disclose and when is yours, and we would be happy to forward any blog posts, slides, or paper drafts you might have to our teams internally (we might even provide you feedback). Thanks again for your help.
PS. If you give a talk, or make a blog post, or similar about this bug, please send us a link, we would love to read it, we might even tweet it from our Twitter account.
Me - 2021-06-16 12:10
So before starting to work on the writeup, I wanted to test the fix. It indeed fixed the original trick of using
But playing around a bit, I have found a way to once again bypass the URL whitelist:
Opening this URL sends a request to my server with an access token.
Actually, it is not specific to the
_ character. Putting anything between the
\ and the
@ bypasses the whitelist, for example:
Me - 2021-06-17 09:36
Will a fix roll out until
If a fix will be deployed, but not until the deadline, I can delay the disclosure 14 days, with a new date of
Let me know if I should delay the disclosure, of if a fix is coming.
Vendor - 2021-06-18 11:38
So this one has been marked fixed by the product team, but its pending verification that the fix is accurate on our side. Ostensibly the fix is live now. (just unverified/checked by us)
You’ll get a notification when we’ve done the verification, but in the above context, I think the answer is the fix is deployed :D
Me - 2021-06-18 12:13
I meant the fix for the bypass I found in comment #13. Did the product team fix that as well?
Vendor - 2021-06-22 18:07
Looks like the team did not know about that bypass. I reopened the bug and the team says it should be fixed now. Thanks!