Troubleshooting 14.2 to 14.2 MP1 AutoUpgrade

Debugging to enable for troubleshooting AutoUpgrade

SEPM Debugging:

1. Enable Finest logging on the SEPM

2. Enable Secreg/Secars debug on SEPM

3. SEPM and SEPM Web Service will need to be restarted after these changes.

SEP Client Debugging:

1. Enable CVE debug as well as SEP Debug in Symdiag.

2. Start the AutoUpgrade and wait for the issue to be reproduced.

3. The Heartbeat and Download Randomization will add to the time. Set a low Heartbeat and disable Download Randomization for faster AutoUpgrade

   Note: Do not set Download Randomization to disabled if you have a large amount of clients in your group.

AutoUpgrade Process

1. Once the client install package is assigned to a group in the SEPM console the clients will request the index file on the first Heartbeat and inspect the checksum in the index file.

2. When the client sees that the config.xml checksum is different from that stored on the local machine, it requests a new config.xml from the server.

   From CVE.log

   [2019-Feb-14 21:36:39.060753] [DEBUG] Config.xml MD5 is changed, attempting to get it from SEPM.
   [2019-Feb-14 21:36:39.068753] [DEBUG] Get Config.xml from SEPM successfully.
   [2019-Feb-14 21:36:39.070755] [DEBUG] Config.xml successfully updated

3. CVE parses the config.xml file, which contains information about the package(s) available on the server. This information is passed to SMC. SMC informs CVE if it would like CVE to download the package. If so, a download request is sent to the server. You can see AutoUpgrade request coming from the client on the SEPM in the exsecars-a.log.

   From exsecars-a.log

   02/14 21:41:35 [4636:4080] The agent doesn't have current package checksum .. setting to send Full version..
   02/14 21:41:35 [4636:4080] <CHttpRequest::HttpExtensionProc> [192.168.2.104] Completed Request action=192 (PostAgentInformation)  Status: 1 (Success)
   02/14 21:41:35 [4636:4080] <CHttpRequest::HttpExtensionProc> [192.168.2.104] GET h=85A1E9878E668E9AB30B887B401E94BF5A5869F70995CEF686DD331FCDC6FB70378AA1F9F9A8BC0C3C94CF6C1E104DB6A116BB1D9646AACD69CC461190F4A25B34580BE3AF313F3585D5D5A53B96B4702855DED47BF4C52FA4B4B0821CAE4452CCE56A6E2A37327604E6DA6F788003DF4B8C9D706A0A1A21C0BFE081CE3C662D97AFB297E595D58BFC7A029830624F65765312C8C432D69FA461B66D0D27A9924E01979CF3776070BACB93D33B876B08003BCB6E5AF65CA39E474CAB88CE41FEFDA30D1505A2E8F612DA6B678E5784A0105A4B8DDBC386BC80D2A2F4A53596B8CF8D50D06B9AAD37BC84FAA168349275053BF8D7798CA200B28358204B9B680F9B0243DA01B1AB5C69D6174EEA30E5B3034DF9B4326ADDD54EBB65A464EFD6A3EE3191A7DD0E61FD7984E3D7C0C904B519F86F406D2208148B22B861CBBA27FF07D669544EA572075E5207CF19A36E69A4EA3857FB5F7DA730F83B02F04C1155805F260A8FA49CABFFF7E914DE0A371479BE11E5034123226F6E5D4A2EA42818 ContentLen:0,UserAgent:Sylink,ConnId:51018592,CurrentlyProcessing:1
   02/14 21:41:35 [4636:4080] <CHttpRequest::HttpExtensionProc> [192.168.2.104] DecodedRequest: l=369&action=301&hostid=4AA92BECC0A802640C2E83292A5B7FF4&groupid=AA4C44F5C0A802643248771F8EC8347D&ClientProductVersion=14.2.770.0000&as=92&lun=[hex]41646D696E6973747261746F72&udn=[hex]4C6F63616C436F6D7075746572&agentpackagechecksum=&agentpackagetargetchecksum=eb4f96c7c348597407a344ed71378b65&agentpackagetargetmoniker={57201BD7-52EE-4841-8368-05C54B1F44DC}&lu=1&osv=06010000
   02/14 21:41:35 [4636:4080] Request from 4AA92BECC0A802640C2E83292A5B7FF4; CurrentAgentVersion: 14.2.770.0; OS version: 0x06010000
   02/14 21:41:35 [4636:4080] The client's OS satisfies the requirements for the latest client package update assigned to this group
   02/14 21:41:35 [4636:4080] The agent doesn't have current package checksum .. setting to send Full version..
   02/14 21:41:35 [4636:4080] <UpdateSignature>: Signature is NOT upto date in the cache for cfgItem: c:\program files (x86)\symantec\symantec endpoint protection manager\inetpub\clientpackages\eb4f96c7c348597407a344ed71378b65\full.zip .. Updating
   02/14 21:41:35 [4636:4080] <DoGetAgentPackageInfo> Signature Details: Item - C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\ClientPackages\eb4f96c7c348597407a344ed71378b65\Full.zip Sig - 09998CE3286EAA675A25C4588F5F57E7B348368886B6FC460B943F60ACD809068FF7D1D31B4FBE2DD7692AE054BAE249C6E72EB6E58ECBE7A7F1444509F556A65A91C8E6053792B2B8EFB233E8BD9000982345850EC06AC3973D3DB8170CC5B27C21343F775D720D373172DA22FB31DB10468EE7C3E71F78AE2C968F738E7A9A2E277731FB25517A32A7955F860499020A7486955C0F10026416A8A078A3AC0DC76BF45F7F7157ED9F7C717CDF7FBD27F69EF1AA3D89F5D42CE93EFBC83E35AB683890F77E21CE5AC7956FAA522222D6BAC52F859C69A58608B83AB526D6B2BE552BB280844A79AB97F0A5BA21204C091176C5918DAB470C328AD32C53F1D4AD
   02/14 21:41:35 [4636:4080] <CHttpRequest::DoGetAgentPackageInfo> Response Header:
   Content-Type: text/html
   Content-Length: 0
   Sem-SetContentLength: 0
   Sem-FileLength: 120704705
   Sem-PackageFull: 1
   Sem-PackageFileName: ClientPackages\eb4f96c7c348597407a344ed71378b65\Full.zip
   Sem-PackageFileLength: 120704705
   Sem-Signatue: 09998CE3286EAA675A25C4588F5F57E7B348368886B6FC460B943F60ACD809068FF7D1D31B4FBE2DD7692AE054BAE249C6E72EB6E58ECBE7A7F1444509F556A65A91C8E6053792B2B8EFB233E8BD9000982345850EC06AC3973D3DB8170CC5B27C21343F775D720D373172DA22FB31DB10468EE7C3E71F78AE2C968F738E7A9A2E277731FB25517A32A7955F860499020A7486955C0F10026416A8A078A3AC0DC76BF45F7F7157ED9F7C717CDF7FBD27F69EF1AA3D89F5D42CE93EFBC83E35AB683890F77E21CE5AC7956FAA522222D6BAC52F859C69A58608B83AB526D6B2BE552BB280844A79AB97F0A5BA21204C091176C5918DAB470C328AD32C53F1D4AD
   Connection: close
   02/14 21:41:35 [4636:4080] <CachedLogQueue::FlushHeadNode> Data written: 235 bytes
   02/14 21:41:35 [4636:4080] <CHttpRequest::HttpExtensionProc> [192.168.2.104] Completed Request action=301 (GetAgentPackageInfo (Full/Delta))  Status: 1 (Success)

4. Once the package is ready, the client will have received the link to the content on the server. At this moment, the client displays a notification to the user that the install is ready to begin (if notifications are on). Optionally, the user may also request for the download to be delayed or canceled if those package options are enabled.

5. The client's download thread downloads the package from the server.

   From Debug.log

   2019/02/14 21:41:40.843 [2008:5276] NVDF: new version will be downloaded.
   2019/02/14 21:41:41.127 [2008:5276] Accepting package for download.
   2019/02/14 21:41:41.128 [2008:5276] Start downloading auto-upgrade package!
    

6. The client will store the downloaded package (full or delta) to C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\CurrentVersion\SmcLU\Setup

   From Debug.log

   2019/02/14 21:42:02.400 [2008:5280] Create folder C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\14.2.770.0000.105\SmcLU\Setup for client package
   2019/02/14 21:42:07.442 [2008:5280] install to-install-SMC service:"C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\14.2.770.0000.105\SmcLU\Setup\smcinst.exe" -install

7. SMC installs the package-->launches **patchwrap.exe and **smcinst.exe. Patchwrap.exe rebuilds the new client package using the cached install files and the delta that was received. Smcinst.exe launches the MSI installer.

   From Debug.log

   2019/02/14 21:42:09.978 [2008:5280] Starting to-install-SMC service "C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\14.2.770.0000.105\SmcLU\Setup\smcinst.exe" -start
    

8. Msiinstaller and setup.exe will be launched to install the product. The client re-registers with the SEPM during startup

What to look for

1. Verify the config.xml was downloaded to the client. This can be seen in the CVE.log or by comparing checksum in client registry HKLM\SOFTWARE\Wow6432Node\Symantec\Symantec Endpoint Protection\SMC\SYLINK\SyLink\ClientConfigFileChecksum with the Config.xml on the SEPM here "C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\outbox\agent\GroupFolderGuid".

2. Check if the delta or full package was received. This can be done by reviewing the Debug.log on the client. Alternatively, check the "C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\VersionNumber\SmcLU" for install files. This directory should contain the last unzipped delta or full package. If the package is a delta package, and files exist here, it is likely that patchwrap.exe was able to successfully rebuild the patched files.

3. Look for the smcinst.log underneath Install Dir/smcLU. If that log file exists, smcinst.exe was run and MsiInstaller was launched. If this is the case troubleshoot the install like you would any other SEP Client installation.

4. Full packages are requested by CVE ONLY in these scenarios: When SMC is unable to install the delta package and when the client's base version does not exist on the server.