Skip to main content

Agent Troubleshooting

How to troubleshoot Agent Offline Issues​

Before commencing the Steps below, please double check that the time on the device has been set correctly and there is no clock drift.

  1. Restart the OrchestratorAgent Service. Check if the Agent is now online. if not, move on to the next step below.
  2. Reboot the endpoint. Check if the Agent is now online. if not, move on to the next step below.
  3. Re-install the Pia Agent using the -clearcertificate flag. Check if the Agent is now online. if not, move on to the next step below.
  4. Uninstall the Pia Agent using the -clearcertificate flag.
  5. Download Sysinternals onto the device - https://download.sysinternals.com/files/SysinternalsSuite.zip
  6. Extract Sysinternals from the .zip file.
  7. Start Powershell as Administrator.
  8. Change the working directory to where the contents of the Sysinternals .zip file was extracted to.
  9. Run the following command from the Administrator Powershell Window: .\PsExec.exe -i -s cmd.exe
  10. A cmd.exe window should now appear on the screen, reviewing Task Manager will show that the cmd.exe process will be running under the "SYSTEM" user.
  11. Run the following command from the System cmd.exe window: certutil -user -v -csp "Microsoft Software Key Storage Provider" -key "OrchestratorAgent" This should print out the key to the console if it is present, take a screenshot of the result.
  12. In the same cmd.exe window running as system account run the following command: certutil -user -v -csp "Microsoft Software Key Storage Provider" -delkey "OrchestratorAgent" This should delete the key if it is present, take a screenshot of the result.
  13. Run the first command again(listed below) and take a screenshot of the result. certutil -user -v -csp "Microsoft Software Key Storage Provider" -key "OrchestratorAgent"
  14. Follow the normal procedure for installing the Pia Agent.
  15. Repeat Step 13.
  16. Check in the Agents section of the Pia Portal if the device status is now showing as “Online” in the list.
  17. If the agent is still showing as offline after 5-10mins, please copy the 3 following files from the server(if multiple servers offline, please collect the files for each)

All 3 files are in C:\Program Files (x86)\OrchestratorAgent

• Config.json • Logfile.txt • Logfile_installer.txt

  1. At this point, please review the affected environments and confirm the following:
    • Does the client have a HTTPS proxy or web caching solution in their environments?
    • Does the client have a MITM or SSL inspection solution in their environments?
    • Does the client have outbound/inbound HTTPS traffic restrictions?
  2. If the answer to any of the above is Yes, then continue with the troubleshooting steps below. If the answer is No, then skip to Step 32.
  3. In this case, you will need to whitelist Pia inside the applicable solution in the client's environment. This will involve whitelisting two components:
    • Your Pia Tenant URL/FQDN
    • IoT Hub FQDN (See Instructions below)
  4. Open the config.json file located in C:\Program Files (x86)\OrchestratorAgent
  5. Copy the FQDN value from the “IotHostName” line (See screenshot below).

agents_3.png

  1. Add both to the whitelist in your solution.
  2. Restart the Orchestrator Agent service.
  3. Review the Pia Portal Agent page to see if the Agent is now online. If the Agent is still not online, Please continue with the following.
  4. Copy the IoT Hub FQDN same as in Step 8.
  5. Open Google Chrome or Chromium Edge and enter in the IoTHostName FQDN, pre-fixed with “https://” – it will have an error like in the screenshot below, this is normal.
  6. Inspect the certificate as shown in the example below showing the certificate chain.

agents_4.png

  1. Review the 'Issued To' and 'Issued By' sections, both should reference Microsoft and/or Azure. If you find that it references something else i.e. localhost or the device name or another solution provider, either the whitelisting has not been applied correctly or there is another SSL inspection/ MITM Solution in the environment.
  2. Review the results and actions accordingly. If the SSL certificate is now displaying correctly but your device still offline, please continue with the steps below.
  3. Restart the service and then check the Agents page in the Pia portal.
  4. We have observed in some circumstances where cipher suites and/or Schannel settings on a device may impact the connectivity of the Agent. To troubleshoot and review the current settings, please continue with the steps below.
  5. On the affected device, download and run IISCrypto from https://www.nartac.com/Products/IISCrypto/Download
  6. Run IISCrypto and review the Schannel and Cipher Suites tabs as displayed in the example below.

agent_5.png agent_6.png

  1. Our IoT Hub accepts TLS 1.2 with the following Cipher Suites:
  • TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  • TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
  • TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
  • TLS_RSA_WITH_AES_256_GCM_SHA384
  • TLS_RSA_WITH_AES_128_GCM_SHA256
  • TLS_RSA_WITH_AES_256_CBC_SHA256
  • TLS_RSA_WITH_AES_128_CBC_SHA256
  • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
  • TLS_RSA_WITH_AES_256_CBC_SHA
  • TLS_RSA_WITH_AES_128_CBC_SHA

For more information, please refer to the links below:

  1. If you are missing any Cipher Suites listed above, or TLS 1.2 is not enabled on the device, our recommendation is to enable them.

  2. If you're still experiencing Agent offline issues, please follow the steps in the "How to change the Agent Communication Protocol" section below.

How to change the Agent Communication Protocol​

  1. Stop the Orchestrator service

  2. Navigate to the installation directory of Pia which is C:\Program Files (x86)\OrchestratorAgent

  3. Open the config.json file in this directory

  4. There are multiple communication protocols which can be set. Our recommendation is to set each one in order as per the list below, until the agent comes online or all communication protocol methods have been attempted.

    • "IOTHubConnectionMode" : "AMQP"
    • "IOTHubConnectionMode" : "AMQP_GENERIC"
    • "IOTHubConnectionMode" : "MQTT_GENERIC"
info

Note: Please ensure only one connection mode is set in the file at a time

agent_1.png

  1. Add in the communication protocol and then Save the config.json file
  2. Start the Orchestrator Agent Service. See line 69 in the screenshot below showing the Agent communicating with the AMQP Protocol as an example of the communication method switching.

agent_2.png

  1. Open the logfile.txt located in C:\Program Files (x86)\OrchestratorAgent and review and confirm if you are able to see 'Success' or 'Connection Attempt' as shown in line 80 in the screenshot above.
info

Note: If you do not see a successful connection attempt, repeat from Step 1 and choose a different communication protocol method.

  1. Review Agents List in Pia portal to confirm that the Agent is online.
info

Note: If you do not see that the agent is online after a few minutes, repeat from Step 1 and choose a different communication protocol method.

  1. Try executing some packages against the endpoint.

  2. If you are still experiencing difficulties, please reach out to our Partner Support Team or if you are still going through implementation, please contact the PMO Team. In your email, please include the following:

a. The following 3 files from C:\Program Files (x86)\OrchestratorAgent on the affected device:

• Config.json • Logfile.txt • Logfile_installer.txt

b. Screenshots of the Schannel and Ciphers screens from IISCrypto on the affected device.

c. Screenshots of the Sysinternals output from Steps 11, 13 & 15.

d. Screenshot of the SSL certificate info from the device from Step 28.

How to troubleshoot missing DLL error message​

When trying to install the Pia Agent on a machine you receive a missing DLL error message.

agent_install_failure_missing_win_crt_dll.png

OrchestratorAgent.exe - System Error The program can't start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your computer. Try reinstalling the program to fix this problem.

Explanation

The Pia Agent requires the Windows Universal C Runtime library that was introduced in Windows 10 and Visual Studio 2015.

This library is available for Windows systems before Windows 10 (back to Windows Vista and Windows Server 2008) but may need to be installed manually.

Solution

Your Windows system is missing the Universal C Runtime library, and you will need to trigger the installation manually.

How to install Windows Universal C Runtime There are 2 ways to install the Windows Universal C Runtime on older Windows systems:

  1. Windows Update If you are using a supported Windows version, the Windows Universal C Runtime should appear as an option to install in Windows Update. (If you don't see it available in Windows Update try the 'Microsoft Download Center' method.)
  1. Microsoft Download Center You can also download the library from Microsoft Download Center and install it manually.
    1. Visit the [Update for Universal C Runtime in Windows](https://support.microsoft.com/en-us/topic/update-for-universal-c-runtime-in-windows-c0514201-7fe6-95a3-b0a5-287930f3560c) page.
    1. Next to your version of Windows click the '**Download the package now**' link.
    1. On the Windows Download Center page that opens click the '**Download**' button.
    1. Run the '.msu' installed that is downloaded to install the Windows C Runtime.
info

You may need to restart your machine after installing the Windows Universal C Runtime before running the Pia Agent installer again.

After installing Windows Universal C Runtime You can now try installing the Pia Agent again.