[GH-ISSUE #262] services crash when installed along-side other software #170

Closed
opened 2026-03-02 02:14:09 +03:00 by kerem · 30 comments
Owner

Originally created by @bbrendon on GitHub (Jan 27, 2021).
Original GitHub issue: https://github.com/amidaware/tacticalrmm/issues/262

We've been having a problem for the last few weeks where services start crashing randomly on servers. The reason this got so much attention internally is because dhcpserver kept crashing. Some of the services that crash are:
nxlog, dhcpserver, Wecsvc, Schedule, lmhosts, eventlog

The most popular service to crash is nxlog

EDIT: I should add that by "crashing", I mean it didn't recover. It seems that after digging further many services crash but only a few don't recover.

This has been seen on OS: 2008r2, 2016std, win10pro, sbs2011

All of these servers also have another program that collects event logs which doesn't crash.

Timeline from today's crashes
1/27/2021 - 09:30 - updated Tactical from 0.4.0 to 0.4.1 (agent 1.3 to 1.4)
1/27/2021 - 09:47 - Monitoring triggered 4x servers, all different customers, all at about exactly the same time.

Since this seems related to nxlog + Tactical , my next step is going to see which version of nxlog these machines have and see if I can predict it in the future based on nxlog being installed and versions.

Originally created by @bbrendon on GitHub (Jan 27, 2021). Original GitHub issue: https://github.com/amidaware/tacticalrmm/issues/262 We've been having a problem for the last few weeks where services start crashing randomly on servers. The reason this got so much attention internally is because dhcpserver kept crashing. Some of the services that crash are: nxlog, dhcpserver, Wecsvc, Schedule, lmhosts, eventlog The most popular service to crash is nxlog EDIT: I should add that by "crashing", I mean it didn't recover. It seems that after digging further many services crash but only a few don't recover. This has been seen on OS: 2008r2, 2016std, win10pro, sbs2011 All of these servers also have another program that collects event logs which doesn't crash. Timeline from today's crashes 1/27/2021 - 09:30 - updated Tactical from 0.4.0 to 0.4.1 (agent 1.3 to 1.4) 1/27/2021 - 09:47 - Monitoring triggered 4x servers, all different customers, all at about exactly the same time. Since this seems related to nxlog + Tactical , my next step is going to see which version of nxlog these machines have and see if I can predict it in the future based on nxlog being installed and versions.
kerem closed this issue 2026-03-02 02:14:09 +03:00
Author
Owner

@bbrendon commented on GitHub (Jan 27, 2021):

After looking through the System event logs on h-dc2 a bunch of services are stopped immediately after the tactical agent stops which I'm presuming is because of the 1.3.0 to 1.4.0 agent update. Many are "stopped" and many are terminated unexpectedly. By "many", I mean about 20+ services. Somehow the server didn't totally melt-down.

After all the stops (error and informational events), tactical starts and other services begin starting again. But it seems not all services recover fully.

<!-- gh-comment-id:768504952 --> @bbrendon commented on GitHub (Jan 27, 2021): After looking through the System event logs on h-dc2 a bunch of services are stopped immediately after the tactical agent stops which I'm presuming is because of the 1.3.0 to 1.4.0 agent update. Many are "stopped" and many are terminated unexpectedly. By "many", I mean about 20+ services. Somehow the server didn't totally melt-down. After all the stops (error and informational events), tactical starts and other services begin starting again. But it seems not all services recover fully.
Author
Owner

@bbrendon commented on GitHub (Jan 27, 2021):

I inspected the System Event log on h-dc1 (does not have nxlog). It looks much better. no errors. There are a few services that start and stop around the time of the tactical update but they seem be doing it by design.
Services restarted : windows module installer service, appxsvc, diagnostic host service

No critical services were restarted.

<!-- gh-comment-id:768511900 --> @bbrendon commented on GitHub (Jan 27, 2021): I inspected the System Event log on h-dc1 (does not have nxlog). It looks much better. no errors. There are a few services that start and stop around the time of the tactical update but they seem be doing it by design. Services restarted : windows module installer service, appxsvc, diagnostic host service No critical services were restarted.
Author
Owner

@rtwright68 commented on GitHub (Jan 27, 2021):

Funny, we have been seeing the same exact thing. The same services you mentioned above are what we are observing. I haven't updated yet to 0.41 since I have some agents that haven't updated from 1.12.

<!-- gh-comment-id:768555059 --> @rtwright68 commented on GitHub (Jan 27, 2021): Funny, we have been seeing the same exact thing. The same services you mentioned above are what we are observing. I haven't updated yet to 0.41 since I have some agents that haven't updated from 1.12.
Author
Owner

@bbrendon commented on GitHub (Jan 27, 2021):

@rtwright68 Are you running nxlog as well? This has been going on for a few weeks. Its not specific to agent 1.4.0

<!-- gh-comment-id:768627372 --> @bbrendon commented on GitHub (Jan 27, 2021): @rtwright68 Are you running nxlog as well? This has been going on for a few weeks. Its not specific to agent 1.4.0
Author
Owner

@bbrendon commented on GitHub (Jan 27, 2021):

I just saw someone post this in the chat. This looks exactly like it! Well, not exactly, but crazy similar.

image

<!-- gh-comment-id:768652482 --> @bbrendon commented on GitHub (Jan 27, 2021): I just saw someone post this in the chat. This looks exactly like it! Well, not exactly, but crazy similar. ![image](https://user-images.githubusercontent.com/6364477/106068979-6c2af100-60b6-11eb-9a8c-3bc481750c27.png)
Author
Owner

@tektrak commented on GitHub (Jan 29, 2021):

I've seen the same thing. Didn't really notice the details until last night when updating Tactical RMM Agents since the updates run in the background. However, the log file reveals this issue. One server, a Windows Remote Desktop Host, showed some files in use by a number of services. The updater tried to kill the service processes, but had problems, aborted the install, and rolled back. However, a number of services did suffer in the process. Attached is the contents of the tacticalrmm.txt file from C:\Windows\temp\tacticalrmmxxxxxxxxx (although the line endings of this file may have changed in the transfers between systems).
tacticalrmm.txt

It seems like it would be preferable to delay completion of the upgrade until after a reboot rather than killing off things that might be using the files.

By the way, we are not running nxlog.

<!-- gh-comment-id:769547864 --> @tektrak commented on GitHub (Jan 29, 2021): I've seen the same thing. Didn't really notice the details until last night when updating Tactical RMM Agents since the updates run in the background. However, the log file reveals this issue. One server, a Windows Remote Desktop Host, showed some files in use by a number of services. The updater tried to kill the service processes, but had problems, aborted the install, and rolled back. However, a number of services did suffer in the process. Attached is the contents of the tacticalrmm.txt file from C:\Windows\temp\tacticalrmmxxxxxxxxx (although the line endings of this file may have changed in the transfers between systems). [tacticalrmm.txt](https://github.com/wh1te909/tacticalrmm/files/5891370/tacticalrmm.txt) It seems like it would be preferable to delay completion of the upgrade until after a reboot rather than killing off things that might be using the files. By the way, we are not running nxlog.
Author
Owner

@wh1te909 commented on GitHub (Jan 29, 2021):

ive changed the inno setup executable in agent 1.4.1 to restart applications that were closed during an update
as a test can you guys please try the following:

un-check auto agent update in Settings > Global Settings
update rmm to 0.4.2 if not done so already via update.sh script
download agent 1.4.1 from https://github.com/wh1te909/rmmagent/releases/download/v1.4.1/winagent-v1.4.1.exe
put it somewhere on your agents filesystem then open cmd as admin, cd to the directory of the exe and call it like this please
winagent-v1.4.1.exe /VERYSILENT /SUPPRESSMSGBOXES /LOG=test123.txt
make note of the services that were being stopped but not restarting and see if now they restart and also paste the test123.txt here for me to see. thx.

<!-- gh-comment-id:770095914 --> @wh1te909 commented on GitHub (Jan 29, 2021): ive changed the inno setup executable in agent 1.4.1 to restart applications that were closed during an update as a test can you guys please try the following: un-check auto agent update in Settings > Global Settings update rmm to 0.4.2 if not done so already via update.sh script download agent 1.4.1 from https://github.com/wh1te909/rmmagent/releases/download/v1.4.1/winagent-v1.4.1.exe put it somewhere on your agents filesystem then open cmd as admin, cd to the directory of the exe and call it like this please `winagent-v1.4.1.exe /VERYSILENT /SUPPRESSMSGBOXES /LOG=test123.txt` make note of the services that were being stopped but not restarting and see if now they restart and also paste the test123.txt here for me to see. thx.
Author
Owner

@tektrak commented on GitHub (Jan 30, 2021):

On the client server running Windows Server 2012 R2 Standard that had problems last time, I ran the attempted upgrade from 1.4.0 to 1.4.1 manually as described above. The ugprade failed and afterwards some services were not restarted. Here is the status of the services after this attempt for the services indicated in the log file attached:

Status   Name               DisplayName
------   ----               -----------
Stopped  Audiosrv           Windows Audio
Running  Dhcp               DHCP Client
Running  EventLog           Windows Event Log
Running  lmhosts            TCP/IP NetBIOS Helper
Stopped  Wcmsvc             Windows Connection Manager
Stopped  iphlpsvc           IP Helper
Running  WinHttpAutoProx... WinHTTP Web Proxy Auto-Discovery Se...
Running  netprofm           Network List Service
Running  NlaSvc             Network Location Awareness

After noting the status, I manually restarted the stopped services without issue.
test123.txt

<!-- gh-comment-id:770116397 --> @tektrak commented on GitHub (Jan 30, 2021): On the client server running Windows Server 2012 R2 Standard that had problems last time, I ran the attempted upgrade from 1.4.0 to 1.4.1 manually as described above. The ugprade failed and afterwards some services were not restarted. Here is the status of the services after this attempt for the services indicated in the log file attached: ``` Status Name DisplayName ------ ---- ----------- Stopped Audiosrv Windows Audio Running Dhcp DHCP Client Running EventLog Windows Event Log Running lmhosts TCP/IP NetBIOS Helper Stopped Wcmsvc Windows Connection Manager Stopped iphlpsvc IP Helper Running WinHttpAutoProx... WinHTTP Web Proxy Auto-Discovery Se... Running netprofm Network List Service Running NlaSvc Network Location Awareness ``` After noting the status, I manually restarted the stopped services without issue. [test123.txt](https://github.com/wh1te909/tacticalrmm/files/5896996/test123.txt)
Author
Owner

@wh1te909 commented on GitHub (Jan 30, 2021):

@tektrak thanks can u try now same exe call it like this plz
winagent-v1.4.1.exe /VERYSILENT /FORCECLOSEAPPLICATIONS /LOG=testforceclose.txt

<!-- gh-comment-id:770122543 --> @wh1te909 commented on GitHub (Jan 30, 2021): @tektrak thanks can u try now same exe call it like this plz `winagent-v1.4.1.exe /VERYSILENT /FORCECLOSEAPPLICATIONS /LOG=testforceclose.txt`
Author
Owner

@tektrak commented on GitHub (Jan 30, 2021):

The agent seems to have updated now. Attached is the log file.
testforceclose.txt

<!-- gh-comment-id:770137973 --> @tektrak commented on GitHub (Jan 30, 2021): The agent seems to have updated now. Attached is the log file. [testforceclose.txt](https://github.com/wh1te909/tacticalrmm/files/5897190/testforceclose.txt)
Author
Owner

@wh1te909 commented on GitHub (Jan 30, 2021):

thanks. seems this time there were no applications in use so can you keep trying until u can get the log to show RestartManager found an application using one of our files ... and then i want to see if it will restart them. the original log file says it was aborted, which is the default when using the /SUPPRESSMSGBOXES flag so was hoping that removing that flag would make it retry

Although i still dont understand why its saying it found applications in use. the inno setup exe stops and kill all tacticalrmm.exe processes before it starts to do the update so no sure whats going on. tactical does not interact with any of those services especially nxlog, never even heard of that. and am not able to reproduce on any of my agents

<!-- gh-comment-id:770149596 --> @wh1te909 commented on GitHub (Jan 30, 2021): thanks. seems this time there were no applications in use so can you keep trying until u can get the log to show `RestartManager found an application using one of our files ...` and then i want to see if it will restart them. the original log file says it was aborted, which is the default when using the /SUPPRESSMSGBOXES flag so was hoping that removing that flag would make it retry Although i still dont understand why its saying it found applications in use. the inno setup exe stops and kill all tacticalrmm.exe processes before it starts to do the update so no sure whats going on. tactical does not interact with any of those services especially nxlog, never even heard of that. and am not able to reproduce on any of my agents
Author
Owner

@tektrak commented on GitHub (Jan 30, 2021):

I'll work through other updates and note any issues.

I just tried a Windows 10 Pro workstation that seemed to be stuck on v1.1.11.
So I tried running

winagent-v1.1.2.exe /VERYSILENT /SUPPRESSMSGBOXES /LOG=log-v1.1.2.txt

It failed. Attached is the log.
log-v1.1.2.txt

<!-- gh-comment-id:770150380 --> @tektrak commented on GitHub (Jan 30, 2021): I'll work through other updates and note any issues. I just tried a Windows 10 Pro workstation that seemed to be stuck on v1.1.11. So I tried running ``` winagent-v1.1.2.exe /VERYSILENT /SUPPRESSMSGBOXES /LOG=log-v1.1.2.txt ``` It failed. Attached is the log. [log-v1.1.2.txt](https://github.com/wh1te909/tacticalrmm/files/5897316/log-v1.1.2.txt)
Author
Owner

@wh1te909 commented on GitHub (Jan 30, 2021):

u need to use 1.1.12, not 1.1.2

<!-- gh-comment-id:770153111 --> @wh1te909 commented on GitHub (Jan 30, 2021): u need to use 1.1.12, not 1.1.2
Author
Owner

@tektrak commented on GitHub (Jan 30, 2021):

Oops. I'll retry that one.

But here is the output from a Windows Server 2016 Standard that is a Windows Remote Desktop Host being upgraded from agent v1.4.0 to v1.4.1
.\winagent-v1.4.1.exe /VERYSILENT /SUPPRESSMSGBOXES /LOG=log-v1.4.1.txt
Here are the services that got stopped:

diff *before2* *after2*
10c10
< Running  Appinfo            Application Information               
---
> Stopped  Appinfo            Application Information               
14c14
< Running  AppXSvc            AppX Deployment Service (AppXSVC)     
---
> Stopped  AppXSvc            AppX Deployment Service (AppXSVC)     
30c30
< Running  CertPropSvc        Certificate Propagation               
---
> Stopped  CertPropSvc        Certificate Propagation               
39c39
< Stopped  DeviceAssociati... Device Association Service            
---
> Running  DeviceAssociati... Device Association Service            
61c61
< Stopped  fdPHost            Function Discovery Provider Host      
---
> Running  fdPHost            Function Discovery Provider Host      
74c74
< Running  iphlpsvc           IP Helper                             
---
> Stopped  iphlpsvc           IP Helper                             
92c92
< Running  MSMQ               Message Queuing                       
---
> Stopped  MSMQ               Message Queuing                       
98c98
< Running  NetMsmqActivator   Net.Msmq Listener Adapter             
---
> Stopped  NetMsmqActivator   Net.Msmq Listener Adapter             
149c149
< Running  SENS               System Event Notification Service     
---
> Stopped  SENS               System Event Notification Service     
239c239
< Running  WinRM              Windows Remote Management (WS-Manag...
---
> Stopped  WinRM              Windows Remote Management (WS-Manag...
241c241
< Running  wlidsvc            Microsoft Account Sign-in Assistant   
---
> Stopped  wlidsvc            Microsoft Account Sign-in Assistant   
244c244
< Running  WpnService         Windows Push Notifications System S...
---
> Stopped  WpnService         Windows Push Notifications System S...

log-v1.4.1.txt

<!-- gh-comment-id:770154111 --> @tektrak commented on GitHub (Jan 30, 2021): Oops. I'll retry that one. But here is the output from a Windows Server 2016 Standard that is a Windows Remote Desktop Host being upgraded from agent v1.4.0 to v1.4.1 `.\winagent-v1.4.1.exe /VERYSILENT /SUPPRESSMSGBOXES /LOG=log-v1.4.1.txt` Here are the services that got stopped: ``` diff *before2* *after2* 10c10 < Running Appinfo Application Information --- > Stopped Appinfo Application Information 14c14 < Running AppXSvc AppX Deployment Service (AppXSVC) --- > Stopped AppXSvc AppX Deployment Service (AppXSVC) 30c30 < Running CertPropSvc Certificate Propagation --- > Stopped CertPropSvc Certificate Propagation 39c39 < Stopped DeviceAssociati... Device Association Service --- > Running DeviceAssociati... Device Association Service 61c61 < Stopped fdPHost Function Discovery Provider Host --- > Running fdPHost Function Discovery Provider Host 74c74 < Running iphlpsvc IP Helper --- > Stopped iphlpsvc IP Helper 92c92 < Running MSMQ Message Queuing --- > Stopped MSMQ Message Queuing 98c98 < Running NetMsmqActivator Net.Msmq Listener Adapter --- > Stopped NetMsmqActivator Net.Msmq Listener Adapter 149c149 < Running SENS System Event Notification Service --- > Stopped SENS System Event Notification Service 239c239 < Running WinRM Windows Remote Management (WS-Manag... --- > Stopped WinRM Windows Remote Management (WS-Manag... 241c241 < Running wlidsvc Microsoft Account Sign-in Assistant --- > Stopped wlidsvc Microsoft Account Sign-in Assistant 244c244 < Running WpnService Windows Push Notifications System S... --- > Stopped WpnService Windows Push Notifications System S... ``` [log-v1.4.1.txt](https://github.com/wh1te909/tacticalrmm/files/5897353/log-v1.4.1.txt)
Author
Owner

@wh1te909 commented on GitHub (Jan 30, 2021):

replace /SUPPRESSMSGBOXES with /FORCECLOSEAPPLICATIONS when u call the exe
suppress will by default abort. i want to see if without that flag if will restart them

<!-- gh-comment-id:770155747 --> @wh1te909 commented on GitHub (Jan 30, 2021): replace `/SUPPRESSMSGBOXES` with `/FORCECLOSEAPPLICATIONS` when u call the exe suppress will by default abort. i want to see if without that flag if will restart them
Author
Owner

@bbrendon commented on GitHub (Jan 30, 2021):

Ran.... winagent-v1.4.1.exe /VERYSILENT /FORCECLOSEAPPLICATIONS /LOG=log-v1.4.1.txt

After a some seconds, this dialogue box appeared.

image

...I tried selecting "try again", but that didn't do anything so I selected ignore and continue.

After all was said and done, the list of automatic services that were running before and after the agent upgrade did not change. The list is below. It appears though that some of these should be running and were not. I think this was because the server was never rebooted after the last agent upgrade debacle.

clr_optimization_v4.0.30319_32
clr_optimization_v4.0.30319_64
lmhosts
Schedule
SolutionreachService
sppsvc
Wecsvc

Log below.

2021-01-29 21:30:30.897   Log opened. (Time zone: UTC-08:00)
2021-01-29 21:30:30.897   Setup version: Inno Setup version 6.1.2
2021-01-29 21:30:30.897   Original Setup EXE: C:\Users\user\Downloads\winagent-v1.4.1.exe
2021-01-29 21:30:30.897   Setup command line: /SL5="$4038A,3456399,824832,C:\Users\user\Downloads\winagent-v1.4.1.exe" /VERYSILENT /FORCECLOSEAPPLICATIONS /LOG=log-v1.4.1.txt
2021-01-29 21:30:30.897   Windows version: 6.1.7601 SP1  (NT platform: Yes)
2021-01-29 21:30:30.897   64-bit Windows: Yes
2021-01-29 21:30:30.897   Processor architecture: x64
2021-01-29 21:30:30.897   User privileges: Administrative
2021-01-29 21:30:30.900   Administrative install mode: Yes
2021-01-29 21:30:30.900   Install mode root key: HKEY_LOCAL_MACHINE
2021-01-29 21:30:30.900   64-bit install mode: No
2021-01-29 21:30:30.905   Created temporary directory: C:\Users\user\AppData\Local\Temp\is-FH4M2.tmp
2021-01-29 21:30:33.586   Stop tacticalagent: 0
2021-01-29 21:30:36.448   Stop tacticalrpc: 0
2021-01-29 21:30:36.656   taskkill: 128
2021-01-29 21:30:36.757   Found 2 files to register with RestartManager.
2021-01-29 21:30:36.757   Calling RestartManager's RmGetList.
2021-01-29 21:30:36.803   RmGetList finished successfully.
2021-01-29 21:30:36.803   RestartManager found an application using one of our files: DHCP Client
2021-01-29 21:30:36.803   RestartManager found an application using one of our files: Windows Event Log
2021-01-29 21:30:36.803   RestartManager found an application using one of our files: WinHTTP Web Proxy Auto-Discovery Service
2021-01-29 21:30:36.803   RestartManager found an application using one of our files: nxlog
2021-01-29 21:30:36.803   RestartManager found an application using one of our files: DHCP Server
2021-01-29 21:30:36.803   Can use RestartManager to avoid reboot? Yes (0)
2021-01-29 21:30:36.814   Starting the installation process.
2021-01-29 21:30:36.816   Shutting down applications using our files. (forced)
2021-01-29 21:30:49.358   Some applications could not be shut down.
2021-01-29 21:30:49.358   Message box (Abort/Retry/Ignore):
                          Setup was unable to automatically close all applications. It is recommended that you close all applications using files that need to be updated by Setup before continuing.
2021-01-29 21:31:11.630   User chose Retry.
2021-01-29 21:31:11.630   Retrying to shut down applications using our files. (forced)
2021-01-29 21:31:11.647   Some applications could not be shut down.
2021-01-29 21:31:11.647   Message box (Abort/Retry/Ignore):
                          Setup was unable to automatically close all applications. It is recommended that you close all applications using files that need to be updated by Setup before continuing.
2021-01-29 21:31:13.687   User chose Ignore.
2021-01-29 21:31:13.688   Directory for uninstall files: C:\Program Files\TacticalAgent
2021-01-29 21:31:13.688   Will append to existing uninstall log: C:\Program Files\TacticalAgent\unins000.dat
2021-01-29 21:31:13.688   -- File entry --
2021-01-29 21:31:13.688   Dest filename: C:\Program Files\TacticalAgent\unins000.exe
2021-01-29 21:31:13.695   Time stamp of our file: 2021-01-29 21:30:30.551
2021-01-29 21:31:13.695   Dest file exists.
2021-01-29 21:31:13.695   Time stamp of existing file: 2021-01-27 09:35:20.972
2021-01-29 21:31:13.695   Version of our file: 51.1052.0.0
2021-01-29 21:31:13.719   Version of existing file: 51.1052.0.0
2021-01-29 21:31:13.719   Installing the file.
2021-01-29 21:31:13.727   Leaving temporary file in place for now.
2021-01-29 21:31:13.728   -- File entry --
2021-01-29 21:31:13.728   Dest filename: C:\Program Files\TacticalAgent\tacticalrmm.exe
2021-01-29 21:31:13.728   Time stamp of our file: 2021-01-29 00:05:16.000
2021-01-29 21:31:13.728   Dest file exists.
2021-01-29 21:31:13.728   Time stamp of existing file: 2021-01-26 23:27:08.000
2021-01-29 21:31:13.728   Installing the file.
2021-01-29 21:31:14.492   Successfully installed the file.
2021-01-29 21:31:14.492   -- File entry --
2021-01-29 21:31:14.492   Dest filename: C:\Program Files\TacticalAgent\nssm.exe
2021-01-29 21:31:14.493   Time stamp of our file: 2020-12-15 01:27:00.000
2021-01-29 21:31:14.493   Dest file exists.
2021-01-29 21:31:14.493   Time stamp of existing file: 2020-12-15 01:27:00.000
2021-01-29 21:31:14.493   Installing the file.
2021-01-29 21:31:14.528   Successfully installed the file.
2021-01-29 21:31:14.528   -- Icon entry --
2021-01-29 21:31:14.528   Dest filename: C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Tactical RMM Agent.lnk
2021-01-29 21:31:14.529   Creating the icon.
2021-01-29 21:31:14.638   Successfully created the icon.
2021-01-29 21:31:14.641   Saving uninstall information.
2021-01-29 21:31:14.641   Renaming uninstaller.
2021-01-29 21:31:14.643   Deleting uninstall key left over from previous administrative 32-bit install.
2021-01-29 21:31:14.645   Creating new uninstall key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall\{0D34D278-5FAF-4159-A4A0-4E2D2C08139D}_is1
2021-01-29 21:31:14.645   Writing uninstall key values.
2021-01-29 21:31:14.652   Detected previous non administrative install? No
2021-01-29 21:31:14.652   Detected previous administrative 64-bit install? No
2021-01-29 21:31:14.779   Installation process succeeded.
2021-01-29 21:31:14.779   Attempting to restart applications.
2021-01-29 21:31:27.464   Need to restart Windows? No
2021-01-29 21:31:27.468   Deinitializing Setup.
2021-01-29 21:31:31.747   Start tacticalagent: 0
2021-01-29 21:31:33.943   Start tacticalrpc: 0
2021-01-29 21:31:33.949   Log closed.

<!-- gh-comment-id:770162123 --> @bbrendon commented on GitHub (Jan 30, 2021): Ran.... `winagent-v1.4.1.exe /VERYSILENT /FORCECLOSEAPPLICATIONS /LOG=log-v1.4.1.txt` After a some seconds, this dialogue box appeared. ![image](https://user-images.githubusercontent.com/6364477/106348214-692c2e00-6279-11eb-89b4-acdca5c7c76e.png) ...I tried selecting "try again", but that didn't do anything so I selected ignore and continue. After all was said and done, the list of automatic services that were running before and after the agent upgrade did not change. The list is below. It appears though that some of these should be running and were not. I think this was because the server was never rebooted after the last agent upgrade debacle. ``` clr_optimization_v4.0.30319_32 clr_optimization_v4.0.30319_64 lmhosts Schedule SolutionreachService sppsvc Wecsvc ``` Log below. ``` 2021-01-29 21:30:30.897 Log opened. (Time zone: UTC-08:00) 2021-01-29 21:30:30.897 Setup version: Inno Setup version 6.1.2 2021-01-29 21:30:30.897 Original Setup EXE: C:\Users\user\Downloads\winagent-v1.4.1.exe 2021-01-29 21:30:30.897 Setup command line: /SL5="$4038A,3456399,824832,C:\Users\user\Downloads\winagent-v1.4.1.exe" /VERYSILENT /FORCECLOSEAPPLICATIONS /LOG=log-v1.4.1.txt 2021-01-29 21:30:30.897 Windows version: 6.1.7601 SP1 (NT platform: Yes) 2021-01-29 21:30:30.897 64-bit Windows: Yes 2021-01-29 21:30:30.897 Processor architecture: x64 2021-01-29 21:30:30.897 User privileges: Administrative 2021-01-29 21:30:30.900 Administrative install mode: Yes 2021-01-29 21:30:30.900 Install mode root key: HKEY_LOCAL_MACHINE 2021-01-29 21:30:30.900 64-bit install mode: No 2021-01-29 21:30:30.905 Created temporary directory: C:\Users\user\AppData\Local\Temp\is-FH4M2.tmp 2021-01-29 21:30:33.586 Stop tacticalagent: 0 2021-01-29 21:30:36.448 Stop tacticalrpc: 0 2021-01-29 21:30:36.656 taskkill: 128 2021-01-29 21:30:36.757 Found 2 files to register with RestartManager. 2021-01-29 21:30:36.757 Calling RestartManager's RmGetList. 2021-01-29 21:30:36.803 RmGetList finished successfully. 2021-01-29 21:30:36.803 RestartManager found an application using one of our files: DHCP Client 2021-01-29 21:30:36.803 RestartManager found an application using one of our files: Windows Event Log 2021-01-29 21:30:36.803 RestartManager found an application using one of our files: WinHTTP Web Proxy Auto-Discovery Service 2021-01-29 21:30:36.803 RestartManager found an application using one of our files: nxlog 2021-01-29 21:30:36.803 RestartManager found an application using one of our files: DHCP Server 2021-01-29 21:30:36.803 Can use RestartManager to avoid reboot? Yes (0) 2021-01-29 21:30:36.814 Starting the installation process. 2021-01-29 21:30:36.816 Shutting down applications using our files. (forced) 2021-01-29 21:30:49.358 Some applications could not be shut down. 2021-01-29 21:30:49.358 Message box (Abort/Retry/Ignore): Setup was unable to automatically close all applications. It is recommended that you close all applications using files that need to be updated by Setup before continuing. 2021-01-29 21:31:11.630 User chose Retry. 2021-01-29 21:31:11.630 Retrying to shut down applications using our files. (forced) 2021-01-29 21:31:11.647 Some applications could not be shut down. 2021-01-29 21:31:11.647 Message box (Abort/Retry/Ignore): Setup was unable to automatically close all applications. It is recommended that you close all applications using files that need to be updated by Setup before continuing. 2021-01-29 21:31:13.687 User chose Ignore. 2021-01-29 21:31:13.688 Directory for uninstall files: C:\Program Files\TacticalAgent 2021-01-29 21:31:13.688 Will append to existing uninstall log: C:\Program Files\TacticalAgent\unins000.dat 2021-01-29 21:31:13.688 -- File entry -- 2021-01-29 21:31:13.688 Dest filename: C:\Program Files\TacticalAgent\unins000.exe 2021-01-29 21:31:13.695 Time stamp of our file: 2021-01-29 21:30:30.551 2021-01-29 21:31:13.695 Dest file exists. 2021-01-29 21:31:13.695 Time stamp of existing file: 2021-01-27 09:35:20.972 2021-01-29 21:31:13.695 Version of our file: 51.1052.0.0 2021-01-29 21:31:13.719 Version of existing file: 51.1052.0.0 2021-01-29 21:31:13.719 Installing the file. 2021-01-29 21:31:13.727 Leaving temporary file in place for now. 2021-01-29 21:31:13.728 -- File entry -- 2021-01-29 21:31:13.728 Dest filename: C:\Program Files\TacticalAgent\tacticalrmm.exe 2021-01-29 21:31:13.728 Time stamp of our file: 2021-01-29 00:05:16.000 2021-01-29 21:31:13.728 Dest file exists. 2021-01-29 21:31:13.728 Time stamp of existing file: 2021-01-26 23:27:08.000 2021-01-29 21:31:13.728 Installing the file. 2021-01-29 21:31:14.492 Successfully installed the file. 2021-01-29 21:31:14.492 -- File entry -- 2021-01-29 21:31:14.492 Dest filename: C:\Program Files\TacticalAgent\nssm.exe 2021-01-29 21:31:14.493 Time stamp of our file: 2020-12-15 01:27:00.000 2021-01-29 21:31:14.493 Dest file exists. 2021-01-29 21:31:14.493 Time stamp of existing file: 2020-12-15 01:27:00.000 2021-01-29 21:31:14.493 Installing the file. 2021-01-29 21:31:14.528 Successfully installed the file. 2021-01-29 21:31:14.528 -- Icon entry -- 2021-01-29 21:31:14.528 Dest filename: C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Tactical RMM Agent.lnk 2021-01-29 21:31:14.529 Creating the icon. 2021-01-29 21:31:14.638 Successfully created the icon. 2021-01-29 21:31:14.641 Saving uninstall information. 2021-01-29 21:31:14.641 Renaming uninstaller. 2021-01-29 21:31:14.643 Deleting uninstall key left over from previous administrative 32-bit install. 2021-01-29 21:31:14.645 Creating new uninstall key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall\{0D34D278-5FAF-4159-A4A0-4E2D2C08139D}_is1 2021-01-29 21:31:14.645 Writing uninstall key values. 2021-01-29 21:31:14.652 Detected previous non administrative install? No 2021-01-29 21:31:14.652 Detected previous administrative 64-bit install? No 2021-01-29 21:31:14.779 Installation process succeeded. 2021-01-29 21:31:14.779 Attempting to restart applications. 2021-01-29 21:31:27.464 Need to restart Windows? No 2021-01-29 21:31:27.468 Deinitializing Setup. 2021-01-29 21:31:31.747 Start tacticalagent: 0 2021-01-29 21:31:33.943 Start tacticalrpc: 0 2021-01-29 21:31:33.949 Log closed. ```
Author
Owner

@tektrak commented on GitHub (Jan 30, 2021):

Here is a Windows Server 2012 R2 Standard that was upgraded successfully from v1.4.0 to v1.4.1.

.\winagent-v1.4.1.exe /VERYSILENT /FORCECLOSEAPPLICATIONS /LOG=log-v1.4.1.txt

All of the services mentioned:

2021-01-30 00:15:43.749   RestartManager found an application using one of our files: DHCP Client
2021-01-30 00:15:43.749   RestartManager found an application using one of our files: Windows Event Log
2021-01-30 00:15:43.749   RestartManager found an application using one of our files: TCP/IP NetBIOS Helper
2021-01-30 00:15:43.749   RestartManager found an application using one of our files: Windows Connection Manager
2021-01-30 00:15:43.749   RestartManager found an application using one of our files: IP Helper
2021-01-30 00:15:43.749   RestartManager found an application using one of our files: WinHTTP Web Proxy Auto-Discovery Service
2021-01-30 00:15:43.749   RestartManager found an application using one of our files: Network List Service
2021-01-30 00:15:43.749   RestartManager found an application using one of our files: Network Location Awareness
2021-01-30 00:15:43.749   RestartManager found an application using one of our files: Sage Service Host (v20.3)

were running after the upgrade.
log-v1.4.1.txt

<!-- gh-comment-id:770166604 --> @tektrak commented on GitHub (Jan 30, 2021): Here is a Windows Server 2012 R2 Standard that was upgraded successfully from v1.4.0 to v1.4.1. .\winagent-v1.4.1.exe /VERYSILENT /FORCECLOSEAPPLICATIONS /LOG=log-v1.4.1.txt All of the services mentioned: ``` 2021-01-30 00:15:43.749 RestartManager found an application using one of our files: DHCP Client 2021-01-30 00:15:43.749 RestartManager found an application using one of our files: Windows Event Log 2021-01-30 00:15:43.749 RestartManager found an application using one of our files: TCP/IP NetBIOS Helper 2021-01-30 00:15:43.749 RestartManager found an application using one of our files: Windows Connection Manager 2021-01-30 00:15:43.749 RestartManager found an application using one of our files: IP Helper 2021-01-30 00:15:43.749 RestartManager found an application using one of our files: WinHTTP Web Proxy Auto-Discovery Service 2021-01-30 00:15:43.749 RestartManager found an application using one of our files: Network List Service 2021-01-30 00:15:43.749 RestartManager found an application using one of our files: Network Location Awareness 2021-01-30 00:15:43.749 RestartManager found an application using one of our files: Sage Service Host (v20.3) ``` were running after the upgrade. [log-v1.4.1.txt](https://github.com/wh1te909/tacticalrmm/files/5897488/log-v1.4.1.txt)
Author
Owner

@rtwright68 commented on GitHub (Jan 30, 2021):

log-v141-1.txt
v141-2.txt

Ran on a couple Windows 2019 VMs that were stuck on 1.1.12. Still showing the old version number in the agent dashboard.

<!-- gh-comment-id:770197816 --> @rtwright68 commented on GitHub (Jan 30, 2021): [log-v141-1.txt](https://github.com/wh1te909/tacticalrmm/files/5897846/log-v141-1.txt) [v141-2.txt](https://github.com/wh1te909/tacticalrmm/files/5897847/v141-2.txt) Ran on a couple Windows 2019 VMs that were stuck on 1.1.12. Still showing the old version number in the agent dashboard.
Author
Owner

@rtwright68 commented on GitHub (Jan 30, 2021):

One other piece of info. Both of the agents I attempted the 1.4.1 update on are currently yellow. All services are up and running at this point, attempted a reboot on one of the agents.

<!-- gh-comment-id:770198886 --> @rtwright68 commented on GitHub (Jan 30, 2021): One other piece of info. Both of the agents I attempted the 1.4.1 update on are currently yellow. All services are up and running at this point, attempted a reboot on one of the agents.
Author
Owner

@tektrak commented on GitHub (Jan 30, 2021):

@rtwright68 I had some older agents on v1.1.11 and v1.1.12. The v1.1.11 agent I first updated to v1.1.12. Then updated the v1.1.12 agents to v1.2.0, then v1.3.0, then v1.4.1. I may not have needed to do all these intermediate steps, but I believe at least that you shouldn't skip v1.3.0 before going to v1.4.0 or v1.4.1.

<!-- gh-comment-id:770214161 --> @tektrak commented on GitHub (Jan 30, 2021): @rtwright68 I had some older agents on v1.1.11 and v1.1.12. The v1.1.11 agent I first updated to v1.1.12. Then updated the v1.1.12 agents to v1.2.0, then v1.3.0, then v1.4.1. I may not have needed to do all these intermediate steps, but I believe at least that you shouldn't skip v1.3.0 before going to v1.4.0 or v1.4.1.
Author
Owner

@wh1te909 commented on GitHub (Jan 30, 2021):

@tektrak yes that's correct, always need to update incremental based on the minor version number so you did good

@rtwright68 you need to uninstall those agents they are broken, straight upgrade from 1.1.12 to 1.4.1 will break the agents

<!-- gh-comment-id:770269860 --> @wh1te909 commented on GitHub (Jan 30, 2021): @tektrak yes that's correct, always need to update incremental based on the minor version number so you did good @rtwright68 you need to uninstall those agents they are broken, straight upgrade from 1.1.12 to 1.4.1 will break the agents
Author
Owner

@wh1te909 commented on GitHub (Jan 30, 2021):

are you guys able to this try with this exe please?

winagent-v1.3.555.zip

please run it like this and then upload the txt file

winagent-v1.3.555.exe /VERYSILENT /LOG=13555.txt

then wait like 15 seconds and check if the exe was replaced by running

"C:\Program Files\TacticalAgent\tacticalrmm.exe" -version

and see if it shows version 1.3.555
ive changed this exe to not close and not restart any services, since my theory is that those services are not in use anyway by tactical so no point in trying to close them

<!-- gh-comment-id:770296366 --> @wh1te909 commented on GitHub (Jan 30, 2021): are you guys able to this try with this exe please? [winagent-v1.3.555.zip](https://github.com/wh1te909/tacticalrmm/files/5898921/winagent-v1.3.555.zip) please run it like this and then upload the txt file ``` winagent-v1.3.555.exe /VERYSILENT /LOG=13555.txt ``` then wait like 15 seconds and check if the exe was replaced by running ``` "C:\Program Files\TacticalAgent\tacticalrmm.exe" -version ``` and see if it shows version 1.3.555 ive changed this exe to not close and not restart any services, since my theory is that those services are not in use anyway by tactical so no point in trying to close them
Author
Owner

@tektrak commented on GitHub (Jan 31, 2021):

Here's the first upgrade I tried on the first Windows Server 2012 R2 Standard server I mentioned above. It upgraded fine to the new version and connected to the TRMM server. The log is attached.

I'll also try it on a few more previously troublesome systems.
13555.txt

<!-- gh-comment-id:770310137 --> @tektrak commented on GitHub (Jan 31, 2021): Here's the first upgrade I tried on the first Windows Server 2012 R2 Standard server I mentioned above. It upgraded fine to the new version and connected to the TRMM server. The log is attached. I'll also try it on a few more previously troublesome systems. [13555.txt](https://github.com/wh1te909/tacticalrmm/files/5899051/13555.txt)
Author
Owner

@tektrak commented on GitHub (Jan 31, 2021):

And here's another run on the Windows Server 2016 Standard server mentioned above. It also upgraded fine to the new version and connected to the TRMM server. The log is attached.
13555.txt

No services were harmed in the making of these upgrades. Thanks!

<!-- gh-comment-id:770310986 --> @tektrak commented on GitHub (Jan 31, 2021): And here's another run on the Windows Server 2016 Standard server mentioned above. It also upgraded fine to the new version and connected to the TRMM server. The log is attached. [13555.txt](https://github.com/wh1te909/tacticalrmm/files/5899064/13555.txt) No services were harmed in the making of these upgrades. Thanks!
Author
Owner

@tektrak commented on GitHub (Jan 31, 2021):

Just ran the v1.3.555 agent upgrade test on 3 more servers including a Microsoft Windows Server Core 2016 without incident.

<!-- gh-comment-id:770315773 --> @tektrak commented on GitHub (Jan 31, 2021): Just ran the v1.3.555 agent upgrade test on 3 more servers including a Microsoft Windows Server Core 2016 without incident.
Author
Owner

@wh1te909 commented on GitHub (Jan 31, 2021):

@tektrak that's amazing wow! really hope this is the fix lol
@bbrendon can u try plz? if it works for you i'll get this released asap

<!-- gh-comment-id:770316588 --> @wh1te909 commented on GitHub (Jan 31, 2021): @tektrak that's amazing wow! really hope this is the fix lol @bbrendon can u try plz? if it works for you i'll get this released asap
Author
Owner

@bbrendon commented on GitHub (Jan 31, 2021):

Seems like it worked. Log https://pastebin.com/QpNs3m34
It just had an issue with nssm.exe

No service issues that I could see.

<!-- gh-comment-id:770322613 --> @bbrendon commented on GitHub (Jan 31, 2021): Seems like it worked. Log https://pastebin.com/QpNs3m34 It just had an issue with nssm.exe No service issues that I could see.
Author
Owner

@wh1te909 commented on GitHub (Feb 1, 2021):

@bbrendon thanks. I'll be getting rid of nssm eventually since it's no longer actively maintained, for now ive just changed the updater to not attempt to replace that file.

I'll be releasing an update to rmm shortly and with it agent v1.4.2
Since I changed the function inside the agent that handles agent update, when your agents update to 1.4.2 it will probably still attempt to force close those services since it will still be using the code from 1.4.1, so it won't be until the next agent after 1.4.2 until the issue is fully resolved so you might still need to manually update agents until they are all on 1.4.2

<!-- gh-comment-id:771083269 --> @wh1te909 commented on GitHub (Feb 1, 2021): @bbrendon thanks. I'll be getting rid of nssm eventually since it's no longer actively maintained, for now ive just changed the updater to not attempt to replace that file. I'll be releasing an update to rmm shortly and with it agent v1.4.2 Since I changed the function inside the agent that handles agent update, when your agents update to 1.4.2 it will probably still attempt to force close those services since it will still be using the code from 1.4.1, so it won't be until the next agent after 1.4.2 until the issue is fully resolved so you might still need to manually update agents until they are all on 1.4.2
Author
Owner

@tektrak commented on GitHub (Feb 2, 2021):

I have installed the rmm server update and more than half the agents (the half that are online now, including the servers) are now on the new v1.4.2. They seem to have updated without incident. I updated two servers via command line so that I could watch the process in more detail. The rest were manually updated via the web interface, as I currently have agent auto update disabled. Thanks for your efforts improving the update process!

<!-- gh-comment-id:771322907 --> @tektrak commented on GitHub (Feb 2, 2021): I have installed the rmm server update and more than half the agents (the half that are online now, including the servers) are now on the new v1.4.2. They seem to have updated without incident. I updated two servers via command line so that I could watch the process in more detail. The rest were manually updated via the web interface, as I currently have agent auto update disabled. Thanks for your efforts improving the update process!
Author
Owner

@bbrendon commented on GitHub (Feb 2, 2021):

Same here. No sirens went off. Looking good.

<!-- gh-comment-id:771339268 --> @bbrendon commented on GitHub (Feb 2, 2021): Same here. No sirens went off. Looking good.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/tacticalrmm#170
No description provided.