[GH-ISSUE #95 ] Memory leak in Exchange 2013 #75

Author

Owner

@Pro commented on GitHub (Aug 6, 2015):

Please answer the following questions:

Windows Server Version:
Exchange Server Version:
DKIM Singer version is installed:
estimated number of mails sent per day:
number of domains handled by the DKIM Signer:
Are you sure that the DKIM signer is responsible for the memory usage? What happens if you disable it?

@Pro commented on GitHub (Aug 6, 2015): Please answer the following questions: - Windows Server Version: - Exchange Server Version: - DKIM Singer version is installed: - estimated number of mails sent per day: - number of domains handled by the DKIM Signer: - Are you sure that the DKIM signer is responsible for the memory usage? What happens if you disable it?

kerem commented

Author

Owner

@rbfajardo commented on GitHub (Aug 6, 2015):

Sam, what version of Exchange? CU8 or 9? If you don't already have .Net framework 4.5.2 installed, I recommend upgrading to this version. There is not enough information in here to say this is DKIM-Exchange.

@rbfajardo commented on GitHub (Aug 6, 2015): Sam, what version of Exchange? CU8 or 9? If you don't already have .Net framework 4.5.2 installed, I recommend upgrading to this version. There is not enough information in here to say this is DKIM-Exchange.

kerem commented

Author

Owner

@samk1 commented on GitHub (Aug 6, 2015):

It is not Exchange 2013 actually.

Windows Server Version: 2008 R2 Enterprise
Exchange Server Version: 2010 14.3.123.4
DKIM Signer Version: 2.1.4
Avg No of emails per day: 2000 out 5000 in
No of domains: 5

We have tried disabling the DKIM signer, and we do not have the issue in
that case.

.NET 4.5.2 is installed.

@samk1 commented on GitHub (Aug 6, 2015): It is not Exchange 2013 actually. Windows Server Version: 2008 R2 Enterprise Exchange Server Version: 2010 14.3.123.4 DKIM Signer Version: 2.1.4 Avg No of emails per day: 2000 out 5000 in No of domains: 5 We have tried disabling the DKIM signer, and we do not have the issue in that case. .NET 4.5.2 is installed.

kerem commented

Author

Owner

@gogglespisano commented on GitHub (Aug 6, 2015):

I have the same OS/Exchange combination, but 8 domains with about 2000 emails a day.

Check this folder and see if you have a large number of key files:
C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Crypto\RSA\S-1-5-20\ There was a bug that would have created a files for every domain for every email sent or received, so about 35k a day in your case. The computer could be struggling with that folder if you've been running DKIM for a long time.

I haven't seen EdgeTransport.exe spike, but I will watch it's memory use.

The graph looks like there is a sudden increase in CPU and something gets stuck, not allowing garbage collection to take place enough.

@gogglespisano commented on GitHub (Aug 6, 2015): I have the same OS/Exchange combination, but 8 domains with about 2000 emails a day. Check this folder and see if you have a large number of key files: C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Crypto\RSA\S-1-5-20\ There was a bug that would have created a files for every domain for every email sent or received, so about 35k a day in your case. The computer could be struggling with that folder if you've been running DKIM for a long time. I haven't seen EdgeTransport.exe spike, but I will watch it's memory use. The graph looks like there is a sudden increase in CPU and something gets stuck, not allowing garbage collection to take place enough.

kerem commented

Author

Owner

@samk1 commented on GitHub (Aug 9, 2015):

Hello

If I try to open that folder, it hangs my explorer.exe. So there is
definitely a problem there as you say. How do we resolve this?

Regards, Sam

On Thu, Aug 6, 2015 at 9:44 PM, Stuart Wyatt notifications@github.com
wrote:

I have the same OS/Exchange combination, but 8 domains with about 2000
emails a day.

Check this folder and see if you have a large number of key files:
C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Crypto\RSA\S-1-5-20
There was a bug that would have created a files for every domain for every
email sent or received, so about 35k a day in your case. The computer could
be struggling with that folder if you've been running DKIM for a long time.

I haven't seen EdgeTransport.exe spike, but I will watch it's memory use.

The graph looks like there is a sudden increase in CPU and something gets
stuck, not allowing garbage collection to take place enough.

—
Reply to this email directly or view it on GitHub
https://github.com/Pro/dkim-exchange/issues/95#issuecomment-128338247.

@samk1 commented on GitHub (Aug 9, 2015): Hello If I try to open that folder, it hangs my explorer.exe. So there is definitely a problem there as you say. How do we resolve this? Regards, Sam On Thu, Aug 6, 2015 at 9:44 PM, Stuart Wyatt notifications@github.com wrote: > I have the same OS/Exchange combination, but 8 domains with about 2000 > emails a day. > > Check this folder and see if you have a large number of key files: > C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Crypto\RSA\S-1-5-20\ > There was a bug that would have created a files for every domain for every > email sent or received, so about 35k a day in your case. The computer could > be struggling with that folder if you've been running DKIM for a long time. > > I haven't seen EdgeTransport.exe spike, but I will watch it's memory use. > > The graph looks like there is a sudden increase in CPU and something gets > stuck, not allowing garbage collection to take place enough. > > — > Reply to this email directly or view it on GitHub > https://github.com/Pro/dkim-exchange/issues/95#issuecomment-128338247.

kerem commented

Author

Owner

@gogglespisano commented on GitHub (Aug 9, 2015):

explorer may show up eventually, but it will be slow at first. i selected and deleted a few thousand at a time.

try using a command window and cd to that directory. the dir and del commands won't need to read a cache everything like explorer does. in my case, all the files in the folder were from dkim, so it was safe to delete them all.

@gogglespisano commented on GitHub (Aug 9, 2015): explorer may show up eventually, but it will be slow at first. i selected and deleted a few thousand at a time. try using a command window and cd to that directory. the dir and del commands won't need to read a cache everything like explorer does. in my case, all the files in the folder were from dkim, so it was safe to delete them all.

kerem commented

Author

Owner

@samk1 commented on GitHub (Aug 10, 2015):

Deleting the files is not a problem. If they are superfluous, how do we
stop them from being created in the first place?

On Mon, Aug 10, 2015 at 9:07 AM, Stuart Wyatt notifications@github.com
wrote:

explorer may show up eventually, but it will be slow at first. i selected
and deleted a few thousand at a time.

try using a command window and cd to that directory. the dir and del
commands won't need to read a cache everything like explorer does. in my
case, all the files in the folder were from dkim, so it was safe to delete
them all.

—
Reply to this email directly or view it on GitHub
https://github.com/Pro/dkim-exchange/issues/95#issuecomment-129259395.

@samk1 commented on GitHub (Aug 10, 2015): Deleting the files is not a problem. If they are superfluous, how do we stop them from being created in the first place? On Mon, Aug 10, 2015 at 9:07 AM, Stuart Wyatt notifications@github.com wrote: > explorer may show up eventually, but it will be slow at first. i selected > and deleted a few thousand at a time. > > try using a command window and cd to that directory. the dir and del > commands won't need to read a cache everything like explorer does. in my > case, all the files in the folder were from dkim, so it was safe to delete > them all. > > — > Reply to this email directly or view it on GitHub > https://github.com/Pro/dkim-exchange/issues/95#issuecomment-129259395.

kerem commented

Author

Owner

@gogglespisano commented on GitHub (Aug 10, 2015):

There's a fix that stops the excess files from being created, but I don't know if the current prerelease build includes it. Someone else needs to answer whether there's a build available.

Does deleting the files stop the memory problem you have?

@gogglespisano commented on GitHub (Aug 10, 2015): There's a fix that stops the excess files from being created, but I don't know if the current prerelease build includes it. Someone else needs to answer whether there's a build available. Does deleting the files stop the memory problem you have?

kerem commented

Author

Owner

@samk1 commented on GitHub (Aug 10, 2015):

I just deleted everything in that folder. Will have to wait a few hours to
see if this stops the memory from continuously increasing.

On Mon, Aug 10, 2015 at 1:56 PM, Stuart Wyatt notifications@github.com
wrote:

There's a fix that stops the excess files from being created, but I don't
know if the current prerelease build includes it. Someone else needs to
answer whether there's a build available.

Does deleting the files stop the memory problem you have?

—
Reply to this email directly or view it on GitHub
https://github.com/Pro/dkim-exchange/issues/95#issuecomment-129291122.

@samk1 commented on GitHub (Aug 10, 2015): I just deleted everything in that folder. Will have to wait a few hours to see if this stops the memory from continuously increasing. On Mon, Aug 10, 2015 at 1:56 PM, Stuart Wyatt notifications@github.com wrote: > There's a fix that stops the excess files from being created, but I don't > know if the current prerelease build includes it. Someone else needs to > answer whether there's a build available. > > Does deleting the files stop the memory problem you have? > > — > Reply to this email directly or view it on GitHub > https://github.com/Pro/dkim-exchange/issues/95#issuecomment-129291122.

kerem commented

Author

Owner

@samk1 commented on GitHub (Aug 10, 2015):

Still seem to be having the same issue unfortunately. I cleared out the files yesterday using del * /a /q. Running dir /a again this morning I can see that the directory has 46 000 files in it.

We have another server running the DKIM signer that is not having this issue. I think the difference is the server having the issue is the one that is actually sending outgoing e-mails.

The memory usage only seems to increase during business hours when people are sending e-mails.

In the image below the service was restarted at about 5PM to make a configuration change.

@samk1 commented on GitHub (Aug 10, 2015): Still seem to be having the same issue unfortunately. I cleared out the files yesterday using del \* /a /q. Running dir /a again this morning I can see that the directory has 46 000 files in it. We have another server running the DKIM signer that is not having this issue. I think the difference is the server having the issue is the one that is actually sending outgoing e-mails. The memory usage only seems to increase during business hours when people are sending e-mails. In the image below the service was restarted at about 5PM to make a configuration change. ![image](https://cloud.githubusercontent.com/assets/8982168/9186274/8a68f06c-4006-11e5-9ae2-32d7f7d76d73.png)

kerem commented

Author

Owner

@gogglespisano commented on GitHub (Aug 11, 2015):

If you have 46k files in one day, you're doing far more traffic than I am, so although I see a little memory growth in EdgeTransport,exe, it's hard to know if it's proportional to the amount you're seeing.

I don't have time to do it until next week, but I'll look at the main code path to see if I can spot a leak. I'd also like to look at when the keys are loaded or reloaded since the keys for all domains get loaded on each incoming or outgoing mail which seems inefficient, but I don't know what restrictions the transport service agent structure puts on it.

@gogglespisano commented on GitHub (Aug 11, 2015): If you have 46k files in one day, you're doing far more traffic than I am, so although I see a little memory growth in EdgeTransport,exe, it's hard to know if it's proportional to the amount you're seeing. I don't have time to do it until next week, but I'll look at the main code path to see if I can spot a leak. I'd also like to look at when the keys are loaded or reloaded since the keys for all domains get loaded on each incoming or outgoing mail which seems inefficient, but I don't know what restrictions the transport service agent structure puts on it.

kerem commented

Author

Owner

@samk1 commented on GitHub (Aug 11, 2015):

OK sounds good. If you want us to run an instrumented build to gather more information about this problem that should be OK.

@samk1 commented on GitHub (Aug 11, 2015): OK sounds good. If you want us to run an instrumented build to gather more information about this problem that should be OK.

kerem commented

Author

Owner

@Pro commented on GitHub (Aug 14, 2015):

I'll check the code too, the next days.
In the meantime you can try out the most recent build which already includes the fix for huge number of files created. Just replace the .dll (under C:\Program Files\Exchange DkimSigner) with this one: https://github.com/Pro/dkim-exchange/tree/master/Lib

(You may need to stop the "EdgeTransport" Service to overwrite the file)

@Pro commented on GitHub (Aug 14, 2015): I'll check the code too, the next days. In the meantime you can try out the most recent build which already includes the fix for huge number of files created. Just replace the .dll (under `C:\Program Files\Exchange DkimSigner`) with this one: https://github.com/Pro/dkim-exchange/tree/master/Lib (You may need to stop the "EdgeTransport" Service to overwrite the file)

kerem commented

Author

Owner

@samk1 commented on GitHub (Aug 18, 2015):

Hi Stefan

I cannot find ExchangeDkimSigner.dll in the Exchange 2010 SP3 directory in the link you sent.

@samk1 commented on GitHub (Aug 18, 2015): Hi Stefan I cannot find ExchangeDkimSigner.dll in the Exchange 2010 SP3 directory in the link you sent.

kerem commented

Author

Owner

@Pro commented on GitHub (Aug 19, 2015):

Oh, sorry. Try this one:
https://github.com/Pro/dkim-exchange/tree/master/Src/Exchange.DkimSigner/bin

@Pro commented on GitHub (Aug 19, 2015): Oh, sorry. Try this one: https://github.com/Pro/dkim-exchange/tree/master/Src/Exchange.DkimSigner/bin

kerem commented

Author

Owner

@Pro commented on GitHub (Sep 22, 2015):

@samk1 Do you still have the problem with memory leaks?

@Pro commented on GitHub (Sep 22, 2015): @samk1 Do you still have the problem with memory leaks?

kerem commented

Author

Owner

@samk1 commented on GitHub (Sep 23, 2015):

Hi Stefan

Actually we have set up a scheduled task to restart the service every 24
hours and configured thresholds and server memory so that it can last 24
hours without going into back pressure. Obviously this is not an ideal
solution, but it has meant we've stopped paying so much attention to the
issue.

I have not tried the new build yet. I'll see if we can give this a go and
get back to you.
On 23 Sep 2015 7:12 am, "Stefan Profanter" notifications@github.com wrote:

@samk1 https://github.com/samk1 Do you still have the problem with
memory leaks?

—
Reply to this email directly or view it on GitHub
https://github.com/Pro/dkim-exchange/issues/95#issuecomment-142424300.

@samk1 commented on GitHub (Sep 23, 2015): Hi Stefan Actually we have set up a scheduled task to restart the service every 24 hours and configured thresholds and server memory so that it can last 24 hours without going into back pressure. Obviously this is not an ideal solution, but it has meant we've stopped paying so much attention to the issue. I have not tried the new build yet. I'll see if we can give this a go and get back to you. On 23 Sep 2015 7:12 am, "Stefan Profanter" notifications@github.com wrote: > @samk1 https://github.com/samk1 Do you still have the problem with > memory leaks? > > — > Reply to this email directly or view it on GitHub > https://github.com/Pro/dkim-exchange/issues/95#issuecomment-142424300.

kerem commented

https://github.com/Pro/dkim-exchange/issues/103

Author

Owner

@gogglespisano commented on GitHub (Oct 30, 2015):

Adding a link in this issue to another issue/fix that may help with this problem.

@gogglespisano commented on GitHub (Oct 30, 2015): Adding a link in this issue to another issue/fix that may help with this problem. https://github.com/Pro/dkim-exchange/issues/103

kerem commented

Author

Owner

@gogglespisano commented on GitHub (Oct 30, 2015):

I think we should wait for @samk1 to confirm if the memory leak is fixed before closing this issue. I never really saw the leak, but I have far less email traffic. I think this change will dramtically reduce the code executed per email, but it still might not fix the leak.

@gogglespisano commented on GitHub (Oct 30, 2015): I think we should wait for @samk1 to confirm if the memory leak is fixed before closing this issue. I never really saw the leak, but I have far less email traffic. I think this change will dramtically reduce the code executed per email, but it still might not fix the leak.

kerem commented

Author

Owner

@Pro commented on GitHub (Oct 30, 2015):

@samk1 can you test version 2.1.8
It is currently released as a prerelease and includes a fix which may have caused the memory leaks.

@Pro commented on GitHub (Oct 30, 2015): @samk1 can you test version 2.1.8 It is currently released as a prerelease and includes a fix which may have caused the memory leaks.

kerem commented

Author

Owner

@samk1 commented on GitHub (Nov 1, 2015):

Hello

Sorry for not getting back to you sooner on this.

We noticed our backups of one of our exchange servers were starting to take longer and longer. It turns out this was caused by the creation of 1000s of files by DKIM signer (Since it's in the windows directory, it causes the VSS system writer to slow down), We fixed this by installing the dev version linked above.

I have not checked to see if the memory leak issue has been fixed by that release. I have started up the performance monitor of edgetransport.exe (It takes a few hours to see if their is an increase in memory usage). After confirming if the problem still exists or not I will install 2.1.8.

Regards, Sam

@samk1 commented on GitHub (Nov 1, 2015): Hello Sorry for not getting back to you sooner on this. We noticed our backups of one of our exchange servers were starting to take longer and longer. It turns out this was caused by the creation of 1000s of files by DKIM signer (Since it's in the windows directory, it causes the VSS system writer to slow down), We fixed this by installing the dev version linked above. I have not checked to see if the memory leak issue has been fixed by that release. I have started up the performance monitor of edgetransport.exe (It takes a few hours to see if their is an increase in memory usage). After confirming if the problem still exists or not I will install 2.1.8. Regards, Sam

kerem commented

Author

Owner

@gogglespisano commented on GitHub (Nov 4, 2015):

I'm running 2.1.8. EdgeTransport.exe climbed to 351MB, but hasn't changed more than 1 MB in 2 days.

@gogglespisano commented on GitHub (Nov 4, 2015): I'm running 2.1.8. EdgeTransport.exe climbed to 351MB, but hasn't changed more than 1 MB in 2 days.

kerem commented