I already opened up a case with support on this issue and the provided me with the steps Nate posted above to create the dump file.

 So far I didn't have a freeze, but maybe you could supply your own dump file to support at StorageCraft so they can evaluate it.  I don't know if that's doable, Nate maybe can help with this.
 

just filled out the form....you guys are fast!

Thanks Mike.  I'll make sure that you receive FTP credentials ASAP.  By the way, what is the size of your dump file?  I'm asking because I'll guess, based on the size, if it's a full, kernel, or small memory dump.  The larger the better (more information to analyze the issue).  Make sure you zip it up before you upload it.  You should receive an email within an hour or so with upload instructions.  Thanks! 

Mike, how did you generate that crash dump file?  As I understand it, you experienced a hang.  Somehow you generated the crash dump in order to diagnose the hang.  Did you use the registry setting to enable the right-control-scrolllock-scrollock on-demand BSOD technique?

I just want to make sure that the dump file is a system dump file that occurred due to a BSOD which you provoked when your system was in a hung state.  Is this correct? 

 yes, donwloaded fix from microsoft and setup key.  it hung, then was dumped via ctrl scroll scroll.

the dump is ~350m compressed, full memory dump. 

Wow!  That's an optimal dump then.  Fantastic! 

The server froze again and I forgot to press the keys to create the dump file :(

It took exactly one month since the last freeze. Hopefully next time I'll remind myself to create the dump.  Any progress with the the dump file Mike sent?

No, it (the dump file Mike provided) was for the old driver, for which we've already implemented a workaround.  We really need someone to repro this with the new stcvsm.sys. 

To clarify:

1. Does the issue appear only for incrementals or even for full backups ?

2. Does it occur only on SBS ?(see how to correct the high memory allocation in SQL for reporting instance of MSDE)

3. Considering on average you restart the server once a month (for patching) did you try to release memory ,stopping and restarting service before backups after the period of 15 days ?

 I think it's only for incremental backups.  I have the software installed on SBS and I don't have memory allocation issues with the SQL instance.  Everything is tuned.

 I didn't try to release the memory every 15 days because that implies the software is leaking memory and I would prefer if it is fixed instead of me having to do workaround.  I'll try to create the dump file next time it freezes.  Hopefully it won't happen again.
 

I have see this on three different servers (two of which are running Trend Micro)

Once Trend Micro was patched to remove the memory leak all three servers had the same issue which over 14 to 15 days would cause a server freeze and the only recourse was to reboot.

The issue came down to Disk I/O being to high. In two  cases it came down to a full AV scan being done at the same time as an incremental image creation with the other being a weekly base being run at the same time as a normal incremental.

If in the System and or the Application log you see a VSS error that references a "object call could not be made " this is because the Disk I/O is too high.

Finally run a performance log on Disk Queue Length over a week period and see if the the average is over two (take the number subtract the number of spindles and the remainder should be less that two) this is a good indication that Disk I/O is an issue. 

I believe you might be up to something here.  I used to have I/O issues with the server that freezes.  I saw high I/O reads from the software I was using for the UPS.  I deleted the process and waited to see if it will appear again. 

 The server froze again. This time I managed to capture the log and I emailed it to support.  Hopefully something will come out of it

 Since I haven't heard anything yet on this issue, and today one of my client's machine froze, I have to see which direction to look for a solution.  The high I/O issue that was mentioned here previously could be a reason for the freeze.

 With that in mind, what exactly the 'Performance Throttling' does?  If I slide the gadget in the middle of the range, does that mean it will  use just 50% what?  Available throughput?  Overall throughput?  If I/O is the problem I have, will that solve the problem?

 Thanks for any input

 

The performance throttling slows down the read operations. ShadowProtect can stress your system in a few ways. 

First of all, we are reading from the I/O channel in a sustained, sequential and non-cached way.  This means that the disk is probably serving up more I/O than it usually does in a sustained fashion.  The disk will heat up more than normal so if cooling is not working well, the disk may freeze.

Compression is very CPU intensive; again, since it is a sustained process, the CPU will heat up.  If cooling is an issue you may see a freeze.

The same goes with the destination, but since we are writing about half of what we read due to compression, then this is not normally the issue, but if it is a weak spot, then you may have issue.

Throttling will slow down the entire process so that the stress levels on different components are reduced.  You can also try different backup options such as No compression and no encryption to see if the backup will finish.

You may want to throttle it all the way down to 10%-25% if you do not see a big difference in backup speed with a 50% throttle.  Since we throttle the read speed, sometimes 50% doesn't do much because the bottleneck in the whole process is the writes.

The only current software interop issue we have seen is Trend Micro.   Doesn’t mean there couldn’t be a new one so in addition to your logs, please send your “System Information Log.”  You can get to this by using the Help->About option in the ShadowProtect UI.

STC