So its been a month or so since I purchased Shadow Protect and I think I'm getting pretty familiar with the functionality of it, but I still have a few questions.

 First, how does SP decide when to do a "Full" backup versus an incremental in a Monthly backup situation.  I have a schedule set for Full on the last day of the month and incrementals every day of the week.  I open Shadow Protect to initiate an incremental, which I've done many times before, but instead it creates a differential backup.

 My other question is regarding the proper way to discard old differential, and its associated incremental sets.  Is it okay just to delete it off the drive or is there a better way to do this?

 

In this response I will assume that your job schedule type, as specified in the Schedule page of the New/Edit Job wizard, is "Monthly."

For monthly backup jobs, full backups will be taken on the "Days of the month" that you add to the Included list in the Full Backups section of the schedule page.  Incremental backups are taken on the days which are checked in the "Incremental Backups" section at the time you specify.

When ShadowProtect goes to create an incremental, it checks to see if it can find the image files upon which the new incremental depends and if it can't find them it will create a new base image file.  If the previous image files are found, then ShadowProtect will attempt to create an incremental backup.  It can create an incremental using one of two different technique.  In either case the incremental will contain the same data, however, one technique (VDIFF or FAST) is much faster than the other (DIFF or DIFFGEN).  VDIFF/FAST incrementals are only possible if your system has not crashed since the last backup image.  Oh, by the way, I'd always recommend you turn on the crash proof incremental "Enable self-healing incremental healing" advanced option feature, otherwise the backup job will create, at the next scheduled time, a new base image file instead of a DIFF/DIFFGEN incremental if your system is powered off ungracefully or crashed.

Crash proof (DIFF/DIFFGEN) incrementals, when enabled, ensures that the backup job will generate another incremental when it does a backup after a system crash or hard power-off.  Normally, when an incremental is created, it will be created quickly utilizing the in-memory incremental tracking capability of stcvsm.sys.  However, if the system crashes, then the in-memory data structures which are necessary to support fast incrementals are lost, and so the next backup must be either a full backup, or an incremental which is created using a diff technique.  This is the purpose of Crash-Proof-Incrementals (which I believe .  If the system crashes, then the in-memory data strucutres for the next incremental (as a fast incremental) are lost, however, the when the system comes back up the job can still generate another incremental as long as it uses the diff technique, and at this time incremental tracking is enabled again such that only the single incremental generated after the crash in created with the diff technique and subsequent incrementals are fast again.

What is the diff technique?  Simply to compare the state of the current snapshot against the data in the point-in-time (which is represented by the combined contents of the previous backup image files).  Interestingly, by performing this diff operation, the backup job is necessarily reading the data in the previous backup image files in order to compare them with the current snapshot state, and this reading of the data in the previous image files includes the exact same verification tests as a verify operation, so when a diff-technique-generated incremental is created successfully, you can have the warm fuzzy feeling that the system has also, in the process, successfully verified the previous image files in the chain leading up to to the latest diff-generated incremental.

As far as deleting files go, you must obviously be very careful about this.  It's easy to accidentally delete image files upon which others are dependent and thereby orphan some of your backups.  I suggest instead that you use the built-in Retention Policy capability of the Monthly backup job type, which you can find on the Advanced options dialog's Retention tab.
 

 Nate, thank you very much for the detailed response.  One more thing to make sure is I have it also set to only create differentials on the next back-up image.

Couple of follow-up questions.

First, just to make sure I'm reading it correctly, any system or hard power off at any time, not just when the schedule is running, can cause a Differential to be created instead of an Incremental even with the self-healing enabled (which I do have on for that schedule)?

If you could help me understand the Retention Policy better, I'd appreciate it.  Currently I have it sent for the past 10 and not delete the Full/differentials.  If I let it delete the Full/differentials will it allow 10 differentials and 10 incrementals or 10 total combined differentials/incrementals?  If it's the latter how does it decide what to delete, for example with my schedule policy, I do not allow missed tasks to execute, so in theory it should delete the past 10 (or 9?) incremental backups, keeping the last differential set (and the original full) associated with the incrementals and then delete the rest?

One more question, if I change Retention policy it will take affect on the next backup and not from that point on correct?
 

If you hard power-off your machine, or if it crashes, then the in-memory data structures needed for fast incrementals are lost, and so the next time an incremental is created, that new incremental will be created using the slower diff/comparison technique rather than the fast in-memory-change-tracking technique.  The incremental that follows it will be back to fast.  If you gracefully shutdown your machine, then the in-memory-change-tracking structures are saved to disk (in the .IDX file in the volume root) and are re-loaded on the next boot so that fast incrementals continue uninterrupted.

Note that in ShadowProtect, a differential is simply an incremental which is created using the diff technique.  A differential is not necessarily dependent upon the base image file.  In fact, in the case of the "self healing incremental" functionality, the diff-generated incremental will actually be dependent upon the previous (latest) incremental image file.

You'll also find that, if you do a manual differential backup job, you can specify *any* previous image (not just a base) to be the point-in-time upon which the new diff-generated incremental will be depdendent.  This is different (more flexible) than some other products, where a "differential" is always strictly dependent upon a base/full image.  In this discussion, let's call an image file which is immediately dependent upon the base, which was created using the diff technique, a "true differential."  I believe our documentation also calls "true differentials" "optimized bases."

In the Retention tab on the job's Advanced Options dialog, the "backup image set" that it refers to is a base with all of the incremetnals (fast and diff incrementals) which are dependent upon that base.  You can tell it how many sets you want it to keep around, and it will delete any older sets beyond the number of sets that you specify.  The caveat is that when it deletes older sets, you can tell it to either delete the entire older set, or to delete ALL files in the set other than the base.  The differentials are not immediately dependent upon the base - they are just incrementals which are made using the diff technique, and are dependent upon the previous incremental.  Therefore, the retention policy doesn't trim out individual incremental image file while keeping the latest incrementals.  If incrementals are deleted in the middle of a backup set then this will orphan any incrementals which were dependent upon them.  Therefore the retention policy works with an entire set (base image with all incrementals that are dependent upon it) at a time.  I think this is the point that was confusing you.  Am I right, or are you still confused?

The retention policy is always enforced at the end of each full (or true differential) backup, unless you check the box to specify that you want the policy to be enforced before a full (or true differential) backup.

Now, if you are using the advanced option "Second and subsequent full backups are differentials" then this complicates the above explanation because there will only be one true base image file created by the schedule - the other bases created by the schedule will actually be true differentials.

 

LoL, man a lot of information to take in.  Thank you, I think I'm truly getting it.

 So using the "Second and subsequent full backups are differentials" with the Retention Policy set to only delete incrementals would be the best method right?  Because it should delete both the Differential-Incrementals and the Incremental-Incrementals correct?  The Full will remain but I can consolidate that using the Backup image tool correct?
 

 Hmm went to do an incremental backup of my monthly job and it attempted to do a Differential-Incremental (d00x set)... not sure why, system hasn't crashed since the last incremental-incremental (i00x set),