Obsolete objects - Reuse your ”Old” SCCM Client Objects in Mixed Mode

When ConfigMgr is configured in Mixed mode and client computers are re-imaged/re-installed; by default a new ConfigMgr computer object is created and the “old” computer object will become “Obsolete”. Well, this is by design, but it might become a problem. If the client has many direct collection memberships, these direct memberships will not be preserved when a new computer object is created. Therefore, these have to be re-created.
When ConfigMgr is configured in “Native Mode”, the client identity is automatically migrated since we are using certificates.


Well, with ConfigMgr, this problem can be handled. However, it is a manually process. In the “Site properties” dialog, on the “Advanced” tab, there is a section called “Conflicting records”. From here it can be configured how ConfigMgr should handle conflicting records. That is, clients with duplicate hardware Id’s. By default, it is configured to “Automatically create new client records for duplicate hardware IDs” as shown in the picture below.


And that is exactly what happens when a client is re-imaged; A new record is created and the “old” record is marked as “Obsolete”.
If “conflicting records” is configured to “Manually resolve conflicting records”, the ConfigMgr Admin must decide what to do with conflicting records. This is done through “Conflicting records” node in the ConfigMgr Admin Console (See picture below)

In the “Conflicting records” node, all conflicting records are listed. By right-clicking a record, 3 actions are exposed to the administrator: Merge, New, and Block.
Merge
This method combines the new detected record with the existing client record. This is equal to reusing the ConfigMgr “old” computer object. This is the method to use if we want to reuse the “old” object.

BlockThis method creates a new computer object, but it will be marked as blocked, and will remain unmanaged.

NewThis method creates a new computer object, and the “old” one will be marked as “obsolete”. This method is the default one, when “conflicting records” is configured to “Automatically create new client records for duplicate hardware IDs”.
It is important to note, that client computers having a conflicting record, will be unmanaged until the administrator actively decides what to do with the conflicting record (merge, block, create new).
So, by using the “Merge” method, we can reuse the “old” computer object. So that is very nice, but it is a manually process, and it might not be feasible to the administrator, that every time a client computer gets reinstalled, he or she have to go to the “conflicting records” node in the Admin Console and decide what to do with the computer object. So now the question is: “Can this be done automatically ?”….. Yes it can! However, the method is not documented by Microsoft, and neither can it be found it the ConfigMgr SDK, and there is a good reason: Security !!! I will get back to that later on.
However, I did a little research on the ConfigMgr server WMI-namespace, and I found an interesting class; the SMS_PendingRegistrationRecord class. I found that every time a “conflicting record” occurs, an instance of the “SMS_PendingRegistrationRecord” is created. Further, the SMS_PendingRegistrationRecord” has a method called “ResolvePendingRegistrationRecord” which takes 2 input parameters: ”SMSID” and ”action”. The “SMSID” parameter specifies the SMS Client GUID of the conflicting record, and the “action” parameter specifies what to do with the record: Merge, Block, or create new.
Input values for the “action” parameter
1 = Merge
2 = New
3 = Block
I have made a small sample script for resolving conflicting records. The script loops through all conflicting records and resolves them by using the “Merge” method. The script is just a small sample script, and provides no error handling or logging
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Dim swbemLocator
Dim swbemServices
Main()
Sub Main()
Dim oProviderLocation
Dim oLocation
Dim oReg
Dim oPendingRegs
    Set swbemLocator = CreateObject("WbemScripting.SWbemLocator")
    swbemLocator.Security_.AuthenticationLevel = 6 'Packet Privacy.
    Set swbemServices = swbemLocator.ConnectServer(".", "root\SMS")
    Set oProviderLocation = swbemServices.InstancesOf("SMS_ProviderLocation")
    For Each oLocation In oProviderLocation
        If oLocation.ProviderForLocalSite = True Then
            Set swbemServices = swbemLocator.ConnectServer(oLocation.Machine, "root\sms\site_" + oLocation.SiteCode)
        End If       
    Next
    Set oPendingRegs = swbemServices.ExecQuery("SELECT * FROM SMS_PendingRegistrationRecord")
    For Each oReg In oPendingRegs
        Resolve 1, oReg.SMSID
    Next
End Sub
Sub Resolve(action, SMSID)
Dim InParams
    Set InParams = swbemServices.Get("SMS_PendingRegistrationRecord").Methods_("ResolvePendingRegistrationRecord").InParameters.SpawnInstance_
    InParams.Action = action
    InParams.SMSID = SMSID
    swbemServices.ExecMethod "SMS_PendingRegistrationRecord","ResolvePendingRegistrationRecord", InParams
End Sub
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
I have tested this script and implemeted it at a couple of customers, and found no problems with it. The script is executed on the site server on regular basis (every 10 minutes or so) by using a “Scheduled Task”.
The script can not be run as a part of a Task Sequence, since there will be a delay from when the SMS Agent Host is installed on the client computer, until til it appears in the "Conflicting records" node.
As mentioned earlier; this method is not documented or supported by Microsoft and for a good reason: Automatically merging records introduces a security risk: If a rogue user that knows or guesses the hardware ID of a previously assigned client computer, this person could take over the identify of that client and gain access to the SCCM site. It would be very difficult to detect this if records are automatically merged. 
So if you want to reuse your “old” ConfigMgr computer objects, you might consider using this script. Just keep the security risk in mind.

Introducing SCCM 2007 R3

SCCM 2007 R3 (R3) has released!  R3 is not a service pack – it is an addon that brings additional functionality.  R3 includes all of the features of R2 and requires that SCCM 2007 SP2 be installed because the SP2 version of the SCCM client has the needed functionality for R3.  In addition, there is a requirement to deploy a client side hotfix for R3 – this is described in KB977384.  R3 won’t allow the install to proceed until this hotfix has been applied.  There tends to be a decent amount of confusion about an Rx release compared to a service pack so hopefully this helps clear things up.
When deploying KB977384 you have a choice to allow the install to create a package for you or to create the package manually.  If you choose to have the package created automatically it will be similar to what I have below.

So what exactly is in this R3 addition anyway?  R3 is mostly about power management.  There are some other interesting additions as well.  We will walk through each.


Power Management The power management feature of R3 makes use of SCCM hardware inventory – so make sure hardware inventory is enabled when R3 is installed.  During installation R3 will add some inventory classes to the SMS_DEF.MOF.  The R3 inventory changes are an addition to the MOF files rather than an overwrite of the MOF files as we have seen with service packs so there really shouldn’t be a need to backup your MOF’s first – but it’s always a good idea just to be safe.
If you take a look in the SMS_DEF.MOF after R3 is installed you will see the specific sections that are added – just search for PWR.

So what are the components of power management?  There are two – the power management client agent and the collection specific power settings.  The power management client agent is shown below – the only setting here is either enabled or disabled.

The per collection power management settings is where the configuration is done.  Notice that in my lab I’ve built two collections – one for systems where I specifically do not want a power policy applied and one where I will define my power policy.


If we right-click on any collection we can set power policy for the collection.  Let’s start with my no power policy collection.

Notice the power management setting – in my no power policy collection I first enable power management settings and explicitly configure to never apply power management to systems in the collection.  This will ensure that these systems never have power policy applied.  So what happens if a system is in this no policy collection and another collection with power policy?  First, that shouldn’t happen if you configure your collections properly but if it does the ‘never’ policy will supersede any configured power policy so the system won’t have any power management applied to it.
OK, so lets look at the settings for power management where we do have configured power policy.

Here we have configured power settings – the peak plan will be high performance and the non-peak plan will be power saver.  Notice that you have full control of the hours where peak policy applies.  Note also that there is the ability to configure a wakeup time for systems in this collection.  The wake up time will power up systems to allow them to do any pending work assigned through SCCM. 
A common question here is what mechanism is used to wake the systems – is it wake on LAN, is it AMT based?  No to both.  Windows has internal powerup timers that will function as long as there is power available to the system.  When the preconfigured time arrives the system will know to wake up.
Note:  Wakeup is not supported for laptop systems to prevent a scenario where the system will wake up on battery and exhaust power before shutting back down.
Another common question – suppose a system wakes up but power policy is configured to have it sleep again in 15 minutes of inactivity.  Suppose also that SCCM has work queued up that will take longer than 15 minutes – will the system sleep in the middle of SCCM jobs?  No.  While SCCM has work to do we keep the system in an active state so that it will sleep only when all scheduled work is done.
So what power policies can we configure here?  If we select view we see the list and the default/configurable values.

We can change whatever we like here and the modifications will be saved against the current collection – other collections will be unaffected.
OK – we have the power policy configured – what does it look like on the client?  We have both a Windows 7 and Windows XP test client – the settings apply to both.  If we look at the Windows 7 client we see that the power policy that is in place indeed is coming from SCCM – and the user has the ability to change the plan settings.  It’s a futile effort though as with every 24 hours (not every policy cycle) we will just set power settings back to what we have configured for the collection.

Our XP test system also has applied the SCCM power settings but notice that with XP the user has no ability to change the settings.  Also notice that in XP the SCCM options show up as customized.

So that’s about it for setting up power management.  Now that we know how to configure things is there any general strategy for rolling out power management?  Yes.
In most cases organizations are interested to know what the impact of power management has been �� how much money has been saved?  Getting a good handle on these questions is best addressed by rolling out power management in phases.  In general terms there are four phases
1.  Monitor the current state and power consumption
2.  Plan and create the power management policy
3.  Calculate savings
4.  Check compliance and report savings
R3 power management includes a good number of reports that help track through these phases.  At the most basic level we can start with our hardware inventory information.  Looking at resource explorer for my Windows 7 system we see the information returned.  Note that my test system is a VM so some of the numbers may look odd.




Looking machine by machine is interesting but the reports that come along with R3 provide the ability to get a more holistic view of the environment so you can track what your power usage is currently, how it has changed since applying power policy, savings associated with the change etc.
The reports that come along with R3 will not work with classic reporting.  Why?  Classic reporting is at the end of life – all future reporting work is focused on SQL Reporting Services so the R3 reports will only be usable if SCCM SQL Server Reporting Services integration is enabled.  If you haven’t started working toward SQL Server Reporting Services integration, take a look at my technet article that discusses how to do this and some of the advantages.
Once you have SQL Server Reporting Services integration enabled all you have to do is import the R3 reports.
Select to copy reports to Reporting Services from the right-click menu

Walk through the wizard to the ‘select reports’ screen and you have two choices – either select classic reports to migrate to SSRS or choose to import reports to SSRS (an option that was added as part of the R3 installation).  In our case with R3 we want to import reports to SSRS.  The cab file we want is in reports > power management folder of our R3 installation. 

Once imported you will see the new power management reports under ‘all reports’

Running a sample report – such as the Power Settings reports will return data about the various settings configure on our test systems as follows:

The report shows all of the settings available, how many computers are configured with each and is a drill down report – so if the value in the computers column is selected we further drill into the report to show which computers have the listed setting.

A final question – does power management support both workstation and server OSes?  No – the biggest benefit you will see for power management is on the workstation side and workstations OSes Windows XP and forward are supported for power management.  Power management of server OSes is not supported at this time.
And that’s it - fairly straight forward for power management.  There are a couple of other changes in R3 as well.
Delta AD Discovery Delta AD discovery looks for changes in AD membership – additions, changes, deletions – and reports just those at the end of the discovery cycle.  The advantage here is that AD discovery can be run much more frequently without overloading the site.  There is still a requirement to have a full AD discovery on a schedule but with the delta feature the full discovery can be run far less frequently.

Dynamic Collection Updating Dynamic Collection Updating – also known as fast collection updating – allows for collection additions to be picked up quickly.  One of the key benefits of adding systems to collections more frequently is in terms of software distribution.  Before dynamic collection updating a system may not receive new advertisements for up to 24 hours (assuming a collection update frequency is once a day – which is the default).  With the dynamic scenario and if the system is capable of being found by this method (more on that shortly) the time for software to reach the system is dramatically reduced.

Dynamic collection updates do not apply in all scenarios – only four specific types of configurations can be picked up by dynamic collection updates.
1.  Systems being discovered for the first time
2.  Systems provisioned by OSD
3.  Systems scanned for initial hardware inventory
4.  SCCM clients that have been upgraded to a higher client version.

Collection updating happens normally for any system besides those in one of the 4 listed conditions.
While dynamic collection updating is available on all collections with R3 we only recommend it’s use where needed. 
PreStaged Media
PreStaged Media allows for the offloading of some imaging responsibility to the OEM.  This is a feature that has been in MDT for a while but is now native to SCCM with R3 installed.  The idea is that the OEM is provided an image and as part of building PCs they load the provided image.  Then, when the system arrives at your company it is plugged into the network and imaging continues from where it was paused.  For more information on PreStaged media take a look at this blog entry.
Scalability
R3 increases the scalability of SCCM in that now we support 300,000 clients per hierarchy.  Other support numbers stay the same – we still only support up to 100,000 clients on a single primary, etc.
And that’s it for R3 – definitely worth checking out and very easy to use!

Understanding Site to Site Communication in SMS/SCCM

I’ve been intending for a while to write a post on the internals of site to site communication in SMS/SCCM.  The internals really haven’t changed much since SMS 2.0 so no matter what version of the product you are using, the detail here will apply.
For some reason when I think of site to site communication I always think in terms of package distribution.  Certainly site to site communication comes into play with package distribution assuming the package is being staged to distribution points at a child site – but package distribution is only one item that requires solid site to site communication.  Other examples include site control file updates, status messages, state messages, discovery records, inventory, etc. 


Information going site to site does so based on priority.  These priorities help ensure ordering so critical data goes first.  There are three priorities – high, medium and low.  Site control file updates always go high priority, status messages by default go low priority and most everything else goes medium priority by default.  Some priorities are configurable (packages, status messages) but most are not (site control file updates). 
So what if I have a very large package in progress at medium priority and I want a new package to be deployed with high priority?  Will SCCM pause sending the medium priority package in preference to my high priority job?  No.  Once sending is underway it continues until completion and then the highest priority item after that will be transferred next.  The ‘scheduling’ of site to site jobs is handled by a component strangely named the scheduler!  More on that as we go along.
So what needs to be configured for site to site communication?  Two things are needed - an address (to tell us where the other site is) and a sender which is used for transferring the data.  There are several senders to choose from but the two most common are the LAN sender and the courier sender.  I’ve only seen another type of sender in use once years ago. 
The courier sender is a mechanism that allows sending of data between sites using external media – such as a DVD, CD-ROM, flash drive, etc.).  This sender is typically used in scenarios where large amounts of data need to be transferred and the WAN environment is insufficient to handle the load.  Further discussion of the courier sender is beyond the scope of this post.
The LAN sender is the most common sender choice and simply means that data will be transferred between sites using the network. 
The discussion of senders and address opens up a whole different branch of this topic related to strategies for placing addresses.  In a three tier hierarchy, for example, should the central site have direct addresses for all child sites – including those at the third tier?  The answer almost always is no – instead of direct addressing the central site benefits by taking advantage of fan-out distribution (if you aren’t sure what that is, see the ‘using fan-out’ section of this document)
OK, with that introduction let’s get into the topic.  For simplicity we will use a two tiered hierarchy for discussion.  CEN is the central site and PRI is the primary child site.  The concepts discussed also apply to 3 and 4 tier hierarchies and secondaries follow as well.  For purposes of discussion, a two tiered structure keeps things simple.
On CEN we have an address configured for PRI – so we know were it is and can send data to it – and we have a sender configured.

We also have an address configured on PRI back to CEN

The address settings are where all of the configurations are available.  On the general tab we can configure items such as the destination computer name and the account to be used for site to site communications.  If left blank the computer account is used – and that often is preferable so we can avoid maintaining passwords.  If you prefer a user account can be specified here and must be specified if sites are separated by an untrusted security boundary. We also see a choice here for address priority

The schedule and rate limit tabs are used to configure when the particular address will be available and whether or not it can run at 100% or needs to be throttled.  Note that if any throttling is selected the ability to send multiple packages to a site is disabled. 
The sender is separate from the address.  The sender node defines all senders in use at a site.  In our example we just have the standard sender.

On the sender advanced tab we can specify concurrent sending settings between sites but, as mentioned earlier, if address  throttling is enabled the settings are overridden and only one thread is active per site.

OK, so thats all of the configuration to be done in the UI.  Remember that these two items need to be configured on both sites that are communicating.  With this done, lets see how site to site communication works behind the scenes.  For our walk through we will trace a package as it leaves distribution manager en route from CEN to PRI.
The flow of events is as shown. 

Note that to illustrate the movement of data through the system I have artificially paused processing along the way.  If you look at your equivalent folders you likely won’t see data in many of the folders I’m showing unless there is a problem or the site is very busy.
Our test package is created and sent to a distribution point at our child site.  We see in the distribution manager log file that a minijob has been created that will send the content to PRI.
Distmgr.log
--------------
Created minijob to send compressed copy of package CEN0000D to site PRI.  Transfer root = \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK.   
STATMSG: ID=2335 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=STEVERACCEN SITE=CEN PID=2580 TID=5112 GMTDATE=Tue Jul 13 01:58:23.632 2010 ISTR0="CEN0000D" ISTR1="PRI" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="CEN0000D"   
Package CEN0000D is new or has changed, replicating to all applicable sites.   
Sent package changes to site PRI for package CEN0000D   
Again a reminder that we started our site to site communications demo using distribution manager content – but it could just as easily have been collection replication, site control file replication, status message replication, state message replication, etc.
The minijob is placed in replmgr.box which is the component that handles replication.  The contents of replmgr.box is show below:

There are 5 folders here.  

Incoming This is where all data that is incoming from a parent or child site is placed.  Replication manager has the job of figuring out what kind of data it is, which component should handle it and where it should be placed – and then the content is copied there for final processing (more on this once we get over to the server side.  This folder is not part of our example on CEN (but will be on PRI) so we will skip for now.
Outbound
This is the location where all data is placed by various ‘sending’ components that is destined for another site.  Data is categorized as to priority and placed in the appropriate subfolder under Outbound – either low, normal or high priority.  Our minijob – known as a replication job - was created here  in the normal priority folder as shown.


On a side note (rant) – I can’t count the number of administrators that seem to think deleting files in the various inboxes is an OK thing to do.  In many cases you will get away with it – once you enter the site to site mechanism (and we have just entered it) you better know exactly what you are doing before deleting.  Best rule – don’t delete unless you KNOW.  Sometimes it’s required but many times its not.
OK, so what is this RPL file – if we open it in notepad we can see some detail about it.

Note:  It’s OK to look at files in an editor but don’t try to edit or save them.  Many  files contain binary information that will not save correctly and will render the file unusable.
Much of the file contents appears to be gibberish but some content is readable – we see details such as the name of the package – software dist flow – the source directory for the package (and the content that will be transferred) – pkgsrc\rockstar demo CEN, the compressed PCK directory - \STEVERACCEN\SMS_CPSC$\CEN0000D.PCK, the security rights assigned to the package, the destination for the package – the destination site PRI – Primary Child, the sending side CEN and the thread involved – Distribution Manager.  Oh, before you go look for a folder called SMS_CPSC$, it’s a hidden share – it’s actually the SMSPKG folder as shown.

Note:  If we jump ahead there is already a .job file created in the schedule.box folder as shown – this is the minijob that will act on data in replmgr.box\ready – but more on that in a bit.

Process Data moves from outbound to process where everything is checked and made ready for transfer.  Typically you won’t see files in the process folder unless there is a problem.  The processing was so quick I wasn’t able to catch it to show here.
Ready
This folder contains data that is ready to be sent.  Once data arrives here the next steps are to schedule it for sending and to send.  If you look in the ready folder on a reasonably active site you will typically see folders with data inside that are destined for another site in the hierarchy.  The highlighted area below shows that the contents of this folder are destined for site PRI.
If we look inside the folder we see another .RPL file.
  The filename has changed from the one we found in the incoming folder – does that mean the contents of the file have changed?
Nope, same file – except now it has been processed, validated and is ready for sending.

History
The history folder maintains a history of all transactions that have flowed through replication manager – either incoming from other sites or outbound to other sites.  If we look in the history folder we see a .TRS file- known as a transaction file – which records movement through the replication manager inbox.  In some scenarios information will fail to replicate due to invalid values in this file.  In earlier versions – such as SMS 2.0 – this fact wasn’t logged in replmgr.log but is now.

All of the activity just described is recorded in just a few lines from the replmgr.log
Scanning low priority outbound replication directory.    
Did not find any additional replication files.    Minijob created.  Priority 2, transfer root (\\STEVERACCEN\SMS_CEN\inboxes\replmgr.box\ready\2_gg1xd8.PRI).   
As we leave the replmgr.box note that content will remain here until notification is received that sending has completed.  This is important to ensure pointers to content are not removed until sending is finished.
The next step is to schedule the jobs for sending.  If we take a look at the schedule.box we see jobs waiting for processing.  We also see a few folders. 

We will go through the folders in the order of processing but first we need to get processing started!  The log snip below shows the processing loop as schedule handles these jobs.  I’ve added commentary to the log to point out a few key points of interest.
SMS_EXECUTIVE started SMS_SCHEDULER as thread ID 3344 (0xD10).   
========  Starting Processing Cycle ( 07/13/10 18:07:40 Pacific Daylight Time) ========   
Updating in-memory send request list...   
Deleting in-memory send requests that are no longer in inboxes.   
Deleting in-memory slave send requests that are not referenced by a  master request.   
Deleting any completed jobs     
      processing loop>.   
========  Processing Routing Requests  ========   
========  Updating Send Request List  ========   
Updating in-memory send request list...   
Deleting in-memory send requests that are no longer in inboxes.   
Deleting in-memory slave send requests that are not referenced by a  master request.   
========  Building Outbox Send Request Lists  ========   
Examining 0 send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests...    Sorting list of 0 send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests by priority and time...   
Examining 0 send requests for outbox \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN...   
Sorting list of 0 send requests for outbox \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN by priority and time...   
Sorting complete, updating perf counters with number of master send requests...         
     sending in the outboxes\LAN folder – remember we are using
     the LAN sender> ========  Checking Status of All Send Requests  ========   
========  Scheduling All Unscheduled Send Requests  ========   
========  Cleaning Up Routing Packages  ========   
========  Finished Processing Cycle ( 07/13/10 18:07:40 Pacific Daylight Time) ========   
Sleeping for maximum 10 seconds.   
========  Starting Processing Cycle ( 07/13/10 18:07:50 Pacific Daylight Time) ========   
========  Processing Jobs  ========   
Updating the in memory job list...   
0 jobs found in memory, 2 jobs found in job source.     
      anything new>   
[Software Distribution for Software Dist Flow, Package ID = CEN0000D]   
    Destination site:  PRI, Preferred Address: *, Priority: 2   
    Instruction type:  MICROSOFT|SMS|MINIJOBINSTRUCTION|PACKAGE   
    Creating instruction file: \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\0000004F.I16   
    Transfer root:  \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK   
    Begin to calculate signature on \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK   
    Signature on \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK is 3ABF1ADA72390CF25F3499D925D2EA2FB60C3D3B0801FF6B85E7F8135A1B3B8CDB1052AC6DDCC1F6E273BACB9545FBEB16120F19E7A6DA2540482CDEC393AEA225D3A1E1CBE115843B44C4361090507E6AC7D95B4D769DA211A0E1B89EE6902490345E8938DCE5AFD6F359A4A8A9A2C27B532450844356DF433886DEE97AEFAB   
    Hash algorithm ID on \\STEVERACCEN\SMS_CPSC$\CEN0000D.PCK is 0x800C   
    Instruction (and package) file created.  Mark job active.   
       
[Software Distribution for Software Dist Flow, Package ID = CEN0000D]   
    Destination site:  PRI, Preferred Address: *, Priority: 2   
    Created new send request ID:  20027CEN        
     being activated along with identifying info – in this case a
     package – what the destination site is – in this case a direct
     child but it could be that we are sending this to a third tier site
     in which case that site code would be seen.  We also see the
     instruction file being created and the fact it is placed in the
     tosend directory. The location of the compressed content is also
     seen along with the signature on the package.  Finally we see
     the job marked active> 
[Inter Site Replication]   
    Destination site:  PRI, Preferred Address: , Priority: 2   
    Instruction type:  MICROSOFT|SMS|MINIJOBINSTRUCTION|REPORTING   
    Creating instruction file: \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Imh        Transfer root:  \\STEVERACCEN\SMS_CEN\inboxes\replmgr.box\ready\2_gg1xd8.PRI        Creating package file: \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Pj8        Begin to calculate signature on \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Pj8        Signature on \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Pj8 is 4FE19318CFC83252BA271A43260CB0DAEC2D513028E4F6AFEA0E4DEAE56C62A4FC5650CB40AB2EDEB064B618C74242351552DA12B95EEACFE1066C49C55F844C4DE051CC583C7D4498CF741AB3721631E9C8877533959E153C633FACACE41D77EB204A48CBD4E33B9591A452E0138B0BACA7B37DA4B2B7D7E5564FA280E71602        Hash algorithm ID on \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\tosend\00000050.Pj8 is 0x800C   
    Instruction (and package) file created.  Mark job active.   
       
[Inter Site Replication]   
    Destination site:  PRI, Preferred Address: , Priority: 2   
    Created new send request ID:  20028CEN   
Updating the in memory job list...   
2 jobs found in memory, 2 jobs found in job source.   
Job count is 2.       
     some of the differences but still basically the same structure –
     now we have both jobs activated> 
========  Processing Routing Requests  ========   
========  Updating Send Request List  ========   
Updating in-memory send request list...        
      in the tosend folder – the next step begins to evaluate our
      progress sending – in this case the sender is stopped (on
      purpose for this blog entry) so no activity is logged – we will
      start it shortly.>
====  Found 2 send requests in outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests.   
Send Request 20027CEN   JobID: 0000004F   DestSite: PRI   FinalSite:      State:  Pending   Status:            Action:    None   Total size:       80k   Remaining:       80k   Heartbeat: 18:07   Start: 12:00   Finish:   12:00   Retry:         SWD PkgID: CEN0000D   SWD Pkg Version:    1    
    
     one below.  This status is very useful troubleshooting
     information when looking to see if data is flowing through the
     sender or if there are large jobs in progress>
Send Request 20028CEN   JobID: 00000050   DestSite: PRI   FinalSite:      State:  Pending   Status:            Action:    None   Total size:        1k   Remaining:        1k   Heartbeat: 18:07   Start: 12:00   Finish:   12:00   Retry:   
Deleting in-memory send requests that are no longer in inboxes.   
Deleting in-memory slave send requests that are not referenced by a  master request.   
========  Building Outbox Send Request Lists  ========   
Examining 2 send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests...    Sorting list of 2 send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests by priority and time...        
     appropriate sending folder if the aren’t already there – in this 
     case the LAN folder>
Examining 2 send requests for outbox \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN...   
Sorting list of 0 send requests for outbox \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN by priority and time...    Sorting complete, updating perf counters with number of master send requests...   
========  Checking Status of All Send Requests  ========   
====  Checking send requests for outbox C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests.   
    Checking unscheduled send request 20027CEN   
    Checking unscheduled send request 20028CEN        
     working on that>
========  Scheduling All Unscheduled Send Requests  ========   
Starting routing analysis for 2 requests   
Routing Analysis: Combining requests for same destination site and package...   
Completed routing analysis       
     content – this really is important if more than two tiers deep –
     this is where fan out distribution comes into play> 
====  Scheduling send requests.   
Starting routing analysis for 2 requests   
There is direct address to site PRI, no need for routing for send request 20027CEN.   
There is direct address to site PRI, no need for routing for send request 20028CEN.   
Routing Analysis: Combining requests for same destination site and package...   
Completed routing analysis   
    Scheduling send request 20027CEN.   
        Selected outbox.  From: C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests to \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN.   
    Scheduling send request 20028CEN.   
        Selected outbox.  From: C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\requests to \\STEVERACCEN\SMS_CEN\inboxes\schedule.box\outboxes\LAN. 
    
      scheduled for sending.and we cleanup>  
========  Cleaning Up Routing Packages  ========   
Time to clean up completed jobs and send requests list.   
Deleting completed jobs.   
Deleting job data that's not associated with valid jobs.   
========  Finished Processing Cycle ( 07/13/10 18:08:00 Pacific Daylight Time) ========   
Sleeping for maximum 300 seconds.   

Note:  Obviously there is a lot going on here and you would see even more if you looked at a production site with multiple sites and lots of activity.
Note:  Scheduler is a single threaded component.  That means that when it starts it has to work it’s way through the entire processing loop before it starts again.  On a very busy site it is common to see that one full loop of logging is not captured in a single log file.
So we’ve seen the processing in the log – what about the contents of the folders at the end of the processing loop?  As mentioned earlier, lets take them in order of processing.
If we look at the .job file in the root of schedule.box we notice some interesting things.

We definitely see some familiar content based on examining the RPL file but we see three new pieces of information that are worth noting.  First note the 0000004F identifier.  This is both a consistent identified that helps us link files together in schedule.box and also the filename of our job file.  We also see here the filename of our file in the tosend folder – 0000004F.I16 and also the filename of the send request for this job that has been placed in outboxes\lan to initiate sending – 20027CEN.  Review the other elements of the file to see what you can identify.
ToSend The to send folder is the staging area showing what content is to be sent to the destination.


Following our filename our job of interest is 0000004F.I16.  Reviewing the contents of this file we see the same identifier and more pointers back to our source.

Requests
Requests is a folder that we see as part of processing in the log entries above but it’s hard to catch data in this folder since requests is a quick stop along the way to outboxes\lan.
Outboxes\LAN
The outbox folders – in this case LAN since we are using the LAN sender – is where our send requests are placed to flag sender to wake up and begin sending the content defined in the tosend folder. Note that the .CPB and .DAT files are always present in this folder.

If we look at our appropriate SRQ file – 20027CEN.SRQ as found earlier in the .job file content – we see this file pulls everything together so the sending process can begin.  We will watch the sending process take place next.

UID Before leaving the schedule.box folder we should stop a the UID folder.

In this folder we have two files – a .REQ and a .JOB.  These are files used to track the progress of various items as they are sent through the scheduler.  These files are critical to the operation of scheduler and should never be deleted.  If they are you can recreate them as blank text files.  If you have one file but not the other just take the filename of the one you have, increment it by a hundred or so and create the missing file with that new filename.  If neither file are present you can recreate both – use a filename for both that would likely be above the value you might expect there.  I like to use a filename of 00100000 since it would take a really long time for a site to send that many jobs so the filename is likely safe and leaves plenty of room to grow.
OK, so lets SEND ALREADY!!!!  When we bring the sender back online the content is immediately seen and sending starts.  The relevant portion from sender.log is below with comments added.
Wrote 60123 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.PCK at position 21504   
    
Sending completed [C:\SMSPKG\CEN0000D.PCK]   
Finished sending SWD package CEN0000D version 1 to site PRI   
STATMSG: ID=3533 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_LAN_SENDER" SYS=STEVERACCEN SITE=CEN PID=2108 TID=3972 GMTDATE=Thu Jul 15 14:21:21.646 2010 ISTR0="CEN0000D" ISTR1="1" ISTR2="PRI" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=3 AID0=400 AVAL0="CEN0000D" AID1=410 AVAL1="1" AID2=409 AVAL2="PRI"   
    
Attempt to create/open the remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP   
Created/opened the remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP   
Attempt to create/open the remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP   
Created/opened the remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP   
Sending Started [C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\tosend\0000004F.I16]   
Attempt to write 1024 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP at position 0   
Wrote 1024 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP at position 0   
Attempt to write 32 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP at position 1024   
Wrote 32 bytes to \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP at position 1024   
Sending completed [C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\schedule.box\tosend\0000004F.I16]    Finished sending SWD package CEN0000D version 1 to site PRI   
STATMSG: ID=3533 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_LAN_SENDER" SYS=STEVERACCEN SITE=CEN PID=2108 TID=3972 GMTDATE=Thu Jul 15 14:21:21.658 2010 ISTR0="CEN0000D" ISTR1="1" ISTR2="PRI" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=3 AID0=400 AVAL0="CEN0000D" AID1=410 AVAL1="1" AID2=409 AVAL2="PRI"   
    
     itself and the instruction file>
Renaming remote file \\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP to \\steveracpri.startrekng.com\SMS_Site\20027CEN.SNI   
Rename completed [\\steveracpri.startrekng.com\SMS_Site\20027CEN.TMP]    Sending completed successfully
    
Note:  At this point all sending is complete from CEN to PRI – now PRI has to process that sent data forward. 
Notice also that we are sending to SMS_Site – that is a share name that translates to the inboxes\despoolr.box\receive folder on PRI. 

All sending, regardless of content, is sent directly to the SMS_Site folder on the destination site servers.  The content is sorted further from there.
How is it sorted further?  If we look at the despoolr.log on PRI we will see our .SNI file being processed.  Before we jump into the details of the log – a log reading tip.  Many components in SCCM are multi-threaded meaning they can work on multiple things at once.  This is potentially confusing as the entries in the log may get mixed between various lines of processing.  One way to keep things straight is the thread ID that is recorded in the log.  If we look at despoolr.log we see this very thing. 

Note that we see thread 5FC (1532) is being spawned to handle processing of our 20027CEN.SNI file.  There are several other despooling threads spawned after that and then the second hightlight shows thread 5FC starting to work.    You can also filter based on the thread ID – you have to use the decimal thread ID – which will display only the parts of the log that are processed on the thread ID you are interested in.  Thats what I did for the log snip below.

Just a couple of tips to help you keep things straight.
Now to the log showing processing of our .SNI file – with commentary added.
Verifying signature for instruction C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\despoolr.box\receive\ds_9rem4.ist of type MICROSOFT|SMS|MINIJOBINSTRUCTION|PACKAGE   
      
CPublicKeyLookup::CPublicKeyLookup("CEN")   
CPublicKeyLookup::CPublicKeyLookup("CEN") Initializing to file: C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\hman.box\pubkey\CEN.pkc   
CPublicKeyLookup::GetNextKey() Getting Iteration: 2   
CPublicKeyLookup::GetNextKey() Checking C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\hman.box\pubkey\CEN.pkc for Key0   
CPublicKeyLookup::GetNextKey() No Match Found, Trying C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\hman.box\pubkey\CEN.pkp   
CPublicKeyLookup::GetNextKey() Found Key: 0602000000A40000525341310004000001000100FD8307BF5E9F68D42C559EBD8183CF87F200C949130B9A16C54271CB03B657755D13DB82B55DE87A6290AC62C99367BC44A32EA8B77688E70A46C3FB5576C24CC3F3929F9E1F51F9A07CACC9BD11831D73526F3A0B6DBD3287FEA2653EAA5E3682B64183F2383857EC3A1590430C2EDFA6894A7A5D17938B95674F88D096E6D1                 
    
      CEN so we can validate this data came from a trusted source> Begin to calculate signature on C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\despoolr.box\receive\ds_9rem4.pkg using hash algorithm ID 0x800C   
Verified package signature from site CEN   
CPublicKeyLookup::CPublicKeyLookup("CEN")   
Signature checked out OK for instruction coming from site CEN, proceed with the instruction execution.   
    
Executing instruction of type MICROSOFT|SMS|MINIJOBINSTRUCTION|PACKAGE   
STATMSG: ID=4407 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DESPOOLER" SYS=STEVERACPRI SITE=PRI PID=1224 TID=1532 GMTDATE=Thu Jul 15 14:21:26.489 2010 ISTR0="CEN0000D" ISTR1="1" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="CEN0000D"  
      
Received package CEN0000D version 1   
Use drive C for storing the compressed package.   
Insert package CEN0000D to the distribution source.  
    
     where it gets interesting – in every other case I know of we
     forward to replmgr.box/incoming for further processing – I’ll
     show an example of that shortly> 
Stored Package CEN0000D. Stored Package Version = 1   
Forward package status for pkg CEN0000D to site CEN   
STATMSG: ID=4400 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DESPOOLER" SYS=STEVERACPRI SITE=PRI PID=1224 TID=1532 GMTDATE=Thu Jul 15 14:21:30.723 2010 ISTR0="CEN0000D" ISTR1="\\STEVERACPRI\C$\SMSPKG\CEN0000D.PCK" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="CEN0000D"   
Despooler successfully executed one instruction.    
And there you have it – despooler is finished and now the package is being nicely processed further by the distribution components.  So what about other types of replicated items – the log snip below shows another random transaction (not package related) where despooler forwards to replmgr.box/incoming to allow it to further process.
Signature checked out OK for instruction coming from site CEN, proceed with the instruction execution.   
Executing instruction of type MICROSOFT|SMS|MINIJOBINSTRUCTION|REPORTING   
Moved C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\replmgr.box\ut19p8x7.TMP\1ebn31f1.RPL to C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\replmgr.box\incoming\v8xgi1h6.RPL    Despooler successfully executed one instruction.   

As you can see – the trick to following the flow on the destination site is seeing which inbox despooler forwards to after processing.
If replmgr is used to handle the file type all it does is look at the content and forward it on – something similar to the below log snip.
Moved incoming replication file to C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\objmgr.box\INCOMING\CEN_24.SDM   
Moved incoming replication file to C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\ciamgr.box\CEN_2.CAR   
Moved incoming replication file to C:\Program Files (x86)\Microsoft Configuration Manager\inboxes\objmgr.box\INCOMING\CEN_11.CID 
Before we wrap up – lets go back to discussion of despooler.  What exactly does a .SNI file look like anyway?  A sample .SNI file is below.

Beyond SNI’s there may be other accompanying files that need to be processed as part of the instruction – in this case there is also a .pck as shown.

Not a great deal to discuss about the files above but I show them for completeness of the flow.
So there you have it – site to site communication end to end.  Will this flow exactly match your scenario?  Likely not – but generally should be similar.  Hope this helps!