NGWS SDK Documentation  

This is preliminary documentation and subject to change.
To comment on this topic, please send us email at ngwssdk@microsoft.com. Thanks!

Lifetime

Lifetime management features help keep actual consumption of isolated storage within bounds set by quotas with a minimum user/admin intervention, working as automatically as feasible (given a simple implementation) to efficiently utilize storage. From time to time the system reclaims expired or expendable data.

Isolated storage data lifetime is managed by the system in units of domain (application) identity scopes.

Data can have an expiration date – or not – and can optionally be marked for guaranteed retention as detailed below. The following table shows how these features combine in use.

Expiration Retention Description Reclaimable?
in future Not guaranteed keep around unless space runs out second choice
expired (any) exceed expiration date first choice
in future Guaranteed must be kept until expiration no
none (permanent) Guaranteed must be kept indefinitely no
none Not guaranteed this is not possible  

Expiration

Expiration dates are set to allow the system to automatically reclaim old information no longer needed. There are two purposes for having expiration of data: applications may volunteer to have data removed after a given date, or restricted permission may force an application to only be able to keep persistent data in the isolated store for a certain maximum length of time. Either way, expiration is important to prevent a continual buildup of old data that may no longer be used.

Expiration is set by the application code in terms of days since last use. Any read or write access resets the expiration for another N days in the future, but the number of days since last use remains the same.

We can implement expiration without needing a background service to actively reclaim storage. Expiration is voluntary rather than mandatory, meaning that if storage expires on a certain day (say, a holiday) we don’t necessarily have to remove it then; rather we can expire the data when the store is next used in a more “lazy” fashion. Also a simple tool that passes over the stores deleting expired data can easily be fired off periodically from existing OS services, say at 3am every day.

Maximum expiration

The machine administrator can set the maximum number of days information can be marked for expiration on a machine-wide basis. Only data marked for retention (see next section) is exempt from this limitation. For example, if the parameter is set to 30 days then an attempt to set a longer expiration will result in the smaller value being used. See section 3.1.

Guaranteed retention of data

Given permission to do so, data in isolated storage may be created marked for guaranteed retention. Guaranteed retention can be used with an expiration date or without (meaning it never expires). Guaranteed retention data is not subject to administrative maximum expiration date restriction described above.

This storage can only be deleted by the code with this identity and necessary permission, or by explicit user/admin action. However, this data will be deleted when expiration date is exceeded.

All other storage is “expendable” and subject to reclamation before expiration if quotas are exceeded.

Reclamation

Reclamation is the process of deleting data in a store to make room for more, both: old expired data can be automatically deleted to make room, or as a next step any expendable (but not guaranteed retention) unexpired data.

Reclamation happens when:

  1. a store reaches total maximum quota and more room is needed
  2. automatically as a storage consumption level for a store is exceeded (details follow)
  3. as an explicit administrative operation (a simple utility)

When a write to isolated store exceeds some quota reclamation should be attempted before raising an exception indicating an error. Only after attempted reclamation fails to get back any storage should the exception be fired. This will potentially cause a significant performance hit to the unlucky application that triggers this, however until we can implement a more sophisticated background reclamation service the slow-down is preferable to an error. Implementation should remember when reclamation has been done and minimize cost of repeated attempts (for instance, if no new expendable storage is created since the last reclamation any subsequent attempt is sure to not find anything and can be completely skipped).

We should build in a simple heuristic to do automatic reclamation (otherwise, clean up of expired data only happens when store is full – resulting in often-near-full stores which is not a desirable common state).

Here’s one simple algorithm (with experience we can adjust the parameters):

Used = bytes used; Trigger, Step = variables; Max = store quota in bytes

  1. For a new store set Step = min ( Max*0.25, 10MB )
  2. For a new store set Trigger = Step
  3. On close of each substore, check if Used > Trigger, continue if so
  4. Reclaim expired storage in the entire store
  5. If Trigger –Used< Step/4 (i.e. not much below Trigger) then
    set Trigger = Trigger + Step, else leave Trigger as is
  6. Go to 3 to wait for another chance

Any store that is currently in use is always exempt from reclamation.