For AI agents
Most people writing against this library now have an agent open beside them. The rest of these docs are written for a human reading one page at a time, which is the wrong shape for that: an agent needs the whole surface at once, in plain text, with the exact spellings.
So there is one file, in thirty-five sections. It carries every public type and member of the four packages, the behaviour behind them, the JSON a client sends and receives, every error string, and the traps that produce code which compiles and is quietly wrong. An agent that reads it needs no other page here.
How to use it
Either hand it over, or let the agent fetch it.
Read https://doc.dynamicwhere.com/llms.txt before writing any
DynamicWhere.ex code. It is the complete API surface.That works with any agent that can read a URL. If yours cannot, use the copy button below and paste the file into your context.
Why plain text rather than a nicer page
What an agent consumes is the text. Syntax highlighting, cards and collapsible sections cost context and carry no meaning once the markup is stripped.
llms.txt is also the path agents and crawlers already look for, so pointing at it needs no explanation.What is in it
- Shapes and results. Every property of
Condition,ConditionGroup,ConditionSet,OrderBy,GroupBy,AggregateBy,PageBy,Filter,SegmentandSummarywith its type and default, and what each member of the three result types holds. - Enums, verbatim, with their numbers — the numbers a JSON body must send when the host registers no string enum converter — including the case-insensitive
Ivariants and the one-mspelling ofSumation. - All twenty-eight extension methods with their real signatures, what each one validates, and which have no synchronous or in-memory form. Plus the generated predicate for every operator, value coercion per
DataType, and how field paths resolve. - The JSON on the wire. Which body binds to which shape, casing and enum converters, how
valuesmust be typed, the result envelope, what rows look like per method, and copy-paste recipes. - Validation and errors. Every rule in the order it is checked, all thirty-one error strings with what raises them, and the other exception types a caller can receive.
- The policy layer. All twenty-two attributes with their parameters, the six precedence levels, enforcement tier by tier, the transform chain with the exact output of every mask and generalize mode, the group floor, dynamic rules and stores, the admin API, and all twenty-two policy error codes.
- The reflection cache. Every
CacheExposemember, the options and their ranges, the presets, and what eviction actually does. - Fifty-four traps that produce silently wrong code: sixteen for the query engine, thirty-eight for policies. A mask without
[DwNoOrder]leaking through sorting is the one an agent reproduces most often, because the attribute reads as sufficient on its own. - Worked examples for a filter, a summary, a segment, an endpoint, a fully protected entity and the policy wiring around it.
The file
This is the exact content served at /llms.txt. The page reads it at build time, so the two are never out of step.
llms.txt · 6,030 lines
# DynamicWhere.ex — complete reference for coding agents
> A .NET library that turns JSON filter objects into Entity Framework Core LINQ queries, with an
> opt-in field-level policy layer that decides what each caller may filter, sort, select, group,
> aggregate and see, and a reflection cache that needs no setup.
>
> Version 3.3.0 · targets net6.0 · runs on .NET 6, 7, 8, 9, 10 · EF Core 6+ · MIT
> Docs: https://doc.dynamicwhere.com · Source: https://github.com/Sajadh92/DynamicWhere.ex
This file is the whole library in one pass: every public type and member of the four packages, the
behaviour behind them, the JSON a client sends and receives, every error, and the traps that produce
code which compiles and is quietly wrong. It was written from the source and checked by running the
library. No other page is needed; where another page disagrees with this file, this file is right.
Where a name is not in this file, it does not exist — do not invent members.
```
dotnet add package DynamicWhere.ex --version 3.3.0
dotnet add package DynamicWhere.ex.Policies.Redis # optional, same version as the core
dotnet add package DynamicWhere.ex.Policies.EntityFrameworkCore # optional, same version as the core
dotnet add package DynamicWhere.ex.Policies.AspNetCore # optional, same version as the core
```
How to read it:
- Querying with JSON filters: sections 1–11. Read 11 (traps) before writing code.
- Field-level policies: 12 and 13 first, then 14–31 as needed. Read 32 (traps) before writing policy attributes.
- Errors: 8 (query engine) and 30 (policies).
- The reflection cache needs nothing by default: 33.
```
Contents
1 Packages, dependencies, namespaces
2 Shapes and results
3 Enums — `DynamicWhere.ex.Enums`
4 Conditions: operators, values, generated predicates
5 Field paths
6 Extension methods
7 Grouping and aggregation — Group<T>, Summary<T>, ToList(Summary)
8 Validation and core error strings
9 JSON wire format
10 JSON recipes
11 Traps — query engine
12 Policy lifecycle and configuration
13 Guarded queries
14 Policy attributes
15 Policy enums
16 Precedence
17 Enforcement by tier and dry run
18 Transforms
19 Group floor and transformed summaries
20 Model validation
21 Trace, explain and custom providers
22 Audit
23 Discovery: catalogue, schema, simulation
24 Token vaults
25 Dynamic rules
26 Rule stores and StorePolicyProvider
27 Redis package — DynamicWhere.ex.Policies.Redis
28 Entity Framework Core package — DynamicWhere.ex.Policies.EntityFrameworkCore
29 ASP.NET Core package — DynamicWhere.ex.Policies.AspNetCore
30 Policy error codes
31 Policy recipes and inference channels
32 Traps — policies
33 Reflection cache
34 Version history, breaking changes and limits
35 Worked examples (C#)
```
## 1. Packages, dependencies, namespaces
Four packages, always the same version. All target `net6.0` and run on .NET 6–10.
```
DynamicWhere.ex the query engine, policies, token vault interface, cache
Microsoft.EntityFrameworkCore 6.0.22
System.Linq.Dynamic.Core 1.6.7
Microsoft.Extensions.Caching.Memory 6.0.2 (3.3.0)
Microsoft.Extensions.Configuration.Abstractions 6.0.0
Microsoft.Extensions.Configuration.Binder 6.0.0
Microsoft.Extensions.DependencyInjection.Abstractions 6.0.0
DynamicWhere.ex.Policies.Redis RedisPolicyStore, RedisTokenVault
StackExchange.Redis 2.8.24
DynamicWhere.ex.Policies.EntityFrameworkCore EfPolicyStore, EfTokenVault, DwPolicyDbContext + configurations
Microsoft.EntityFrameworkCore.Relational 6.0.22
DynamicWhere.ex.Policies.AspNetCore MapDwPolicyAdmin, claims adapter, audit middleware
FrameworkReference Microsoft.AspNetCore.App
```
- `Microsoft.Extensions.Caching.Memory` is named for its patched version (3.3.0); the library does not use it
directly. EF Core 6.0.22 asks for 6.0.1 or later, and 6.0.1 is the last version open to CVE-2024-43483
(GHSA-qj66-m88j-hmgj), so a host on the EF Core 6 floor resolved a vulnerable version through all four packages.
Naming 6.0.2 raises that floor for all four. A host on EF Core 8 or later already resolves a newer one and sees
no change.
- The engine parses every expression it builds with its own `ParsingConfig`: System.Linq.Dynamic.Core's defaults
with `AreContextKeywordsEnabled = false`. It does not read `ParsingConfig.Default` (since 3.1.0), so a change a host
makes there does not reach DynamicWhere queries.
Namespace of every public type:
```
DynamicWhere.ex.Source Extension (all core extension methods), DwDates, DwDateOptions (date formats, 3.1.0)
DynamicWhere.ex.Classes.Core Condition ConditionGroup ConditionSet OrderBy GroupBy AggregateBy PageBy
DynamicWhere.ex.Classes.Complex Filter Segment Summary
DynamicWhere.ex.Classes.Result FilterResult<T> SegmentResult<T> SummaryResult
DynamicWhere.ex.Enums DataType Operator Connector Direction Intersection Aggregator
DynamicWhere.ex.Exceptions LogicException PolicyException
DynamicWhere.ex.Policies.Source PolicyExtensions (ApplyPolicy) PolicyQueryable<T>
DynamicWhere.ex.Policies.Config DwPolicy DwPolicyOptions DwCaps DwPolicyConfiguration (AddDwPolicies, Bind)
DynamicWhere.ex.Policies.Context DwPolicyContext DwSubject
DynamicWhere.ex.Policies.Attributes DwPolicyAttribute and the 22 Dw*Attribute types
DynamicWhere.ex.Policies.Enums PolicyFeature PolicyEffect PolicyAction PolicyLevel PolicyErrorCode
MaskStrategy GeneralizeMode DatePart DwTier DwSubjectKind StoreFailureMode
DynamicWhere.ex.Policies.Masking IValueTransformer DwTransformContext
DynamicWhere.ex.Policies.Tokens IDwTokenVault InMemoryTokenVault DwToken
DynamicWhere.ex.Policies.Audit IDwAuditSink DwAuditEvent
DynamicWhere.ex.Policies.Discovery DwEntityCatalog PolicySchemaBuilder PolicySchema PolicySchemaField
PolicySchemaNode PolicySchemaRequest PolicySimulator PolicySimulation<TClause>
DynamicWhere.ex.Policies.Resolution IDwPolicyProvider AttributePolicyProvider StorePolicyProvider PolicyResolver
DynamicWhere.ex.Policies.Storage IDwPolicyStore IDwPolicyWritableStore IDwPolicyRefresher InMemoryPolicyStore
PolicyRule PolicyRuleDocument PolicyPayload RuleDetail SealedFields
StoreSnapshot NarrowZone
DynamicWhere.ex.Policies.DTOs PolicyTrace PolicyDecision PolicyExplanation FeatureExplanation FieldPolicy
FieldFacts ForcedPredicate PolicyFragment PolicySource TypePolicy
ValueTransform TransformStage TransformKind MutateStage GeneralizeStage
FormatStage MaskStage TruncateStage DefaultStage
DynamicWhere.ex.Policies.Validation PolicyModelValidator PolicyModelReport
DynamicWhere.ex.Optimization.Cache.Source CacheExpose
DynamicWhere.ex.Optimization.Cache.Config CacheOptions
DynamicWhere.ex.Optimization.Cache.Enums CacheEvictionStrategy CacheMemoryType
DynamicWhere.ex.Optimization.Cache.DTOs CacheStatistics CacheMemoryUsage CacheConfiguration
CachePerformanceEvaluation CacheMonitoringSession
DynamicWhere.ex.Optimization.Cache.Input AccessTrackingInput<TKey> CacheFullCheckInput HealthAlertsInput MemoryCalculationInput
DynamicWhere.ex.Optimization.Cache.Output CacheCounts CacheDatabases TrackingCounts
DynamicWhere.ex.Policies.Redis (Redis package) RedisPolicyStore RedisTokenVault
DynamicWhere.ex.Policies.EntityFrameworkCore (EF Core package) DwPolicyDbContext EfPolicyStore EfTokenVault
DwPolicyRuleConfiguration DwPolicyTokenConfiguration
DwPolicyVersionConfiguration DwPolicyRuleRecord DwPolicyTokenRecord
DwPolicyVersionRecord
DynamicWhere.ex.Policies.AspNetCore (ASP.NET Core package) DwPolicyEndpoints (MapDwPolicyAdmin)
DwPolicyAdminOptions DwClaimsOptions DwClaimsAdapter
ClaimsPrincipalPolicyExtensions DwPolicyHttpContextExtensions
DwPolicyAuditMiddleware DwPolicyAuditMiddlewareExtensions
SchemaRequest ExplainRequest SimulateRequest RuleRequest
```
The typical usings: `DynamicWhere.ex.Source`, `.Classes.Core`, `.Classes.Complex`, `.Classes.Result`, `.Enums`;
add `.Policies.Source` for `ApplyPolicy`, `.Policies.Config` for `AddDwPolicies`/`DwPolicy`,
`.Policies.Context` for `DwPolicyContext`, `.Policies.Attributes` and `.Policies.Enums` on entities.
---
## 2. Shapes and results
Three request shapes. Each is a plain class with public get/set properties and no JSON attributes.
- `Filter` — where → order → page → select, one query; returns `FilterResult<T>`
- `Summary` — where → group + aggregate → having → order → page, one query; returns `SummaryResult`
- `Segment` — several condition sets combined with Union / Intersect / Except into one query, then ordered, paged
and projected like a filter; returns `SegmentResult<T>`
Unset enum properties take member 0. Lists start empty unless the property type has `?`.
```
Core — DynamicWhere.ex.Classes.Core
Class Property Type Default Meaning
Condition Sort int 0 order in its group; unique among that group's Conditions
Field string? null member path on T; required
DataType DataType Text predicate form and value parsing (section 4)
Operator Operator Equal
Values List<object> [] operands; count fixed by Operator; null is read as []
ConditionGroup Sort int 0 order among sibling sub-groups; unique among them
Connector Connector And joins every child of this group
Conditions List<Condition> []
SubConditionGroups List<ConditionGroup> [] nests to any depth
ConditionSet Sort int 0 order of the set operations; unique in the Segment
Intersection Intersection? null required on every set but the lowest Sort (ignored there)
ConditionGroup ConditionGroup new() this set's where
OrderBy Sort int 0 lower applies first; duplicates allowed, ties keep list order
Field string? null member path on T; required
Direction Direction Ascending
PageBy PageNumber int 0 1-based; must be >= 1
PageSize int 0 must be >= 1
GroupBy Fields List<string> [] >= 1 paths, unique (case-insensitive), each ending on a simple type
AggregateBy List<AggregateBy> [] optional
AggregateBy Field string? null member path on T; may be omitted only for Count
Alias string? null required identifier; names the result column
Aggregator Aggregator Count
```
```
Complex — DynamicWhere.ex.Classes.Complex
Filter ConditionGroup ConditionGroup? null null = no where
Selects List<string>? null null = whole entities; [] throws MustHasFields
Orders List<OrderBy>? null [] = no ordering; guarded: T's DefaultOrder (section 13)
Page PageBy? null null = every row
Segment ConditionSets List<ConditionSet> [] [] = runs as ToListAsync(new Filter { Selects, Orders, Page })
Selects List<string>? null projected last, after ordering and paging
Orders List<OrderBy>? [] applied in SQL to the combined rows; [] as for Filter
Page PageBy? null applied in SQL after ordering
Summary ConditionGroup ConditionGroup? null where, before grouping
GroupBy GroupBy? null required; null throws ArgumentNullException
Having ConditionGroup? null each Condition.Field is an AggregateBy.Alias, not a path
Orders List<OrderBy>? null each Field is a GroupBy field or an Alias
Page PageBy? null pages the groups
Clone() on Filter, Segment and Summary public since 3.3.0; deep copy, every node new,
a condition's values shared in a new list
with the original: the condition tree with its
groups and conditions, the Selects list, each
OrderBy, the Page, and a Summary's GroupBy with
its AggregateBy list and its Having. A null
branch stays null, and a null entry inside a
list is copied as a null entry (3.3.0), so the
refusal is the running method's and reads the
same for a copy; it used to throw
NullReferenceException. Use it to read one
request again with a different page or order
without editing what the caller handed in.
```
```
Results — DynamicWhere.ex.Classes.Result
FilterResult<T> PageNumber:int PageSize:int PageCount:int TotalCount:int Data:List<T> QueryString:string? Policy:PolicyTrace?
SegmentResult<T> : FilterResult<T>, no members of its own
SummaryResult a separate class, not a FilterResult: the same seven members with Data:List<dynamic>
FilterResult<dynamic> is what ToListDynamic and ToListAsyncDynamic return
PageNumber Page.PageNumber; 0 when Page is null
PageSize Page.PageSize; 0 when Page is null
TotalCount counted before paging: rows matching the where (Filter), groups left after Having (Summary),
rows left after the set operations (Segment)
PageCount (int)Math.Ceiling((double)TotalCount / PageSize), or 1 when no Page was sent (0 with no rows),
on FilterResult, SummaryResult and SegmentResult alike. Before 3.1.0 an unpaged Filter or Summary
reported PageCount = TotalCount (one page per row) and an unpaged Segment with sets reported 0.
Unpaged, PageNumber and PageSize stay 0 — except on a guarded query when DwCaps.DefaultPageSize is
set, which gives it page 1 at min(DefaultPageSize, MaxPageSize) (section 12)
Data the page. Typed rows are whole T objects even with Selects: unselected members keep constructor defaults
QueryString with getQueryString: true, EF Core ToQueryString() of the data query (after order, page and projection;
never the count query), otherwise null. Always null for Segment, which has no such parameter.
On a non-EF source it holds the text "The given 'IQueryable' does not support generation of query strings."
Under ApplyPolicy in the Strict tier, getQueryString: true throws QueryStringDenied
Policy PolicyTrace written by the ApplyPolicy terminals (section 21); null otherwise. Under the Strict
tier null too, unless DwPolicyOptions.IncludeTraceInResult is true (3.1.0); LastTrace still holds it
```
---
## 3. Enums — `DynamicWhere.ex.Enums`
Values are implicit, in declaration order. None is `[Flags]`. Numbers are what a JSON body sends when the
host has no string enum converter (section 9).
```
DataType Text=0 Guid=1 Number=2 Boolean=3 DateTime=4 Date=5 Enum=6
Operator Equal=0 IEqual=1 NotEqual=2 INotEqual=3
Contains=4 IContains=5 NotContains=6 INotContains=7
StartsWith=8 IStartsWith=9 NotStartsWith=10 INotStartsWith=11
EndsWith=12 IEndsWith=13 NotEndsWith=14 INotEndsWith=15
In=16 IIn=17 NotIn=18 INotIn=19
GreaterThan=20 GreaterThanOrEqual=21 LessThan=22 LessThanOrEqual=23
Between=24 NotBetween=25 IsNull=26 IsNotNull=27
Connector And=0 Or=1
Direction Ascending=0 Descending=1
Intersection Union=0 Intersect=1 Except=2
Aggregator Count=0 CountDistinct=1 Sumation=2 Average=3 Minimum=4 Maximum=5 FirstOrDefault=6 LastOrDefault=7
```
- `Sumation` has one `m`. There are no `Sum`, `Avg`, `Min` or `Max` members.
- The `I` prefix means case-insensitive and exists only for text operators (`IEqual`, `IContains`, `IIn` …).
- `FirstOrDefault` / `LastOrDefault` return the smallest / largest value in the group, not the first / last row.
---
## 4. Conditions: operators, values, generated predicates
### DataType × Operator
A pair not listed throws `LogicException("Unsupported combination of DataType 'Guid' and Operator 'GreaterThan'.")`
when the predicate is built, after the value checks have passed.
```
DataType Operators accepted
Text Equal NotEqual Contains NotContains StartsWith NotStartsWith EndsWith NotEndsWith In NotIn,
the I-variant of each of those ten, IsNull IsNotNull (no ranges)
Guid Equal NotEqual In NotIn IsNull IsNotNull
Number Equal NotEqual GreaterThan GreaterThanOrEqual LessThan LessThanOrEqual Between NotBetween
In NotIn IsNull IsNotNull
Boolean Equal NotEqual IsNull IsNotNull
DateTime Equal NotEqual GreaterThan GreaterThanOrEqual LessThan LessThanOrEqual Between NotBetween
IsNull IsNotNull (no In / NotIn)
Date same as DateTime
Enum Equal NotEqual In NotIn IsNull IsNotNull
Contains NotContains StartsWith NotStartsWith EndsWith NotEndsWith (no I-variants)
```
Pick the DataType from the member's CLR type. It is never checked against the member, so a mismatch is not
refused: `Guid` on a `string` member matches nothing, `Text` on a numeric member is converted by the parser.
```
Member type DataType Notes
string Text also Enum, for a string column holding enum names
int long short byte decimal double … Number also works on an enum-typed member with its numeric value
bool / bool? Boolean
Guid / Guid? Guid
DateTime DateTime exact instant comparison
DateTime Date compares .Date on both sides
DateTime? / DateTimeOffset / DateTimeOffset? / DateOnly / DateOnly?
DateTime also Date; the predicate is built from the member's type
(3.1.0; before it, DataType.Date on DateTime? threw)
enum (stored as int or as string) Enum Equal NotEqual In NotIn IsNull IsNotNull; value by member name (any
case) or by number. Contains/StartsWith/EndsWith on an enum-typed
member throw ParseException ("No applicable method 'Contains' exists
in type '<Enum>'") whatever the storage
List<string> and other simple-value — a Where on the collection itself throws ParseException ("Operator '=='
collections incompatible with operand types 'List`1' and 'String'"); ordering by it works
```
### Generated predicate
The predicate is a System.Linq.Dynamic.Core string. `f` is the member access, `V` the value after trimming and escaping.
```
Operator Predicate
Equal / NotEqual f != null && f == V f != null && f != V
Contains / NotContains f != null && f.Contains(V) f != null && !f.Contains(V)
StartsWith / NotStartsWith f != null && f.StartsWith(V) f != null && !f.StartsWith(V)
EndsWith / NotEndsWith f != null && f.EndsWith(V) f != null && !f.EndsWith(V)
I-variants (Text only) f.ToLower() in place of f; V lowered in C# with string.ToLower() (server culture)
In f != null && (f == V1 || f == V2 || ...)
NotIn f != null && (f != V1 && f != V2 && ...)
GreaterThan … LessThanOrEqual f != null && f > V (>=, <, <=)
Between f != null && f >= V1 && f <= V2 inclusive; bounds used in the order given
NotBetween f != null && (f < V1 || f > V2)
IsNull / IsNotNull f == null f != null
V by DataType Text, Guid, Enum "v" Number, Boolean v (unquoted)
DateTime DateTime.Parse("canonical") for a DateTime member;
DateTimeOffset.Parse("canonical") for a DateTimeOffset member
Date the same call with .Date on both sides; f becomes f.Date, or f.Value.Date
where the member is nullable
```
- Every operator except IsNull / IsNotNull starts with `f != null &&`. Negated
operators (NotEqual, NotIn, NotContains, NotBetween …) therefore never return rows whose member is null.
- The date types are the exception: since 3.1.0 they resolve the member's type first and emit the guard only
where the member is actually nullable (section 4, Date / DateTime). Every other DataType still guards
unconditionally.
- IsNull on a non-nullable value member matches nothing; IsNotNull matches everything. On a non-nullable date
member of the entity itself the predicate is now the constant `false` / `true` rather than a comparison with
null. Reached through a navigation, IsNull / IsNotNull test the navigation instead (Date / DateTime below).
- Values are trimmed; `\` and `"` are escaped. A value matches literally and cannot close the literal or inject
predicate text. Values are inlined as literals, not SQL parameters.
- A list of more than 32 values (3.1.0) is nested as a balanced tree of flat chains of at most 32 terms, all joined by
the same operator: 50 values of an `In` become `f != null && ((f == V1 || … || f == V25) || (f == V26 || … || f == V50))`.
This covers `In`, `NotIn`, `IIn` and `INotIn` on Text and `In` / `NotIn` on Guid, Number and Enum. A list of 32 or
fewer is written exactly as the table shows, so its predicate and SQL are unchanged, and a longer list returns the
same rows.
- Security fix. Before 3.1.0 every list was one flat chain, one level of nesting per value, and EF Core and the
expression compiler walk that tree recursively: a condition carrying about seven hundred values overflowed the
request thread's stack. A stack overflow ends the process, and no `catch` can stop it. Guarded and unguarded
queries alike, since before 3.0.0.
- Under `ApplyPolicy`, `Caps.MaxConditionValues` (default 1000) also bounds the values of one condition (section 12).
- `Between` with V1 > V2 matches nothing; `NotBetween` with V1 > V2 matches every non-null row.
- Plain text operators add no case handling: in memory they are ordinal; in SQL the provider and collation
decide (SQLite: `==` and Contains case-sensitive, StartsWith/EndsWith case-insensitive for ASCII). The
I-variants are case-insensitive everywhere. `ToLower()` on the column can defeat an index.
### Values
Each element of `Values` is first normalized to a string:
```
Element Normalized to
null, JsonElement Null ""
string itself
bool "true" / "false"
JsonElement String its string
JsonElement Number the raw token as sent ("1.50" stays "1.50")
JsonElement True / False "true" / "false"
JsonElement Array / Object the raw JSON text
DateTime "yyyy-MM-ddTHH:mm:ss.FFFFFFF", no zone marker (3.1.0; was "MM/dd/yyyy HH:mm:ss");
"yyyy-MM-ddTHH:mm:ss.FFFFFFFzzz" for a Kind Local value compared under DataType.DateTime
with a DateTimeOffset member (3.1.0; see Date / DateTime below)
DateTimeOffset "yyyy-MM-ddTHH:mm:ss.FFFFFFFzzz" (3.1.0)
DateOnly "yyyy-MM-dd" (3.1.0)
other IFormattable ToString(null, InvariantCulture): 12.5 -> "12.5", Guid -> "D" form, enum -> member name
anything else ToString()
```
Then checked per DataType. A failed check throws `InvalidFormat` — or, for a date, `AmbiguousDateFormat`.
```
DataType Check Send
Text none a string; a null element is "" and matches empty strings only
Enum none member name in any case ("Pending", "pending") or its number
Guid Guid.TryParse any Guid format: "D", upper-case, "N" (no hyphens)
Number the parser's own grammar, then the member's type a JSON number or numeric string: 12, -3.5, "15.5", "1e3";
(3.3.0); invariant, not the server's culture see the Number bullet below for what is refused
Boolean bool.TryParse true / false as JSON booleans or strings in any case; 1 and 0 fail
DateTime ISO 8601 / year-first / a declared format (3.1.0) ISO 8601: "2024-06-15T14:30:00", or with Z / an offset
Date ISO 8601 / year-first / a declared format (3.1.0) ISO 8601 date: "2024-06-15"; the time is dropped on both sides
```
- **Null:** a null element is `""`. Text and Enum compare with the empty string; every other DataType throws
`InvalidFormat`. To test for NULL use `IsNull` / `IsNotNull` with `"values": []`.
- **Number (3.3.0 rewrote this):** the token is embedded unquoted exactly as sent, so a value is read the way the
parser reads it rather than the way the host's culture does. Two steps.
- **The grammar**, in the invariant culture, ASCII digits only: optional white space around it (space, tab, LF,
VT, FF, CR, exactly what `TryParse` allowed), an optional minus, digits, an optional fraction (a point with a
digit on both sides), an optional exponent (`e` or `E`, an optional sign, digits). No leading plus, no
thousands separator, no trailing sign, no parentheses, no `NaN` and no `Infinity`. An integer — no fraction,
no exponent — must fit `UInt64`, or `Int64` when negative. A real has no bound: `1e400` still reads as
infinity.
- A suffix (`5L`, `5m`, `5f`, `5d`), hex (`0x1F`), parentheses (`(5)`) and a minus standing apart from its
digits (`- 5`) are refused as they always were, though the parser would read them. Nothing is accepted now
that was not accepted before.
- **The member**, in a `WHERE` condition only, for `Equal`, `NotEqual`, `In`, `NotIn`, `GreaterThan`,
`GreaterThanOrEqual`, `LessThan`, `LessThanOrEqual`, `Between` and `NotBetween`: the literal has to compare
with the member the condition names. The parser itself is asked, against the member's **declared** type — a
collection at the end of the path stands for itself, one along the path stands for its elements. Refused now,
where the parser used to throw:
- a literal written with a point and no exponent (`1.5`, `0.1`, `1.0`) on a **nullable** integral member
(`int?`, `long?`, …). A non-nullable `int` still takes `1.5`, exactly as before, and an exponent form
(`1e5`) still compares with an `int?`;
- an exponent form (`1e5`, `1E-7`) on `decimal` or `decimal?`, and a real with more digits than a `decimal`
holds (`79228162514264337593543950335.5`);
- an integer above `Int64.MaxValue` on a signed integral member (`sbyte`, `short`, `int`, `long`, nullable or
not): such a literal reads as a `ulong`, which none of them converts to. `byte`, `ushort`, `uint`, `ulong`,
`float`, `double` and `decimal` take it;
- a negative number on `ulong` or `ulong?`;
- any number on a `string`, `bool`, `Guid`, `DateTime` or `char` member, or on a collection of simple values
(`Scores` where `Scores` is a `List<int>`; `Lines.Quantity` still works, since the path steps through);
- a nullable enum with an ordering operator. Equality works on it, and a non-nullable enum orders.
- A `Having` condition reads the grammar and stops there: an alias names an aggregate, so there is no member
type to ask the parser about.
- Every refusal is `LogicException` with `InvalidFormat`. Nothing that ran before is refused now: every value
refused is one the parser refused. Under `ApplyPolicy` it is the same in both tiers, and the gate still
answers first — a denied field is `FieldDeniedForWhere` before any value is read.
- Before 3.3.0 the check was `byte` / `short` / `int` / `long` / `float` / `double` / `decimal` `TryParse` in
the host's culture. `"1,000"`, `"5-"`, `"+5"`, `".5"`, `"5."`, `"-.5"`, `"1.e5"`, `"NaN"`, `"Infinity"`,
`"-Infinity"` and an integer past `UInt64` (or below `Int64` when negative) all passed validation and then
threw `System.Linq.Dynamic.Core.Exceptions.ParseException`, which a host maps to a server error. `"1,5"`
passed on a de-DE host and was refused on an en-US one. And `"NaN"` and `"Infinity"` were written into the
expression as identifiers, so on a type with a member of that name the condition compared two columns.
- A number a C# caller places in `Values` is still written in the invariant culture (the normalizer table
above), so `12.5` is `"12.5"` on every host. `double.NaN` in `Values` is now `InvalidFormat`.
- JavaScript's `JSON.stringify(0.0000001)` is `1e-7`, which a `decimal` member refuses. Send `"0.0000001"`.
- **Date / DateTime (3.1.0 rewrote this):** the predicate is built from the member's own type.
- **Which texts are dates (3.1.0).** Read against explicit formats, never the lenient parser; the server's culture
and calendar decide nothing.
- Always accepted: ISO 8601 extended calendar dates — `"2026-09-01"` (also `"2026-9-1"`), optionally `T` or a
space and a time (`"12:30"`, `"12:30:15"`, `"12:30:15.123"`), optionally `Z` or an offset (`"+03:00"`,
`"+0300"`, `"+03"`) — and year-first dates `"2026/09/01"`, `"2026.09.01"` with the same optional time. A
lowercase `t` or `z` and a comma before the fraction are accepted, and a fraction longer than seven digits (Go
and Java write nine) is cut to seven, the 100 ns a `DateTime` holds.
- The other ISO 8601 forms are `InvalidFormat`: basic (`"20260901"`), week (`"2026-W36-2"`), ordinal
(`"2026-244"`) and reduced precision (`"2026-09"`, `"2026-09-01T12"`).
- A numeric date that leads with a day or a month — `"01/09/2026"`, `"15/09/2026"`, `"09/15/2026"`,
`"01.09.2026"`, `"01-09-2026"`, `"1/9/26"` — throws `AmbiguousDateFormat` whatever its numbers, with the field
as `LogicException.Subject` (under `ApplyPolicy`, the name the caller wrote, so an alias is not undone). By
shape, not value: refusing only the values with two readings would fail on the 5th of the month and pass on
the 15th.
- Anything else is `InvalidFormat`, including `"12:00"`, `"1/9"`, `"Sep 2026"`, `"1 September 2026"` — which
the lenient parser used to accept as today at noon, a day of the current year (9 January or 1 September, by
the host's culture), and 1 September.
- A deployment declares a local form once at startup, and it is read with the invariant culture alongside ISO:
`DwDates.Configure(o => o.Formats.Add("dd/MM/yyyy"))` makes `"01/09/2026"` 1 September everywhere. See
"Date formats" below.
- Validation and the builder read a value with the same reader and the member's own type, so a value that passes
validation always builds.
- A `DateTimeOffset` member is compared against a `DateTimeOffset` literal, normalised to UTC; a value carrying
no zone is read as UTC, so `DataType.Date` names the day the caller wrote. On Npgsql `DataType.Date` becomes
`date_trunc('day', col AT TIME ZONE 'UTC')`.
- The member's day under `DataType.Date` is the provider's: its UTC day on PostgreSQL, where `timestamptz` keeps
no offset, but the day in its own offset in memory (and on a provider that stores the offset, such as SQL
Server `datetimeoffset`). A row at `2026-09-01T01:00+03:00` is 31 August on PostgreSQL and 1 September in
memory. The value's day is always its UTC day, so send a date with no zone for a day comparison.
- A `DateTime` member is compared against a `DateTime` literal and keeps the older time-zone behaviour: a value
with `Z` or an offset converts to server local time first (on a +03:00 server `"2024-01-01T10:00:00Z"`
compares as 13:00). Send it in the convention the column stores.
- A nullable member is unwrapped under its guard (`f.Value`, `f.Value.Date`). A non-nullable member on the entity
itself gets no guard, and `IsNull` / `IsNotNull` answer `false` / `true` — `WHERE FALSE` and no predicate on
Npgsql, for `DateTime` and `DateTimeOffset` alike. Reached through a navigation (`Approval.ApprovedAt`), each
navigation is guarded instead (`Approval != null && …`), and `IsNull` / `IsNotNull` test the navigation: a
provider reads the member of a missing approval as NULL, and 3.0.0 answered by it the same way.
- `Having` names an alias, so the type comes from the aggregate it stands for: `Minimum`, `Maximum`,
`FirstOrDefault` and `LastOrDefault` have the member's type (nullable if the member is), and the predicate is
built exactly as for that member. On Npgsql: `HAVING max(col) > TIMESTAMPTZ '…'`. Count, Sumation and Average
aliases are never dates and keep the unconditional guard.
- A `DateOnly` member (3.1.0) is compared as a day under both date data types, against a `DateOnly(y, m, d)`
constructor — never `DateOnly.Parse`, which the runtime evaluates in the host's calendar and reads
`"2026-09-01"` as the year 1483 on a Thai server. On Npgsql: `WHERE "Day" = DATE '2026-09-01'`.
- Before 3.1.0 every comparison on a `DateTimeOffset` member threw — `InvalidOperationException` ("The binary
operator NotEqual is not defined for the types 'System.DateTimeOffset' and 'System.Object'") on a
non-nullable one, `ParseException` on a nullable one — `DataType.Date` on any nullable date member threw
`ParseException` ("No property or field 'Date' exists in type 'DateTime?'"), and no comparison on a `DateOnly`
member worked under either date data type (`IsNull` and `IsNotNull` did).
- A C# `DateTime`, `DateTimeOffset` or `DateOnly` placed in `Values` is written year-first (the normalizer table
above) and then read like any other value; before 3.1.0 it took the month-first invariant form
(`MM/dd/yyyy HH:mm:ss`), which is now refused.
- A `DateTime` of `Kind` `Local` (`DateTime.Now`, or what Newtonsoft.Json makes of a string carrying an offset),
compared under `DataType.DateTime` with a `DateTimeOffset` or `DateTimeOffset?` member — a `Having` alias over
such a member's aggregate included — is written with its UTC offset (`2026-09-17T15:00:00+03:00`), so it
filters on the moment it holds. Without the offset the member would read it as UTC, three hours away on a
host at UTC+3, with no error.
- Every other `DateTime` is written with no zone, as before: under `DataType.Date`, so `DateTime.Today` compares
the day it was written for (with its offset, local midnight on the 17th is the 16th in UTC on a host ahead of
UTC); on a `DateTime` member, which holds wall-clock time, and on a `DateOnly` member; and a `DateTime` of
`Kind` `Utc` or `Unspecified`, which a `DateTimeOffset` member reads as UTC.
- Text values, such as JSON strings bound by System.Text.Json, are never rewritten.
- **Enum:** a name that is not a member passes validation and throws `ParseException` when the query is built.
- In C#, `Values` is `List<object>`: `Values = { "Engineering" }` or `new List<object> { 1, 2 }`. A `List<string>`
is not assignable.
### Date formats — `DwDates` (3.1.0)
```csharp
namespace DynamicWhere.ex.Source;
public sealed class DwDateOptions
{
public IList<string> Formats { get; } // .NET exact formats, read with InvariantCulture; read-only once configured
public bool IsFrozen { get; }
}
public static class DwDates
{
public static DwDateOptions Options { get; } // frozen; declares nothing until configured
public static bool IsConfigured { get; }
public static void Configure(DwDateOptions options);
public static void Configure(Action<DwDateOptions> configure);
public static DwDateOptions Bind(this DwDateOptions options, IConfiguration section); // extension
}
```
```csharp
DwDates.Configure(o => o.Formats.Add("dd/MM/yyyy")); // once, at startup
DwDates.Configure(new DwDateOptions().Bind(configuration.GetSection("DynamicWhere:Dates")));
// appsettings.json: { "DynamicWhere": { "Dates": { "Formats": [ "dd/MM/yyyy", "dd/MM/yyyy HH:mm" ] } } }
```
- Declared formats are accepted **in addition to** ISO 8601 and year-first dates, which every deployment accepts.
- `Configure` freezes the options and may be called once; a second call throws `InvalidOperationException`. Every
query reads the formats without a lock.
- Refused at `Configure` with `ArgumentException`:
- a blank or malformed format (`"'dd/MM/yyyy"`, `"q"`);
- a format that cannot read back the text it writes, or reads a part of it back differently — `dd/MM/yyyy hh:mm`,
a 12-hour clock with no `tt`, reads 4 PM as 4 AM;
- a format with no year — `dd/MM`, `HH:mm`, `t` — which the parser would complete from the clock, so the same
value would name a different date depending on when the query ran;
- a format with a day but no month — `dd/mm/yyyy`, where `mm` is minutes;
- two formats that read one text as different dates — `dd/MM/yyyy` beside `MM/dd/yyyy`, or `yyyy-dd-MM` against
ISO;
- a format whose own text ISO 8601 or a year-first date already reads — `yyyy-MM-dd`, `yyyy/M/d`,
`yyyy-MM-dd HH:mm:ss`, `yyyy-MM-dd'T'HH:mm:ss'Z'` (3.1.0). Declaring one can only change what such a value
means: a quoted `'Z'` is a letter, not a zone, so that format reads `12:00` as a wall time where ISO 8601 reads
an instant. On a `DateTime` member the ISO reading converts to the host's local time, so off UTC the two
readings differed and every such value was refused as `AmbiguousDateFormat` — on that host only. The refusal is
the same on every host. Checked last, after the rules above, which name a sharper reason;
- two formats that lead with the day and the month in opposite orders, even in different shapes —
`dd/MM/yyyy HH:mm` beside `MM/dd/yyyy` would make `"01/09/2026 00:00"` 1 September and `"01/09/2026"`
9 January. Declare one day/month order.
- A format with a year but no day, such as `yyyy-MM`, is accepted and reads the 1st. So are `dd/MM/yyyy`,
`dd/MM/yyyy HH:mm` and `dd MMM yyyy`: ISO 8601 reads none of them.
- `Bind` throws `InvalidOperationException`, at startup instead of leaving the defaults in force, for a key
nothing answers to (`ErrorOnUnknownConfiguration`) such as a misspelt `Fromats`, and for a single value where the
list belongs: `"Formats": "dd/MM/yyyy"`, or one environment variable `DynamicWhere__Dates__Formats`. Write
`"Formats": [ "dd/MM/yyyy" ]`, or `DynamicWhere__Dates__Formats__0`. It also throws on frozen options.
- A value still has to match: with `dd/MM/yyyy` declared, `"09/15/2026"` is `AmbiguousDateFormat`.
- There is no per-condition format. A condition's value is read with the process-wide formats.
```
Value count, checked before the format (Condition and Having alike)
IsNull IsNotNull 0 else ConditionWithOperator[IsNull-IsNotNull]MustHasNoValues
Between NotBetween 2 else ConditionWithOperator[Between-NotBetween]MustHasOnlyTwoValues
In IIn NotIn INotIn 1+ else ConditionWithOperator[In-IIn-NotIn-INotIn]MustHasOneOrMoreValues
any other operator 1 else ConditionWithOperator[<Operator>]MustHasOnlyOneValue e.g. ConditionWithOperator[Equal]MustHasOnlyOneValue
```
Each condition is checked in this order: Field, value count, value format, DataType/Operator pair.
### Groups and connectors
```
And children joined with &&
Or children joined with ||
```
- A group emits its `Conditions` in Sort order, then its `SubConditionGroups` in Sort order, each in parentheses:
`(c1 && c2 && (sub1) && (sub2))`. Conditions always precede sub-groups, whatever their Sort values.
- A group has one connector. `A AND (B OR C)` is an And group holding A plus an Or sub-group holding B and C.
- A group with no condition at any depth emits nothing: as the root it filters nothing; as a sub-group it is skipped.
- Sort must be unique among a group's Conditions (`AnyListOfConditionsMustHasUniqueSortValue`) and, separately,
among its SubConditionGroups (`AnyListOfSubConditionsGroupsMustHasUniqueSortValue`). A condition and a sub-group
may share a Sort. The group's own Sort is never checked.
---
## 5. Field paths
`Condition.Field`, `OrderBy.Field`, `GroupBy.Fields`, `AggregateBy.Field` and `Selects` entries are dotted paths
from T. Examples with T = Customer:
```
Path Where predicate generated
"TotalSpent" (TotalSpent != null && TotalSpent > 100)
"ContactInfo.Email" (ContactInfo.Email != null && ContactInfo.Email == "a@b.c")
"RegisteredAt.Year" (RegisteredAt.Year != null && RegisteredAt.Year == 2024) any public property of a CLR type
"Orders.TotalAmount" (Orders.Any(i1 => i1.TotalAmount != null && i1.TotalAmount > 100))
"Orders.ShippingAddress.Country" (Orders.Any(i1 => i1.ShippingAddress.Country != null && i1.ShippingAddress.Country == "USA"))
"Orders.OrderItems.Quantity" (Orders.Any(i1 => i1.OrderItems.Any(i2 => i2.Quantity != null && i2.Quantity > 2)))
```
- **Matching.** Segments are public instance properties, matched case-insensitively and rewritten to the
declared name (`orders.totalamount` → `Orders.TotalAmount`). Fields and serializer names (`[JsonPropertyName]`,
`[Column]`) are not recognized: camelCase works, snake_case does not. Spaces around segments and empty segments
are removed (`"Orders . TotalAmount"`, `"Orders..TotalAmount"`). A path of only dots fails.
- **Names that look reserved.** A member named `Root`, `It` or `Parent`, in any case, is an ordinary segment, and so
is an alias named `root`, `it` or `parent`. Before 3.1.0 System.Linq.Dynamic.Core read such a name as its context
keyword: `Root.Name` and `It.Name` addressed the row's own `Name`, `Parent` threw `ParseException`, and an
`AggregateBy.Alias` named `root`, `it` or `parent` failed in `Having` and `Summary.Orders`. Under `ApplyPolicy`
the gate decided on the path the caller named while the query read the row's own column, so a denied value could
be projected and a forced scope reached through such a navigation filtered the wrong column.
- Expressions are parsed with the context keywords off, so `it`, `root` and `parent` name members like any other
identifier. The parser's predefined type names name members too: `String`, `Boolean`, `Char`, `Byte`, `SByte`,
`Int16`, `Int32`, `Int64`, `UInt16`, `UInt32`, `UInt64`, `Single`, `Double`, `Decimal`, `DateTime`,
`DateTimeOffset`, `TimeSpan`, `Guid`, `Math`, `Convert`, `Uri`, `Object` and `Enum`.
- **Names the library refuses (3.1.0).** A path whose first segment is one of the parser's own functions or
literals — `new`, `iif`, `np`, `isnull`, `is`, `as`, `cast`, `true`, `false`, `null`, whatever the letter case —
throws `LogicException("FieldPath[<path>]StartsWithReservedName")`, with that first segment, trimmed, on
`LogicException.Subject`.
- Raised where a path is validated, so every clause a caller writes answers alike, guarded or not: condition
fields, `Orders`, `Selects`, `GroupBy.Fields`, `AggregateBy.Field`, and the target of a `[DwAlias]` the caller
names.
- Only the first segment. `Owner.New` names the member, because the parser looks for a member after a dot. An
alias named after one of these words still works: only the path it stands for is checked.
- Under `ApplyPolicy`, the Convenience tier and any dry run give that code. The Strict tier, outside a dry run,
answers with the clause's `FieldDeniedFor*` code and `FieldPath` `"*"`, as it answers for every name it cannot
use (section 17).
- A `DefaultOrder` entry naming one refuses nothing: a guarded query drops it, as it drops an entry it cannot
read, and `ValidateModel` reports it as an error (sections 13 and 20).
- The parser used to answer instead, because it reads its own functions and literals before it looks for a
member: `New`, `Iif`, `Np`, `IsNull`, `Is`, `As` and `Cast` raised its `ParseException`, `True` and `False` an
`InvalidOperationException`, and `Null` was read as the null literal, so the query returned no rows and no
error. A typed `Selects` entry naming such a member did work, because a typed projection is built without the
parser; it is refused now too, so one rule covers every clause.
- The remedy for such a column: rename the CLR property and map the column with `[Column("New")]`.
- **Errors.** A missing segment or a null / blank path throws `LogicException("ConditionMustHasValidFieldName")`,
the same string for Condition, OrderBy, GroupBy, AggregateBy and, since 3.3.0, `Selects` paths — a null or blank
`Selects` entry threw `ArgumentNullException` before. Under `ApplyPolicy` in the Strict tier, outside dry run, a path that matches
nothing is refused like a denied field instead, with a `PolicyException` (3.1.0, section 17).
- **What counts as a collection:** arrays, `List<>`, `IList<>`, `ICollection<>`, `IEnumerable<>`, `HashSet<>`,
`ISet<>`. Anything else (`IReadOnlyList<>`, `IReadOnlyCollection<>`, `Collection<>`, `ObservableCollection<>`, a
class deriving from `List<T>`) is a plain object, so a path continuing past it throws ConditionMustHasValidFieldName.
- **Collections in a where.** The next segment resolves on the element type. Each collection level adds one
`.Any(iN => ...)`, and the whole predicate (null guard and negation included) sits inside the innermost Any.
Negation is per element: NotEqual on `Orders.Status` means "some order has a non-null, different status", not
"no order has it". No `All()` or `!Any()` form exists. Two conditions on the same collection become separate
Any() calls and can be satisfied by different elements.
- **Null navigations.** Only the last member is null-guarded. EF Core translates a null reference navigation to
SQL nulls. In memory (LINQ to Objects) a null navigation earlier in the path throws `NullReferenceException`
in Where, Order and SelectDynamic, so populate navigations in in-memory data.
- **Ordering.** A path through a collection is reduced per collection segment (section 6, Order). Ordering by a
whole reference navigation (`"Category"`) is accepted; EF Core orders by its key, while in memory it throws
`InvalidOperationException("Failed to compare two elements in the array.")`.
- Having fields (aliases) and `Summary.Orders` fields (group fields or aliases) are result columns, not paths.
---
## 6. Extension methods
`public static class Extension`, namespace `DynamicWhere.ex.Source` (the source file is spelled `Extention.cs`;
the class is `Extension`). 28 public methods, all generic with `where T : class`. Every asynchronous one also has
overloads taking a `CancellationToken` (3.2.0).
```
On IQueryable<T> query — composable (validates and builds, executes nothing)
Select<T>(List<string> fields) -> IQueryable<T>
SelectDynamic<T>(List<string> fields) -> IQueryable
Where<T>(Condition condition) -> IQueryable<T>
Where<T>(ConditionGroup group) -> IQueryable<T>
Order<T>(OrderBy order) -> IQueryable<T>
Order<T>(List<OrderBy> orders) -> IQueryable<T>
Page<T>(PageBy page) -> IQueryable<T>
Group<T>(GroupBy groupBy) -> IQueryable
Filter<T>(Filter filter) -> IQueryable<T>
FilterDynamic<T>(Filter filter) -> IQueryable
Summary<T>(Summary summary) -> IQueryable
On IQueryable<T> query — terminal
ToList<T>(Filter filter, bool getQueryString = false) -> FilterResult<T>
ToListAsync<T>(Filter filter, bool getQueryString = false) -> Task<FilterResult<T>>
ToListDynamic<T>(Filter filter, bool getQueryString = false) -> FilterResult<dynamic>
ToListAsyncDynamic<T>(Filter filter, bool getQueryString = false) -> Task<FilterResult<dynamic>>
ToList<T>(Summary summary, bool getQueryString = false) -> SummaryResult
ToListAsync<T>(Summary summary, bool getQueryString = false) -> Task<SummaryResult>
ToListAsync<T>(Segment segment) -> Task<SegmentResult<T>>
On IQueryable<T> query — terminal, cancellable (3.2.0)
ToListAsync<T>(Filter filter, CancellationToken cancellationToken) -> Task<FilterResult<T>>
ToListAsync<T>(Filter filter, bool getQueryString, CancellationToken cancellationToken) -> Task<FilterResult<T>>
ToListAsyncDynamic<T>(Filter filter, CancellationToken cancellationToken) -> Task<FilterResult<dynamic>>
ToListAsyncDynamic<T>(Filter filter, bool getQueryString, CancellationToken cancellationToken) -> Task<FilterResult<dynamic>>
ToListAsync<T>(Summary summary, CancellationToken cancellationToken) -> Task<SummaryResult>
ToListAsync<T>(Summary summary, bool getQueryString, CancellationToken cancellationToken) -> Task<SummaryResult>
ToListAsync<T>(Segment segment, CancellationToken cancellationToken) -> Task<SegmentResult<T>>
On IEnumerable<T> query — terminal, in memory
ToList<T>(Filter filter, bool getQueryString = false) -> FilterResult<T>
ToListDynamic<T>(Filter filter, bool getQueryString = false) -> FilterResult<dynamic>
ToList<T>(Summary summary, bool getQueryString = false) -> SummaryResult
```
These do not exist: a synchronous `ToList<T>(Segment)`, `getQueryString` on Segment, any async or composable
method on `IEnumerable<T>`, names such as `ToListFilter` / `ToListAsyncSegment`, a `new()` constraint, and a
`CancellationToken` parameter on the 3.1 signatures: the token overloads are separate methods, so code compiled
against 3.1 still binds.
### Rules for every method
- The first statement is the `[DwEntity(RequirePolicy = true)]` guard: on such a T, a call outside `ApplyPolicy`
throws `PolicyException` `PolicyRequired` (section 13). Then null arguments throw `ArgumentNullException`, then a
null entry in one of the shape's lists throws `ListOf[<list>]MustNotHasNullEntry` (3.3.0, section 8), then the
validation rules of section 8 throw `LogicException`.
- Composable methods validate and build when called, not when enumerated, so a bad field throws at the call.
- The async terminals are `async` methods: every exception, validation included, surfaces at `await`.
- Validation rewrites the caller's objects in place: path strings get the declared casing (`"price"` → `"Price"`)
in `Condition.Field`, `OrderBy.Field`, `GroupBy.Fields`, `AggregateBy.Field` and `Selects`; null `Values`,
`Conditions`, `SubConditionGroups`, `ConditionSets`, `Fields` and `AggregateBy` become empty lists; the first
`ConditionSet`'s `Intersection` becomes null. Call `Clone()` on the shape before reusing it if that matters (3.3.0).
### Select<T>
```
products.Select(["Id", "Name", "Category.Name", "OrderItems.Quantity"]) builds
e => new Product {
Id = e.Id, Name = e.Name,
Category = e.CategoryId == null ? new Category() : new Category { Id = …, Name = … },
OrderItems = e.OrderItems.AsQueryable().Select(c => new OrderItem { Id = c.Id, Quantity = c.Quantity }).ToList() }
```
- Errors: `fields` empty → `MustHasFields`; a null or blank entry → `ConditionMustHasValidFieldName` (3.3.0; it was
`ArgumentNullException`); an unknown path →
`ConditionMustHasValidFieldName`; no public parameterless constructor on T →
`LogicException("SelectTypeMustHaveParameterlessConstructor")` with `Subject = typeof(T).Name` (checked at
run time; the constraint is only `class`). Before 3.1.0 that message was an English sentence carrying the type
name inside it.
- Rows are whole T objects. An unselected member keeps what T's parameterless constructor gives it: initializers
run (`= string.Empty` stays `""`, `= new List<X>()` stays empty), everything else is default. Serialized, every
member appears.
- A non-dotted scalar (`"Name"`) binds the value; a non-dotted navigation (`"Category"`) binds the whole related
entity; a non-dotted collection (`"OrderItems"`) binds the whole collection.
- `"Category.Name"` builds a new `Category` holding `Name`, plus `Id` when the nested type has an `Id`.
- A null reference navigation comes back as a placeholder `new Category()` with constructor defaults, never null.
The null test reads `CategoryId` when T has a property of that name (a default FK value also gives the
placeholder); otherwise it tests `Category == null`.
- `"OrderItems.Quantity"` projects each element into a `List<OrderItem>` (with `Id` when present). An empty
collection gives an empty list. Deeper paths recurse the same way.
- Skipped silently, keeping the constructor default: members without a public setter; collections whose element
type has no parameterless constructor; collection members whose declared type cannot hold a `List<TElement>`
(`HashSet<>`, `ISet<>`, arrays); a dotted path into a string or struct member (`"CreatedAt.Year"`).
- If `"Category"` and `"Category.Name"` are both listed, the dotted projection wins. Duplicate paths collapse.
- **EF Core only for reference navigations:** a dotted reference-navigation path reads `EF.Property<T>`, so on an
in-memory source it throws `InvalidOperationException("The EF.Property<T> method may only be used within Entity
Framework LINQ queries.")`. Scalar and collection paths work in memory.
### SelectDynamic<T>
- Validation and errors as `Select<T>`, without the constructor requirement; `Id` is never added.
- Emits one Dynamic LINQ `new(...)` selector. Each row is an instance of a runtime-generated class deriving from
`System.Linq.Dynamic.Core.DynamicClass`, with real public properties. Read it as `dynamic` (`row.Category.Name`)
or by reflection.
- Property names are the path segments, nested like the path; nothing is flattened: `"Category.Name"` is
`row.Category.Name`, never `row.CategoryName`.
```
fields emitted selector row
"Id", "Name" new(Id, Name) { Id, Name }
"Category" new(Category) { Category: <whole entity or null> }
"Category.Name" new(new(Category.Name as Name) as Category) { Category: { Name } }
"Category.Name", "Category.Id" new(new(Category.Name as Name, np(Category.Id) as Id) as Category) { Category: { Name, Id } }
"OrderItems.Quantity" new(OrderItems.Select(v0 => new(v0.Quantity as Quantity)) as OrderItems) { OrderItems: [ { Quantity } ] }
"Category.Vendors.Id" new(new(Category.Vendors.Select(v0 => new(v0.Id as Id)) as Vendors) as Category)
```
- Nested collections get lambda parameters `v0`, `v1`, … one per level.
- On EF Core a missing reference navigation still produces the nested object with null members
(`"category": { "name": null }`); a non-nullable value type directly under a reference navigation (outside a
collection) is wrapped in `np()` and comes back nullable. A whole-navigation entry (`"Category"`) is null when
the navigation is null. In memory, a dotted path through a null navigation throws `NullReferenceException`.
- If `"Category"` and `"Category.Name"` are both listed, the whole-object entry is dropped.
### Where<T>
- `Where<T>(Condition)` validates the condition and adds one predicate (section 4).
- `Where<T>(ConditionGroup)` adds the group's predicate; an empty group leaves the query unchanged.
### Order<T>
```
Order(OrderBy) one OrderBy("<path> asc|desc")
Order(List<OrderBy>) items sorted by Sort (ties keep list order) into one OrderBy("p1 asc,p2 desc");
the first item is the primary key, the rest are then-by keys
```
- A null, blank or unknown `Field` throws `ConditionMustHasValidFieldName`. An empty list leaves the query unchanged.
- Each call starts a new ordering and replaces an earlier one; it is not a then-by. Put every key in one list.
- A path with no collection is emitted as written (`Category.Name asc`). A path through a collection reduces each
collection segment to one value — `Min` ascending, `Max` descending — so rows sort by their best element in the
requested direction:
```
field dir emitted
Tags.Value asc Tags.Min(Value) asc
Tags.Value desc Tags.Max(Value) desc
OrderItems.Product.Name asc OrderItems.Min(Product.Name) asc
OrderItems.UnitPrice asc OrderItems.Select(UnitPrice).DefaultIfEmpty().Min() asc
Orders.OrderItems.Quantity desc Orders.Select(OrderItems.Select(Quantity).DefaultIfEmpty().Max()).DefaultIfEmpty().Max() desc
Labels (List<string>) asc Labels.Min() asc
```
- An empty collection sorts as null for a reference or nullable element type, and as the type default for a
non-nullable value type (via `DefaultIfEmpty()`, so in-memory sorting does not throw).
- A path ending on a collection of non-simple elements (`"Tags"`, `"Posts.Tags"`) throws
`OrderField[<field>]CannotEndOnCollectionOfComplexElements`; sort by a member inside it (`"Tags.Value"`).
Collections of simple values are allowed.
- Provider limits apply: SQLite, for one, cannot ORDER BY a `decimal` column (EF Core throws `NotSupportedException`).
- `Summary.Orders` does not use any of this (section 7).
### Page<T>
- Emits `Skip((PageNumber - 1) * PageSize).Take(PageSize)`. `PageNumber <= 0` → `PageNumberMustBeGreaterThanZero`;
`PageSize <= 0` → `PageSizeMustBeGreaterThanZero`.
- The offset is worked out in 64 bits and held to `int.MaxValue` (3.3.0), here and in the three summary methods
(section 7). In 32 bits the product wrapped for a large enough page number: a negative offset is an error on SQL
Server and PostgreSQL, so the request became a five-hundred, and the first page again on SQLite and in memory, so
a page far past the last row returned rows. A page past the last row is an empty page however far past it is, as
it always was for a page number that did not wrap. The policy layer caps `PageSize` (`MaxPageSize`) and never
`PageNumber`, so a guarded query took the same path.
- The core sets no upper bound (the policy layer has `MaxPageSize`). A page past the end is empty. `Page` neither
adds nor requires an ordering; order before paging for stable pages. (The guarded `Page` of section 13 takes the
type's declared `DefaultOrder` on a source nothing has ordered; this one never reads it.)
### Filter<T>, FilterDynamic<T>
```
Filter<T> Where(ConditionGroup) -> Order(Orders) -> Page(Page) -> Select(Selects) -> IQueryable<T>
FilterDynamic<T> Where(ConditionGroup) -> Order(Orders) -> Page(Page) -> SelectDynamic(Selects) -> IQueryable
```
- A step runs only when its member is non-null; `new Filter()` returns the query unchanged.
- `Orders` and `Page` run on T before the projection, so they may name fields that are not selected.
- `Selects = []` throws `MustHasFields`; `Orders = []` means no ordering.
- `FilterDynamic<T>` with `Selects` null returns the `IQueryable<T>` itself, so its rows are T.
### ToList, ToListAsync, ToListDynamic, ToListAsyncDynamic (Filter)
```
ToList, ToListAsync Where -> build Order, Page, Select -> COUNT(where-only query) -> data query
ToListDynamic, ToListAsyncDynamic Where -> COUNT(where-only query) -> build Order, Page, SelectDynamic -> data query
```
- Every call runs two queries: a count of the filtered set (ignoring Page) and the data query.
- In the dynamic pair an invalid `Orders`, `Page` or `Selects` throws after the count has already run.
- `ToListAsync` uses EF Core `CountAsync` / `ToListAsync`. `ToListAsyncDynamic` uses EF Core `CountAsync`, then
EF Core's `ToListAsync` over the query's element type: `T` when `Selects` is null, the projection's generated
class otherwise (3.2.0). It used to read through Dynamic LINQ's `ToDynamicListAsync`, asynchronous as well but
with no token to pass on. Both need an EF Core async provider for the count: on a plain
`list.AsQueryable()` they throw `InvalidOperationException` ("The provider for the source 'IQueryable' doesn't
implement 'IAsyncQueryProvider'…"). Use the synchronous methods in memory.
- The overloads taking a `CancellationToken` (3.2.0) pass it to the count and to the read, so a canceled token stops
whichever is running and the call throws `OperationCanceledException` (EF Core's `TaskCanceledException` derives
from it). The overloads without a token pass `CancellationToken.None`.
- `ToListAsync(filter, default)` does not compile: `default` fits both `bool getQueryString` and
`CancellationToken`. Write `false`, `CancellationToken.None` or a named argument.
- `ToListDynamic` rows are `DynamicClass` objects when `Selects` is set, and T instances when it is null.
### ToListAsync<T>(Segment)
```
validate the sets
combine them in Sort order into one query:
T has a primary key: Where(set1 OR|AND set2 ... AND NOT EXISTS(setN row with the same key))
T has no primary key: set1 UNION|INTERSECT|EXCEPT set2 ... (SQL set operators, whole rows)
then exactly as ToListAsync(Filter): Order(Orders) -> Page(Page) -> Select(Selects); COUNT for TotalCount
```
- The sets combine left to right, `((set1 op2 set2) op3 set3)`, in Sort order, not list order. The database answers
one query: only the requested page is read, plus one COUNT query for `TotalCount`.
- Union and Intersect combine the sets' own conditions with OR and AND. Except removes the rows of its set with
`NOT EXISTS`, matched on T's primary key as EF Core maps it (composite, value-converted and inherited keys
included).
- Rows are matched by key, so a tracking query, `AsNoTracking()` and `Selects` all return the same rows.
- A key the data does not keep unique can make Except remove too much, never return a row no set admitted.
- A type with no primary key (a keyless entity type, or a query EF Core does not map to T) is combined with SQL
`UNION` / `INTERSECT` / `EXCEPT`, which compare whole rows:
- identical rows collapse into one;
- every mapped column must be comparable, even when unselected: not PostgreSQL `json`, SQL Server `xml` or spatial
types;
- the provider must support the operators the request uses (MySQL has `INTERSECT` and `EXCEPT` from 8.0.31).
- `Orders` apply before the projection, as for `Filter`, so an order field need not be selected. Order is the
database's: text sorts by collation and NULLs fall where the provider puts them.
- Under `ApplyPolicy`, `Caps.MaxConditionSets` (default 10) bounds how many sets one statement carries (section 12).
- Validation: a null entry in `ConditionSets`, `Orders` or a condition list →
`ListOf[<list>]MustNotHasNullEntry`, before anything else (3.3.0); duplicate set Sort →
`ListOfConditionsSetsMustHasUniqueSortValue`; a set after the first with a
null `Intersection` → `ConditionsSetOfIndex[1-N]MustHasIntersection`; the first set's `Intersection` is ignored;
a set with a null `ConditionGroup` → `ArgumentNullException`. Every clause is validated before the database is
queried.
- Empty or null `ConditionSets` runs `ToListAsync(new Filter { Selects, Orders, Page })`: there is nothing to combine.
- No synchronous version, no `getQueryString`; needs an EF Core async provider. An overload takes a
`CancellationToken` (3.2.0) and passes it to the count and the read. Only `Except` on a type with a
primary key needs the provider to translate a correlated `EXISTS`; `Union` and `Intersect` there are plain `OR` and
`AND`.
### In memory: the IEnumerable<T> overloads
- `ToList(Filter)`, `ToListDynamic(Filter)` and `ToList(Summary)` on `IEnumerable<T>` call `AsQueryable()` and
run the same pipeline with LINQ to Objects. `ApplyPolicy` takes an `IEnumerable<T>` too, and a summary read
through it is floored like any other: groups smaller than `DwCaps.MinGroupSize`, 5 by default, are dropped. For any other method call `list.AsQueryable()` yourself; the async
methods do not work on such a source.
- Differences from EF Core: text operators follow .NET string semantics; a null reference navigation in a path
throws `NullReferenceException`; typed `Select` through a reference navigation throws (EF.Property);
ordering by a reference navigation throws; `getQueryString` returns the "does not support generation of query
strings" text.
---
## 7. Grouping and aggregation — Group<T>, Summary<T>, ToList(Summary)
```
Summary<T> validate -> Where(ConditionGroup) -> GroupBy + aggregates -> Having -> order -> Skip/Take -> IQueryable
ToList, ToListAsync validate -> Where -> GroupBy + aggregates -> Having -> COUNT(groups) -> order -> Skip/Take -> SummaryResult
Group<T> validate GroupBy -> GroupBy + aggregates -> IQueryable
```
```
GroupBy.Fields emitted row properties
["IsActive"] GroupBy("IsActive").Select("new (Key as IsActive, …)") IsActive
["CreatedAt.Year"] GroupBy("CreatedAt.Year").Select("new (Key as CreatedAtYear, …)") CreatedAtYear
["IsActive", "Category.Name"] GroupBy("new (IsActive, Category.Name)")
.Select("new (Key.IsActive as IsActive, Key.Name as CategoryName, …)")
IsActive, CategoryName
```
- Each row has one property per group field, named by the normalized path with the dots removed, holding the
key's own type (an enum key stays an enum), then one property per `AggregateBy.Alias`, in list order. Rows are
`DynamicClass` objects; `SummaryResult.Data` is `List<dynamic>`.
- A null group key is its own group (`{ "categoryName": null, … }`).
- `ToListAsync(Summary)` counts and reads through EF Core's `CountAsync` and `ToListAsync` (3.2.0; it used to count
synchronously). On a provider that is not EF Core's, rows in memory among them, it counts and reads synchronously.
```
Aggregator Emitted per group Field accepted Result type
Count Count() optional; validated if given, unused int
CountDistinct Select(f).Distinct().Count() any simple type int
Sumation Sum(f) numeric as Sum(f): int -> int, decimal -> decimal
Average Average(f) numeric as Average(f): int -> double, decimal -> decimal
Minimum Min(f) simple, not bool / bool? the field's type
Maximum Max(f) simple, not bool / bool? the field's type
FirstOrDefault Select(f).OrderBy($).FirstOrDefault() any simple type the field's type: the SMALLEST value
LastOrDefault Select(f).OrderByDescending($).FirstOrDefault() any simple type the field's type: the LARGEST value
```
- **numeric** = byte, sbyte, short, ushort, int, uint, long, ulong, float, double, decimal and their nullable forms.
- **simple** = any primitive, string, decimal, DateTime, DateOnly, TimeOnly, DateTimeOffset, TimeSpan, Guid, enum,
and their nullable forms.
- **Alias** must be an identifier: a letter (any script) or `_`, then letters, digits or `_`. Valid: `Total_Sales`,
`Total2`, `المجموع`. Invalid (`AggregationMustHasValidAlias`): `Total Sales`, `Total-Sales`, `Total.Sales`,
`1Total`, `""`. The Alias is checked before the Field, so an entry without an Alias always fails on the Alias.
- **Group and aggregate fields** must end on a simple type. A navigation, or a collection of entities, throws
`GroupByFieldCannotBeComplexType` / `AggregationFieldMustBeSimpleType` (the element type is what is checked, so the
`…CannotBeCollectionType` strings fire only for a collection of collections). A path through a collection to a
scalar, or a collection of simple values, passes validation but gets no Any() / Select, so do not group or
aggregate across a collection.
- **Key names clash silently.** Inside a multi-field key, members are named by their last segment: two group fields
ending in the same segment (`"Name"`, `"Category.Name"`) pass validation and throw
`InvalidOperationException("Sequence contains more than one matching element")` at run time. An Alias equal to a
dot-stripped group field (`CategoryName` beside `Category.Name`) is not refused either — the alias check compares
against the dotted path — and one of the two columns silently disappears from the rows.
- **Having.** `Summary.Having` is a `ConditionGroup` over the grouped rows. Each condition's Field must be an
Alias (case-insensitive), else `HavingField[<field>]MustExistInAggregateByAliases`; a group field is not allowed.
Value count, value format, Sort uniqueness, the null guard and the DataType/Operator table work as in a where.
With no `AggregateBy`, every Having condition fails.
- **Summary.Orders.** Each Field must be a group field — dotted (`Category.Name`) or with the dots removed
(`CategoryName`) — or an Alias, case-insensitive; else
`SummaryOrderField[<field>]MustExistInGroupByFieldsOrAggregateByAliases`. Emitted as `<field without dots> asc|desc`,
sorted by Sort, with no collection rewriting and no duplicate-Sort check.
- Summary validation order: a null entry in any list (3.3.0) → `GroupBy` null (`ArgumentNullException`, parameter
`GroupBy`) → GroupBy and AggregateBy rules → Orders → Page → Having → ConditionGroup → Having DataType/Operator
pairs.
- Aggregation runs in SQL on stored values. Under `ApplyPolicy`, transformed fields need `AllowAggregate`, every
guarded summary is subject to the group floor (section 19), and `Caps.MaxAggregates` (default 50, 3.1.0) bounds how
many `AggregateBy` entries one summary sends (section 12).
---
## 8. Validation and core error strings
A broken rule throws `LogicException` (`DynamicWhere.ex.Exceptions`). The error string is `Message`; there is no
separate code property. Two constructors: `LogicException(string message)` and, since 3.1.0,
`LogicException(string message, string? subject)`, whose `Subject` carries what the refusal is about — a type
name, a field — so the message stays one of the fixed strings a caller matches on. Under `ApplyPolicy` a field is
named as the caller wrote it, so a `[DwAlias]` name is never replaced by the member behind it. `PolicyException`
derives from it, so catch `PolicyException` first.
### Every core error string
```
String Raised when
ConditionMustHasValidFieldName a Condition / OrderBy / GroupBy / AggregateBy / Having / Summary.Orders
field is null or blank, or a path does not resolve on T; also a
Selects entry that is null or blank (3.3.0; it was an
ArgumentNullException). Under ApplyPolicy in the Strict tier,
outside dry run, an unresolved path is a PolicyException
instead (3.1.0, section 17)
ListOf[<list>]MustNotHasNullEntry a list of the request shape holds a null entry (3.3.0). <list> is
Conditions, SubConditionGroups, ConditionSets, Orders or
AggregateBy, spelled as the shape declares it. Before 3.3.0 a
null entry was a NullReferenceException from wherever it was
first touched
FieldPath[<path>]StartsWithReservedName a Condition / OrderBy / GroupBy / AggregateBy / Selects path, or a
[DwAlias] target, whose first segment is one of the parser's own
words — new, iif, np, isnull, is, as, cast, true, false, null,
whatever the letter case. Subject = that segment, trimmed. Not a
member of ErrorCode: the string is built where the path is
validated. 3.1.0, section 5
ConditionWithOperator[<Operator>]MustHasOnlyOneValue an operator other than those below has 0 or 2+ values
ConditionWithOperator[Between-NotBetween]MustHasOnlyTwoValues Between / NotBetween without exactly 2 values
ConditionWithOperator[In-IIn-NotIn-INotIn]MustHasOneOrMoreValues an In-family operator with no values
ConditionWithOperator[IsNull-IsNotNull]MustHasNoValues IsNull / IsNotNull with any value
InvalidFormat a Guid / Boolean / Date / DateTime value fails its check, or a
Number value is not a literal the expression parser reads, or is
one it cannot compare with the member (3.3.0, section 4)
AnyListOfConditionsMustHasUniqueSortValue two Conditions of one group (or Having group) share Sort
AnyListOfSubConditionsGroupsMustHasUniqueSortValue two SubConditionGroups of one group share Sort
ListOfConditionsSetsMustHasUniqueSortValue two ConditionSets share Sort
ConditionsSetOfIndex[1-N]MustHasIntersection a ConditionSet after the first (by Sort) has Intersection null
OrderField[<field>]CannotEndOnCollectionOfComplexElements an order path ends on a collection of entities
PageNumberMustBeGreaterThanZero PageNumber <= 0
PageSizeMustBeGreaterThanZero PageSize <= 0
MustHasFields Select / SelectDynamic / Selects given an empty list
GroupByMustHasAtLeastOneField GroupBy.Fields empty
GroupByFieldsMustBeUnique a group field repeated (case-insensitive)
GroupByFieldCannotBeComplexType a group field ends on a navigation or entity collection
GroupByFieldCannotBeCollectionType a group field ends on a collection of collections
AggregationMustHasValidAlias Alias is not an identifier
AggregationFieldMustBeSimpleType an aggregate field ends on a navigation or entity collection
AggregationFieldCannotBeCollectionType an aggregate field ends on a collection of collections
Aggregator[<Aggregator>]IsNotSupportedForFieldType[<TypeName>] Sumation / Average on a non-numeric type, Minimum / Maximum on bool;
<TypeName> is Type.Name, so a bool? member shows Nullable`1
AggregationAliasesMustBeUnique an Alias repeated (case-insensitive)
AggregationAlias[<alias>]CannotBeUsedInGroupByFields an Alias equals a group field path (case-insensitive)
SummaryOrderField[<field>]MustExistInGroupByFieldsOrAggregateByAliases a Summary order field is neither a group field nor an Alias
HavingField[<field>]MustExistInAggregateByAliases a Having field is not an Alias
ConditionValuesAreNullOrWhiteSpace defined but never thrown
AmbiguousDateFormat a Date / DateTime value that leads with a day or a month
("01/09/2026") and matches no declared format, or one two
accepted formats read differently (section 4, Date formats).
Subject = the field. 3.1.0
SelectTypeMustHaveParameterlessConstructor Select<T> / Filter.Selects on a T with no parameterless
constructor — a positional record, most often. Also a
guarded query where a member carries [DwNoSelect],
because deny-select projects. Subject = T.Name.
3.1.0; before it, an English sentence
Unsupported combination of DataType '<DataType>' and Operator '<Operator>'. a pair outside the section 4 table
```
The last one is a literal message; every other string above is a fixed code. Examples: `ConditionWithOperator[Equal]MustHasOnlyOneValue`,
`Aggregator[Sumation]IsNotSupportedForFieldType[String]`, `HavingField[UnitPrice]MustExistInAggregateByAliases`.
### Other exceptions a caller can see
```
ArgumentNullException a null query or shape argument; Summary.GroupBy null (parameter "GroupBy"); a ConditionSet
whose ConditionGroup is null. A null or blank Selects entry was one until 3.3.0; it is
ConditionMustHasValidFieldName now
NullReferenceException in memory, a null reference navigation inside a path. A null element inside Conditions,
SubConditionGroups, ConditionSets, Orders or AggregateBy was one until 3.3.0; it is
ListOf[<list>]MustNotHasNullEntry now
PolicyException PolicyRequired from the [DwEntity(RequirePolicy = true)] guard; every refusal under ApplyPolicy
ParseException System.Linq.Dynamic.Core.Exceptions.ParseException for input that passes validation but not
the parser: an enum name that is not a member, Contains on an enum-typed member, a Where
on a collection of simple values. No Number value reaches it since 3.3.0 (section 4)
InvalidOperationException async terminal on a non-EF source; typed Select through a reference navigation in memory;
two group fields ending in the same segment; ordering by a non-comparable type in memory
EF Core / provider exceptions unwrapped (for example SQLite NotSupportedException for ORDER BY on decimal)
```
- The engine catches nothing; everything reaches the caller unwrapped.
- Timing: composable methods throw at the call; async terminals at `await`; the dynamic terminals run the COUNT
query before validating Orders, Page and Selects. A Segment validates every set and clause before its first query
(3.1.0).
### Checks by shape, in order
```
Condition Field null/blank → Field resolves → value count → value format → DataType/Operator pair
ConditionGroup Sort unique among Conditions → Sort unique among SubConditionGroups → each Condition in Sort order →
each sub-group in Sort order (recursively)
OrderBy Field null/blank → Field resolves → not ending on a collection of entities
PageBy PageNumber >= 1 → PageSize >= 1
Selects list not null (ArgumentNullException) → not empty → no null or blank entry → each resolves →
constructor (Select<T>)
GroupBy Fields not empty → each field: null/blank, resolves, unique, not complex → each AggregateBy:
Alias identifier → Field (unless Count) → simple type → aggregator valid for type → Alias not a group
field → Alias unique
Summary GroupBy not null → GroupBy rules → Orders → Page → Having (count, format) → ConditionGroup → Having pairs
Segment set Sort unique → Intersection on later sets → each set's ConditionGroup → Orders → Page → Selects
Filter ConditionGroup → Orders → Page → Selects, each only when not null
```
- **A null entry in a list is read before any of this (3.3.0).** Every method that takes a shape walks its lists for a
null entry first: the composables `Where(ConditionGroup)`, `Order(List<OrderBy>)`, `Select`, `SelectDynamic`,
`Group`, `Summary`, and every terminal for a Filter, a Segment and a Summary, sync and async. `Filter` and
`FilterDynamic` compose `Where`, `Order` and `Select`, so each list is walked as that clause is reached. Under
`ApplyPolicy` the walk runs at the top of the sanitizer, before the caps and before the gate, because it is about
the request's shape and not about a policy decision — so a guarded Filter with a null `Orders` entry is refused
before its condition group is validated, where an unguarded one reaches the condition group first. Refusals:
`ListOf[Conditions]MustNotHasNullEntry`, `ListOf[SubConditionGroups]MustNotHasNullEntry`,
`ListOf[ConditionSets]MustNotHasNullEntry`, `ListOf[Orders]MustNotHasNullEntry`,
`ListOf[AggregateBy]MustNotHasNullEntry`, and `ConditionMustHasValidFieldName` for a `Selects` entry that is null
or blank. A list that is itself null still means what it meant — most readers read it as empty.
---
## 9. JSON wire format
The shapes and results carry no JSON attributes or converters, and the query path serializes nothing: the host's
serializer binds the request and writes the result.
```
JSON body Bind to Pass to
["Id", "Category.Name"] List<string> Select, SelectDynamic
{ sort, field, dataType, operator, values } Condition Where
{ sort, connector, conditions, subConditionGroups } ConditionGroup Where
{ sort, field, direction } / [ … ] OrderBy / List<OrderBy> Order
{ pageNumber, pageSize } PageBy Page
{ fields, aggregateBy: [ { field, alias, aggregator } ] } GroupBy Group
{ conditionGroup, selects, orders, page } Filter Filter, FilterDynamic, ToList, ToListAsync,
ToListDynamic, ToListAsyncDynamic
{ conditionGroup, groupBy, having, orders, page } Summary Summary, ToList, ToListAsync
{ conditionSets: [ { sort, intersection, conditionGroup } ],
selects, orders, page } Segment ToListAsync
```
### Property names
- Keys are the C# property names. ASP.NET Core's defaults (`JsonSerializerDefaults.Web`) bind them
case-insensitively, so `conditionGroup` and `ConditionGroup` both work, and write camelCase.
- Path strings (`field`, `fields`, `selects`) are case-insensitive per segment and trimmed; they are rewritten to
the exact CLR names, and results use the rewritten names.
- `alias` becomes an output member name exactly as sent.
### Enums need a converter for names
The library ships no enum converter. Without one, `System.Text.Json` refuses enum names with `JsonException`
("The JSON value could not be converted to DynamicWhere.ex.Enums.Connector") — a 400 in ASP.NET Core. Either send
the numbers of section 3, or register `JsonStringEnumConverter`, which then accepts names in any case
(`"IContains"`, `"icontains"`) as well as numbers:
```csharp
// MVC controllers
builder.Services.AddControllers().AddJsonOptions(o =>
o.JsonSerializerOptions.Converters.Add(new JsonStringEnumConverter()));
// minimal APIs
builder.Services.Configure<Microsoft.AspNetCore.Http.Json.JsonOptions>(o =>
o.SerializerOptions.Converters.Add(new JsonStringEnumConverter()));
```
The same setting decides how results write enums: `policy.tier`, `decisions[].feature`, `decisions[].action`,
and enum-typed members and group keys in `data`.
### Omitted and empty values
- An omitted enum property silently takes member 0: `dataType` Text, `operator` Equal, `connector` And,
`direction` Ascending, `aggregator` Count. An omitted `sort` is 0.
- Omitted or null `values` means `[]`. An `aggregateBy` entry may omit `field` only for Count.
- In Filter, Summary and Segment an omitted or null `conditionGroup`, `selects`, `orders`, `page` or `having` is
skipped; an omitted `intersection` is null.
- `"selects": []` throws `MustHasFields` — omit the key instead. `"fields": []` throws `GroupByMustHasAtLeastOneField`.
- `"orders": []` and a group with no conditions change nothing. `"conditionSets": []` runs the segment as a plain filter.
Under `ApplyPolicy` a Filter or Segment with no orders takes the type's declared `DefaultOrder`, if any
(3.1.0, section 13); unguarded, no orders means no ordering.
- Inside a Segment, a set whose group has no conditions stands for every row: a `Union` with it returns every row, an
`Intersect` with it changes nothing, and an `Except` of it returns nothing.
- A Summary with no `groupBy` throws `ArgumentNullException`, not `LogicException`.
- A `null` inside `values` is `""`, not SQL NULL (section 4).
### Results as JSON
```jsonc
{
"pageNumber": 1, // 0 when the request had no page, unless DwCaps.DefaultPageSize gave a guarded query one
"pageSize": 10, // 0 when the request had no page, with the same exception
"pageCount": 5, // no page: 1, or 0 with no rows
"totalCount": 42, // before paging
"data": [ ],
"queryString": null, // SQL only with getQueryString: true
"policy": null // ApplyPolicy terminals only; under the Strict tier only with IncludeTraceInResult (3.1.0):
// { "tier": "Convenience", "dryRun": false,
// "decisions": [ { "fieldPath": "Email", "feature": "Select", "action": "Masked", "reason": "Mask" } ] }
}
```
```
data rows by method
ToList, ToListAsync (Filter) T with every member. With selects, unselected members hold defaults (0, "", null,
ToListAsync (Segment) "0001-01-01T00:00:00"); a selected reference navigation also carries its Id and is an
empty object, not null, when missing
ToListDynamic, ToListAsyncDynamic no selects: the entity. With selects: only what was asked, nested by path —
"Category.Name" -> { "category": { "name": … } }, "OrderItems.Quantity" ->
{ "orderItems": [ { "quantity": … } ] }; no Id added
ToList, ToListAsync (Summary) one flat row per group: each group field with dots removed ("Category.Name" ->
categoryName), then each alias
```
- Typed rows and `DynamicClass` rows (dynamic filters, unguarded summaries) are objects with properties, so the
host naming policy applies: camelCase under ASP.NET Core defaults (`CategoryName` → `categoryName`).
- Under `ApplyPolicy`, a dynamic or summary row is rebuilt as an `ExpandoObject` when it carries a `[DwAlias]`
column or the group floor applied (every guarded `ToList(Summary)` with the default floor). System.Text.Json writes
`ExpandoObject` keys as they are, so those rows keep PascalCase and alias spelling (`{ "Name": "Ann", "dept": "Eng" }`)
while the envelope is camelCase.
---
## 10. JSON recipes
Field names follow this model:
```
Product Id:Guid Name:string Price:decimal Rating:double StockQuantity:int IsActive:bool
CreatedAt:DateTime UpdatedAt:DateTime? Tags:List<string> CategoryId:Guid?
Category:Category? OrderItems:ICollection<OrderItem> Reviews:ICollection<Review>
Category Id:Guid Name:string ParentCategory:Category?
OrderItem Id:Guid Quantity:int ProductId:Guid Product:Product
Review Id:Guid Rating:int
Order Id:Guid Status:OrderStatus TotalAmount:decimal CustomerId:Guid OrderItems:ICollection<OrderItem>
Customer Id:Guid Orders:ICollection<Order>
OrderStatus Pending Confirmed Processing Shipped Delivered Cancelled Refunded
```
### Projection — Select, SelectDynamic
```json
["Id", "Name", "Category.Name", "OrderItems.Quantity"]
```
```
Select<Product> Product { Id, Name, Category { Id, Name }, OrderItems [ { Id, Quantity } ] }, every other member default
SelectDynamic<Product> { Id, Name, Category: { Name }, OrderItems: [ { Quantity } ] }
```
A bare path list is the body of `Select` / `SelectDynamic`; inside a Filter or Segment the same list goes in `selects`.
### Where — one Condition per DataType
```jsonc
{ "sort": 1, "field": "Name", "dataType": "Text", "operator": "IContains", "values": ["pro"] }
{ "sort": 1, "field": "Price", "dataType": "Number", "operator": "Between", "values": [10, 500.5] }
{ "sort": 1, "field": "IsActive", "dataType": "Boolean", "operator": "Equal", "values": [true] }
{ "sort": 1, "field": "CreatedAt", "dataType": "Date", "operator": "GreaterThanOrEqual", "values": ["2024-01-01"] }
{ "sort": 1, "field": "CreatedAt", "dataType": "DateTime", "operator": "LessThan", "values": ["2024-06-15T14:30:00"] }
{ "sort": 1, "field": "UpdatedAt", "dataType": "DateTime", "operator": "IsNull", "values": [] }
{ "sort": 1, "field": "CategoryId", "dataType": "Guid", "operator": "In", "values": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"] }
{ "sort": 1, "field": "Status", "dataType": "Enum", "operator": "In", "values": ["Pending", "Shipped"] } // on Order
```
Each line is a separate `Condition` body. The Between line becomes `(Price != null && Price >= 10 && Price <= 500.5)`.
### Where — nested AND / OR
```json
{
"connector": "And",
"conditions": [
{ "sort": 1, "field": "IsActive", "dataType": "Boolean", "operator": "Equal", "values": [true] }
],
"subConditionGroups": [
{
"sort": 1,
"connector": "Or",
"conditions": [
{ "sort": 1, "field": "Price", "dataType": "Number", "operator": "LessThan", "values": [20] },
{ "sort": 2, "field": "Rating", "dataType": "Number", "operator": "GreaterThanOrEqual", "values": [4.5] }
]
}
]
}
```
`IsActive AND (Price < 20 OR Rating >= 4.5)`.
### Where — a path through collections (on Customer)
```json
{ "sort": 1, "field": "Orders.OrderItems.Product.Name", "dataType": "Text", "operator": "IContains", "values": ["laptop"] }
```
`(Orders.Any(i1 => i1.OrderItems.Any(i2 => i2.Product.Name != null && i2.Product.Name.ToLower().Contains("laptop"))))`
— a customer matches when some order has some item whose product name contains "laptop".
### Order — several keys, across collections
```json
[
{ "sort": 1, "field": "Category.Name", "direction": "Ascending" },
{ "sort": 2, "field": "Reviews.Rating", "direction": "Descending" },
{ "sort": 3, "field": "OrderItems.Product.Name" }
]
```
`Category.Name asc, Reviews.Select(Rating).DefaultIfEmpty().Max() desc, OrderItems.Min(Product.Name) asc`.
### Page
```json
{ "pageNumber": 3, "pageSize": 25 }
```
Skip 50, take 25.
### Filter — typed and dynamic
```json
{
"conditionGroup": {
"connector": "And",
"conditions": [
{ "sort": 1, "field": "Price", "dataType": "Number", "operator": "GreaterThan", "values": [50] },
{ "sort": 2, "field": "Category.Name", "dataType": "Text", "operator": "IEqual", "values": ["smartphones"] }
]
},
"selects": ["Id", "Name", "Price", "Category.Name"],
"orders": [{ "sort": 1, "field": "Price", "direction": "Descending" }],
"page": { "pageNumber": 1, "pageSize": 10 }
}
```
```
ToListAsync FilterResult<Product>: data[i] = Product { Id, Name, Price, Category { Id, Name } }, everything else default
ToListAsyncDynamic FilterResult<dynamic>: data[i] = { Id, Name, Price, Category: { Name } }
```
### Summary — group, aggregate, having, order by alias
```json
{
"conditionGroup": {
"connector": "And",
"conditions": [{ "sort": 1, "field": "IsActive", "dataType": "Boolean", "operator": "Equal", "values": [true] }]
},
"groupBy": {
"fields": ["Category.Name"],
"aggregateBy": [
{ "alias": "ProductCount", "aggregator": "Count" },
{ "field": "Price", "alias": "AvgPrice", "aggregator": "Average" },
{ "field": "Price", "alias": "Revenue", "aggregator": "Sumation" }
]
},
"having": {
"connector": "And",
"conditions": [{ "sort": 1, "field": "ProductCount", "dataType": "Number", "operator": "GreaterThan", "values": [5] }]
},
"orders": [{ "sort": 1, "field": "Revenue", "direction": "Descending" }],
"page": { "pageNumber": 1, "pageSize": 10 }
}
```
`SummaryResult`: `data[i] = { CategoryName, ProductCount, AvgPrice, Revenue }`; `totalCount` = groups left after
having. `Group<T>` takes the same `groupBy` object and returns the rows without having, order or page.
### Segment — Union, Intersect, Except
```json
{
"conditionSets": [
{ "sort": 1, "conditionGroup": { "conditions": [
{ "sort": 1, "field": "IsActive", "dataType": "Boolean", "operator": "Equal", "values": [true] }] } },
{ "sort": 2, "intersection": "Union", "conditionGroup": { "conditions": [
{ "sort": 1, "field": "Price", "dataType": "Number", "operator": "GreaterThan", "values": [100] }] } },
{ "sort": 3, "intersection": "Except", "conditionGroup": { "conditions": [
{ "sort": 1, "field": "StockQuantity", "dataType": "Number", "operator": "Equal", "values": [0] }] } }
],
"orders": [{ "sort": 1, "field": "Price", "direction": "Descending" }],
"page": { "pageNumber": 1, "pageSize": 20 }
}
```
`SegmentResult<Product>`: (active UNION price > 100) EXCEPT out-of-stock, combined into one query and ordered and
paged in the database. `selects` can be added; it projects the page, as for a filter (section 6).
---
## 11. Traps — query engine
Each of these compiles, passes validation, and returns something other than what was meant.
1. **A `Segment` over a type with no primary key compares whole rows.** A keyed entity combines by row. A keyless
entity type, or a query EF Core does not map to T, uses SQL `UNION` / `INTERSECT` / `EXCEPT`: identical rows
collapse into one, and a column the database cannot compare (PostgreSQL `json`, SQL Server `xml`) fails the
query even when it is not selected. Give the type a key, or map it without such columns.
2. **Typed `Select` still returns whole T objects.** Unselected members hold defaults and a serializer writes all of
them (`"price": 0`, `"createdAt": "0001-01-01T00:00:00"`, `"category": {}`). Use `ToListDynamic` for a payload
holding only the selected members.
3. **Negated operators never match null.** `NotEqual`, `NotIn`, `NotContains`, `NotStartsWith`, `NotEndsWith` and
`NotBetween` all exclude rows whose member is null. Add an `IsNull` condition in an `Or` group to keep them.
4. **Validation rewrites the request objects you pass.** Paths are re-cased, null lists become empty, the first
set's `Intersection` is cleared. Call `Clone()` on the shape before reusing it (3.3.0). (Guarded calls work on a clone.)
5. **Enum names in JSON need `JsonStringEnumConverter`.** Without it the body fails to bind. An omitted enum
property silently becomes member 0: `Text`, `Equal`, `And`, `Ascending`, `Count`.
6. **A `DateTime` member reads a zoned value as server local time.** A value carrying `Z` or an offset is converted
to the host's local time before comparing against a `DateTime` column; a `DateTimeOffset` column normalises to
UTC instead. Send ISO 8601 in the convention the column stores. (Before 3.1.0 this trap was worse: dates were
parsed in the server's culture, so `"01/02/2024"` meant different days on different servers. A day/month-first
date is now refused with `AmbiguousDateFormat` unless the deployment declares its order.)
7. **`FirstOrDefault` and `LastOrDefault` return the minimum and maximum value**, not the first and last row.
8. **Each `Order(...)` call replaces the previous ordering.** Put every key in one `List<OrderBy>`.
9. **Some requests pass validation and then fail in the parser** with `System.Linq.Dynamic.Core.Exceptions.ParseException`:
`Contains` / `StartsWith` / `EndsWith` on an enum-typed member, a `Where`
directly on a `List<string>`, an enum name that is not a member.
Treat `ParseException` as a 400 too. Number values no longer reach it (3.3.0, section 4): one the parser cannot read,
or cannot compare with the member, is `InvalidFormat` at validation.
10. **Summary column names collide silently.** Two group fields ending in the same segment (`Name`, `Category.Name`)
throw `InvalidOperationException` at run time, and an alias equal to a dot-stripped group field (`CategoryName`
beside `Category.Name`) silently drops a column.
11. **In-memory sources behave differently from EF Core.** The async Filter and Segment terminals throw; typed `Select` through a
reference navigation throws; a null navigation inside a path throws `NullReferenceException`; ordering by a
navigation throws; `getQueryString` returns a placeholder sentence instead of SQL.
12. **`ToListAsync(filter, default)` is ambiguous (3.2.0).** `default` fits both `getQueryString` and the new
`CancellationToken` overload, and the call does not compile. Write `false`, a token, or a named argument.
13. **`Selects: []` throws `MustHasFields`.** Omit the property (null) to return whole entities.
14. **A null inside `Values` is the empty string, not NULL.** Test for NULL with `IsNull` / `IsNotNull` and no values.
15. **`Page` does not order.** Paging without `Orders` returns whatever order the database chooses; always send an
order with a page. `[DwEntity(DefaultOrder = ...)]` changes that only for a guarded query (3.1.0, section 13).
16. **Validation is not one pass before the query.** Each clause is checked as it is composed, so some invalid input is
refused only after the database has been hit: `ToListDynamic` / `ToListAsyncDynamic` run the `COUNT` query before
`Orders`, `Page` and `Selects` are validated. The `LogicException` is the same one either way; the round trip is
not undone.
---
## 12. Policy lifecycle and configuration
Sections 12–26 are all in the core package `DynamicWhere.ex`; sections 27–29 are the companion packages.
A project with no policy attributes and no `ApplyPolicy` call behaves exactly as 2.x.
### Lifecycle
```csharp
using DynamicWhere.ex.Policies.Config; // DwPolicy, DwPolicyOptions, AddDwPolicies
using DynamicWhere.ex.Policies.Context; // DwPolicyContext
using DynamicWhere.ex.Policies.Enums; // DwTier, DwSubjectKind
using DynamicWhere.ex.Policies.Source; // ApplyPolicy
using DynamicWhere.ex.Policies.Tokens; // InMemoryTokenVault
// 1. Startup, exactly once. Both forms end in DwPolicy.Configure, which freezes the options.
builder.Services.AddDwPolicies(
builder.Configuration.GetSection("DynamicWhere:Policies"),
options =>
{
options.Entities.Expose<Employee>("Employee");
options.TokenVault = new InMemoryTokenVault();
});
// or, without configuration:
DwPolicy.Configure(new DwPolicyOptions { Tier = DwTier.Strict, HashSalt = secret }, providers);
// 2. Once per request: build the whole caller, prepare, then query.
DwPolicyContext caller = await DwPolicy.PrepareAsync(
new DwPolicyContext { Purpose = "support" }
.WithSubject(DwSubjectKind.User, userId)
.WithSubject(DwSubjectKind.Role, "Support")
.WithSubject(DwSubjectKind.Tenant, tenantId)
.WithValue("TenantId", tenantId));
// 3. Per query.
FilterResult<Employee> result = await db.Employees.ApplyPolicy(caller).ToListAsync(filter);
// 4. Once per request, after the queries, when any field carries [DwAudit] or AuditRefusals is on.
await DwPolicy.DrainAuditAsync(caller, sink);
```
What one guarded call does, in order:
```
Strict getQueryString check → clone the request → count caps → resolve names and aliases → MaxNavigationDepth
→ cost budget (Convenience, and any dry run) → gate each field → default order when no orders were sent
(Filter, Segment) → cost budget (Strict outside dry run, 3.1.0) → inject forced predicates
→ check required filters → group floor (Summary) → unchanged core method over AsNoTracking()
→ transform materialized values → rename aliased columns (dynamic and Summary rows)
→ result.Policy = trace when IncludeTraceInResult allows it (by default: Convenience yes, Strict no)
a refusal at any step: PolicyException, also written to the audit buffer when AuditRefusals is on
```
- Nothing is mandatory.
- Before `Configure`, `ApplyPolicy` still enforces attributes. It uses a frozen default `DwPolicyOptions`: `Convenience` tier, default caps, `MinGroupSize` 5, no store.
- Unguarded calls on `DynamicWhere.ex.Source.Extension` are never sanitized, capped, transformed, traced or given a `DefaultOrder`.
- The only policy check they run is `[DwEntity(RequirePolicy = true)]`.
- The first failing check throws.
- The count caps (`CapExceeded`) run before any name is resolved (3.1.0), so they win over everything after them, a name that matches nothing or is ambiguous included. `MaxNavigationDepth` runs once names are resolved.
- In the Convenience tier and in dry run the cost budget (`QueryCostExceeded`) wins over field denials. Under `Strict`, outside dry run, the budget is checked after every field gate (3.1.0), so a field denial wins over it.
- A field path that names nothing on `T` throws `LogicException` from validation, before any policy decision: unguarded, in the Convenience tier, and in dry run. Under `ApplyPolicy` the count caps run first.
- In the Strict tier (3.1.0), outside dry run, it is kept and gated as a field denied for every feature, after the caps, so it is refused exactly as a denied field is (section 17).
- The `Filter`, `Summary` or `Segment` passed in is never modified. Each guarded call works on a clone.
- `PrepareAsync` reads nothing unless a `StorePolicyProvider` is configured, but it always records that it ran:
`DwPolicyContext.IsPrepared` (3.1.0).
- Since 3.1.0 `ApplyPolicy(ctx)` — the overloads that read `DwPolicy` — refuse a context that never went through
`PrepareAsync` with `PolicyContextNotPrepared` (18), store or no store. Before 3.1.0 only a store provider did,
so an attributes-only deployment accepted an unprepared context and would start refusing the day it gained one.
- The overload taking explicit `DwPolicyOptions` and `PolicyResolver` does not check: that host owns preparation,
and a store it hands in still refuses an unprepared context itself.
- A store provider's own two refusals sit behind that check: no attachment, and a `User` subject added after
preparation.
- The simulator's copy of a prepared context is prepared too.
### DwPolicy
```
DwPolicy static class
Options : DwPolicyOptions frozen; a frozen default instance until Configure
IsConfigured : bool
Resolver : PolicyResolver AttributePolicyProvider + configured providers
StoreProviders : IReadOnlyList<StorePolicyProvider> configured store providers, in the order supplied
Configure(DwPolicyOptions options, params IDwPolicyProvider[] providers) -> void
PrepareAsync(DwPolicyContext context, CancellationToken ct = default) -> ValueTask<DwPolicyContext>
DrainAuditAsync(DwPolicyContext context, IDwAuditSink sink, CancellationToken ct = default) -> ValueTask<int>
ValidateModel(params Type[] types) -> PolicyModelReport
ValidateModel(DwPolicyOptions? options, params Type[] types) -> PolicyModelReport
```
`DwPolicy` has no `Reset`, `Explain` or `IsPrepared`. For isolation (tests, custom hosts), use the four-argument `ApplyPolicy`.
- `Configure` refuses only two things:
- null `options` → `ArgumentNullException`;
- a second call, including one made by `AddDwPolicies`, asking for a **different** posture → `InvalidOperationException`.
- A second call asking for the posture already in force is a no-op (3.3.0). It returns, `DwPolicy.Options` keeps
pointing at the first call's instance, and the options passed to the second call are frozen. The check runs inside
the lock that configures, so a caller needs no `IsConfigured` check of its own — that check is a check-then-act two
hosts starting at once can both pass. This is what lets an integration suite start several
`WebApplicationFactory<Program>` hosts over one composition root.
- Compared: `Tier`, `DryRun`, `IncludeTraceInResult`, `AuditRefusals`, `HashSalt`, `StoreFailure`,
`MaxSnapshotAge`, `RefreshInterval`, every `DwCaps` value, the exposed catalogue — the same types under the same
names, every name each type answers to, and the name each type is reported under — and the provider **types**,
in the order supplied, with `AttributePolicyProvider` left out of both sides. `IsMinGroupSizeSet` is not
compared: nothing in enforcement reads it, so writing `MinGroupSize = 5` and leaving it unset are the same
posture. `IncludeTraceInResult` is compared by the value that applies, not by whether it was written down: it
defaults to the tier's own answer, so writing that answer out is the same posture. A type exposed under two
names is reported under the last one, so two catalogues resolving every name alike are still refused when the
order differs.
- Not compared, and not replaced: `TokenVault`, `Services`, and the provider **instances**. A second host builds
its own and no two are ever the same reference, so they stay as the first call left them — a second host runs
with the first host's vault, container and rule stores.
- `Configure` freezes `options`, `options.Caps` and `options.Entities`. Any setter or `Expose` after that → `InvalidOperationException`.
- `Configure` does not validate the model.
- Invalid values were already refused by the setters.
- A missing `HashSalt` or `TokenVault` only surfaces at query time (`MissingHashSalt` 21 / `MissingTokenVault` 22), unless `ValidateModel(options, types)` ran first.
- `AttributePolicyProvider` is always added first. Passing one yourself is ignored, null elements are skipped, and a null array means none.
- `DwPolicy.Options` is already frozen before `Configure`. Build a new `DwPolicyOptions`; never mutate `DwPolicy.Options`.
- `PrepareAsync`:
- null → `ArgumentNullException`;
- for each store provider, in order, it pins that provider's current snapshot to the context and loads the rules for every `User` subject;
- a load failure propagates;
- it returns the same instance.
- `DrainAuditAsync`:
- null context or sink → `ArgumentNullException`;
- it removes every buffered event, writes them in order, and returns how many were written;
- if the sink throws, the failing event and everything after it go back to the front of the buffer and the exception is rethrown;
- there is no retry.
- `ValidateModel` runs `PolicyModelValidator.Inspect`.
- Any error → `InvalidOperationException` whose message lists every error.
- Otherwise it returns the report, warnings included.
- Pass the options you will `Configure` with, or the salt and vault checks are skipped.
### AddDwPolicies and Bind
```
DwPolicyConfiguration static class
AddDwPolicies(this IServiceCollection services, IConfiguration section,
Action<DwPolicyOptions>? configure = null,
params IDwPolicyProvider[] providers) -> IServiceCollection
Bind(this DwPolicyOptions options, IConfiguration section) -> DwPolicyOptions same instance
```
`AddDwPolicies` does exactly this: `new DwPolicyOptions().Bind(section)` → `configure?.Invoke(options)` → `DwPolicy.Configure(options, providers)` → `services.AddSingleton(DwPolicy.Options)`.
Since 3.3.0 the singleton registered is the posture **in force**, not the instance this call built. On the first call
they are the same object. On a later call asking for the same posture the first one's is registered, so whatever
resolves `DwPolicyOptions` reads what the query path reads. A later call asking for a different posture still throws.
- There is one overload, and `section` is required (null → `ArgumentNullException`).
- `"DynamicWhere:Policies"` is a convention; any section works.
- For configuration from code only, call `DwPolicy.Configure`.
- Configuration binds first and `configure` runs second, so code wins over the file.
- It configures the static `DwPolicy` immediately, during service registration. Anything set in `configure` (`Services` included) must already exist at that point.
- It registers only the frozen `DwPolicyOptions` singleton — the one in force. No resolver, sink, vault, catalogue or provider is registered.
- Calling it a second time with the same posture is safe (3.3.0), which is what an integration suite starting several hosts needs.
- A `StorePolicyProvider` reads `StoreFailure`, `MaxSnapshotAge` and `RefreshInterval` from the options instance passed to `StorePolicyProvider.CreateAsync`.
- `AddDwPolicies` creates its options internally, so a provider cannot share them. With a store, bind by hand:
```csharp
DwPolicyOptions options = new DwPolicyOptions().Bind(builder.Configuration.GetSection("DynamicWhere:Policies"));
options.Entities.Expose<Employee>("Employee");
StorePolicyProvider store = await StorePolicyProvider.CreateAsync(policyStore, options);
DwPolicy.Configure(options, store);
builder.Services.AddSingleton(options);
```
- `options.Bind(section)` is the DynamicWhere extension and binds with `ErrorOnUnknownConfiguration = true`.
- `section.Bind(options)` is Microsoft's binder and silently ignores unknown keys. Do not use it.
- A key that no property answers to (`Caps:MinGropSize`, `Teir`) → `InvalidOperationException` at bind time.
- A value its setter refuses also fails the bind:
- a cap below its minimum;
- a `HashSalt` of 1–15 characters;
- a non-positive `MaxSnapshotAge` or `RefreshInterval`.
- Binding onto frozen options fails.
- An empty or missing section leaves every default in place, and `Caps.IsMinGroupSizeSet` stays false.
- `TokenVault`, `Services` and `Entities` are objects, not values. They cannot come from configuration; set them in `configure`.
### Configuration keys
```
Tier "Convenience" | "Strict"
DryRun true | false
IncludeTraceInResult true | false; leave it out to follow the tier (3.1.0)
AuditRefusals true | false (3.1.0)
HashSalt string; supply via user secrets, an environment variable or a vault, never a committed file
StoreFailure "LastKnownGood" | "FailClosed" | "StaticOnly"
MaxSnapshotAge TimeSpan "hh:mm:ss", e.g. "00:15:00"
RefreshInterval TimeSpan "hh:mm:ss", e.g. "00:00:30"
Caps:MaxPageSize Caps:DefaultPageSize Caps:MaxConditions Caps:MaxConditionDepth Caps:MaxConditionSets
Caps:MaxConditionValues Caps:MaxAggregates Caps:MaxOrderFields Caps:MaxNavigationDepth Caps:MaxQueryCost
Caps:DefaultFieldCost Caps:MaxAuditEvents Caps:MinGroupSize Caps:SchemaDepth Caps:SchemaCycleLimit
Caps:MaxSchemaFields integers
```
```jsonc
{
"DynamicWhere": {
"Policies": {
"Tier": "Strict",
"DryRun": false,
"StoreFailure": "LastKnownGood",
"MaxSnapshotAge": "00:15:00",
"RefreshInterval": "00:00:30",
"Caps": { "MaxPageSize": 200, "SchemaDepth": 2, "MaxSchemaFields": 2000 }
}
}
}
```
Leave `Caps:MinGroupSize` out unless you mean it. Writing any value, even 5, sets `IsMinGroupSizeSet`.
### DwPolicyOptions
```
DwPolicyOptions sealed class; every setter throws InvalidOperationException once frozen
Tier DwTier Convenience
DryRun bool false
IncludeTraceInResult bool? null 3.1.0. null follows the tier: off under Strict, on under Convenience
AuditRefusals bool false 3.1.0. true also writes every refused guarded query to the audit
HashSalt string "" "" = none; null → ArgumentNullException; 1–15 chars → ArgumentException
TokenVault IDwTokenVault? null required only by MaskStrategy.Tokenize
Services IServiceProvider? null resolves [DwMutate] transformers
StoreFailure StoreFailureMode LastKnownGood
MaxSnapshotAge TimeSpan 00:15:00 <= 0 → ArgumentOutOfRangeException
RefreshInterval TimeSpan 00:00:30 <= 0 → ArgumentOutOfRangeException
Caps DwCaps get-only
Entities DwEntityCatalog get-only, empty
IsFrozen bool get-only
Freeze() -> void idempotent; also freezes Caps and Entities; Configure calls it
MinimumHashSaltLength const int = 16
DwTier Convenience=0 Strict=1
StoreFailureMode LastKnownGood=0 FailClosed=1 StaticOnly=2
```
- `Tier`:
- a denied Select or Order is dropped in `Convenience` and thrown in `Strict`;
- Where, Group and Aggregate denials throw in both tiers.
- `Strict` also refuses:
- `getQueryString: true`, with `QueryStringDenied` (14);
- a `Segment` condition on any field the caller may not Select, with `FieldDeniedForSegment` (6).
- `Strict` also discloses less (3.1.0):
- a guarded result carries no trace unless `IncludeTraceInResult` is true;
- outside dry run, a path that matches nothing on `T` is refused as a denied field is (section 17);
- every `FieldDeniedFor*` and `CapExceeded` refusal names no field: its `FieldPath` is `"*"`;
- inside a `Segment` every field refusal is `FieldDeniedForSegment`, whatever clause refused it;
- `MissingContextValue` names neither the scoped field nor the context key: `FieldPath` `"*"`, no `SourceOrigin`;
- outside dry run, `MaxQueryCost` is checked only after every field has passed its gate.
- `IncludeTraceInResult` (3.1.0) decides whether `FilterResult<T>.Policy`, `SummaryResult.Policy` and `SegmentResult<T>.Policy` carry the `PolicyTrace` from the guarded terminals.
- null, the default, follows the tier: off under `Strict`, on under `Convenience`. `true` or `false` overrides the tier in either direction.
- The trace names the fields a policy dropped, the attribute or rule that sealed each one, and every injected predicate: the detail `Strict` already refuses through `getQueryString`. An API that serializes a result sends it to the caller.
- The trace is still recorded on `PolicyQueryable<T>.LastTrace`, and audit events do not depend on the setting.
- `AuditRefusals` (3.1.0, default false) also writes every refusal a guarded query raises to the caller's audit buffer, drained to `IDwAuditSink` like `[DwAudit]` events (section 22).
- Off by default because it changes what reaches a sink, and the ASP.NET Core audit middleware warns on every request whose events find no sink.
- `DryRun` takes effect per query as `DwPolicyOptions.DryRun || DwPolicyContext.DryRun`.
- Decisions that would throw or drop are recorded in the trace and not enforced, and `PolicyTrace.DryRun` is true.
- The group floor is recorded but not applied.
- Still enforced in dry run:
- value transforms (results stay masked);
- the `MaxAuditEvents` refusal;
- `TransformRequiresMaterialization`;
- `PolicyRequired`;
- `PolicyContextNotPrepared` and `StoreUnavailable`.
- `HashSalt` keys `MaskStrategy.Hash`. Keep it stable for the life of a deployment: changing it changes every hashed value.
- `Services`: each `[DwMutate]` type is built through `Services.GetService(type)`, falling back to `Activator.CreateInstance(type)` (which needs a parameterless constructor).
- One instance per transformer type is cached for the process lifetime, so transformers must be stateless and thread-safe.
- A type that is not an `IValueTransformer`, or a null result → `InvalidOperationException`.
- `StoreFailure`, `MaxSnapshotAge` and `RefreshInterval` are read only by a `StorePolicyProvider`, from the options passed to its `CreateAsync`.
- Once a context's pinned snapshot is older than `MaxSnapshotAge`, its queries throw `StoreUnavailable` (17) in every mode except `StaticOnly` while the provider is degraded, which serves attributes alone and never reaches the ceiling.
- `Entities` lists the types that discovery APIs may describe. Nothing is describable until it is exposed.
### DwCaps
```
DwCaps sealed class; DwPolicyOptions.Caps. Setters: frozen → InvalidOperationException, below Min → ArgumentOutOfRangeException
Cap Default Min Counts Refusal
MaxPageSize 1000 1 Page.PageSize > cap; a request with no Page is bounded by DefaultPageSize
instead, if one is set CapExceeded (9)
DefaultPageSize 0 0 3.1.0. 0 = off. When set, a guarded query that sends no Page is given
PageNumber 1 and PageSize min(DefaultPageSize, MaxPageSize). A Page the
caller did send is never replaced. Negative → ArgumentOutOfRangeException refuses nothing
Applies to Filter, Summary and Segment, terminal and composable alike: the
composable Filter, FilterDynamic and Summary return the query already
paged, so page through the request's Page, not a chained Page(). Where,
Order, Select and Group take no page and are never given one.
MaxConditions 50 1 conditions at every nesting depth; Summary: ConditionGroup + Having;
Segment: all condition sets together CapExceeded (9)
MaxConditionDepth 10 1 3.1.0. How deep SubConditionGroups nest, root group = depth 1. Summary:
the deeper of ConditionGroup and Having; Segment: each set on its own CapExceeded (9)
MaxConditionSets 10 1 3.1.0. Segment.ConditionSets.Count, sets with no conditions included.
Each set adds a condition or subquery to one statement CapExceeded (9)
MaxConditionValues 1000 1 3.1.0. Values carried by the largest single condition: the where
clause, Having and every Segment set. An In / NotIn is one comparison
per value, so one condition could build a predicate of any size CapExceeded (9)
MaxAggregates 50 1 3.1.0. Summary GroupBy.AggregateBy.Count: Summary terminals and the
composable Group and Summary. The group floor's own count is not
counted CapExceeded (9)
MaxOrderFields 10 1 Orders.Count CapExceeded (9)
MaxNavigationDepth 4 1 dot segments of any path ("A.B.C.D" passes, 5 segments refused) in
conditions, Selects, Orders, GroupBy.Fields, AggregateBy.Field CapExceeded (9)
4 is also the depth the attribute walk reads to. Raised above it, a request
can name a path no fragment of that walk reached; the member at the end of
such a path is read for its own attributes (3.3.0, section 14)
MaxQueryCost 1000 1 sum of cost over every field reference QueryCostExceeded (19)
DefaultFieldCost 1 0 cost of one reference to a field with no [DwCost] or rule weight, and
(3.1.0) of an aggregate with no Field, such as a Count
MaxAuditEvents 10000 1 undrained audit events one context may hold CapExceeded (9);
under Strict outside a dry run, the clause's own field refusal
MinGroupSize 5 1 k-anonymity floor for guarded grouped summaries; 1 = off groups suppressed
SchemaDepth 2 1 default PolicySchemaRequest.Depth
SchemaCycleLimit 2 1 times one type may appear on one schema path
MaxSchemaFields 2000 1 fields in one PolicySchema; reaching it sets Truncated, never throws
IsMinGroupSizeSet bool, get-only; true once MinGroupSize has been assigned (code or configuration)
DefaultMinGroupSize const int = 5
```
- Caps apply only to guarded queries, simulations and schema building. Unguarded calls have no limits.
- The count caps — `MaxConditions`, `MaxConditionDepth`, `MaxConditionSets`, `MaxConditionValues`, `MaxAggregates`,
`MaxOrderFields` and `MaxPageSize` — are checked before any name is resolved (3.1.0), so an oversized request is
refused with `CapExceeded` even when it also names a field that does not exist; 3.0.0 resolved names first and
answered `ConditionMustHasValidFieldName`. `MaxNavigationDepth` needs canonical paths and is checked after them.
- A Segment is one statement: its sets are combined, ordered and paged in the database (section 6), so `MaxPageSize`
and `DefaultPageSize` bound what it reads as well as what it returns. `MaxConditionSets` bounds how many sets the
statement carries; a set with no conditions spends nothing from `MaxConditions` or `MaxConditionDepth`.
- `MaxConditionValues` compares the one condition carrying the most values, wherever it is. `MaxConditions` and the
cost budget see an `In` of any length as one condition and one field.
- Cost charges every reference, duplicates included, before gating, so a later-dropped field still costs.
- Filter: conditions, Orders, and the Selects the caller wrote.
- Summary: conditions, `GroupBy.Fields`, and every `AggregateBy` entry: its `Field`, or `DefaultFieldCost` for
one with no field, such as a `Count` (3.1.0; it was free). Having and Orders are not charged.
- Segment: every field it names.
- A projection the library synthesizes is free.
- The weight is the elected `[DwCost]` or rule weight, else `DefaultFieldCost`. Under `Strict`, a name that matches nothing costs `DefaultFieldCost` too.
- Where the total is checked depends on the tier (3.1.0). The Convenience tier and every dry run check it before gating. The Strict tier, outside dry run, checks it after every field has passed its gate: a field the caller may not use, weighted or not, is refused as denied first, exactly as a name that matches nothing is, so the budget cannot tell the two apart. An allowed weighted field still gets `QueryCostExceeded` in both tiers.
- Caps and cost count what the caller sent. Forced predicates, the group floor and a default order are added afterwards and count toward neither.
- A cap or cost refusal records a `Denied` decision before throwing.
- `FieldPath` is `"*"`, or, in the Convenience tier, the path for `MaxNavigationDepth`. Under `Strict` every `CapExceeded` refusal carries `"*"` (3.1.0); the trace keeps the path.
- `Reason` and `SourceOrigin` name the cap, e.g. `"MaxConditions cap (50), request had 51"`, `"MaxConditionValues cap (1000), request had 1001"`, `"MaxAggregates cap (50), request had 51"`.
- In dry run it is recorded and not thrown.
- `MaxAuditEvents` throws even in dry run, with `FieldPath` = the audited field. Under `Strict` outside a dry run
it throws the clause's own field refusal instead of `CapExceeded`, with `FieldPath` `"*"` (3.3.0).
- `MinGroupSize`:
- The effective floor is the largest of `Caps.MinGroupSize` and the `MinGroupSize` of the transform chain on any aggregated field.
- Above 1, a guarded `Summary` with a `GroupBy` gets a `Count` column `__dwGroupSize` plus `HAVING __dwGroupSize >= floor` in SQL.
- `TotalCount` and `PageCount` therefore count only surviving groups.
- `ToList` / `ToListAsync(Summary)` remove the column from the rows.
- A summary that itself uses `__dwGroupSize` (as an alias, in Having or in Orders) → `GroupTooSmall` (20).
- Every guarded grouping path applies it: `ToList` / `ToListAsync(Summary)` and the composable `Group` and `Summary`.
- `MinGroupSize = 1` switches the floor off with no warning. `IsMinGroupSizeSet` distinguishes that from a deployment that never set it.
### DwPolicyContext and DwSubject
```
DwPolicyContext sealed class
DwPolicyContext()
Subjects : IReadOnlyList<DwSubject> read-only view, insertion order
DryRun : bool { get; set; } dry run for this caller only
IsPrepared : bool { get; } 3.1.0. True once DwPolicy.PrepareAsync has run against it,
with or without a store. ApplyPolicy(ctx) refuses a
context where this is false
Purpose : string? { get; set; } matched by purpose-bound rules; copied into audit events
PendingAuditEvents : IReadOnlyList<DwAuditEvent> a copy of the undrained buffer
WithSubject(DwSubjectKind kind, string identity) -> DwPolicyContext mutates this instance, returns this
WithValue(string key, object? value) -> DwPolicyContext mutates this instance, returns this
Identities(DwSubjectKind kind) -> IEnumerable<string>
TryGetValue(string key, out object? value) -> bool
DwSubject sealed class : IEquatable<DwSubject>
DwSubject(DwSubjectKind kind, string identity)
Kind : DwSubjectKind
Identity : string trimmed, casing kept; "" for Global
Equals / GetHashCode Kind + Identity compared OrdinalIgnoreCase
ToString() "Global" | "<Kind>:<Identity>"
DwSubjectKind Global=0 Tenant=1 Role=2 User=3 Custom=4
```
- `WithSubject` and `WithValue` return the same instance. Nothing is copied, and a context is not immutable.
- `WithSubject`:
- a null or whitespace identity → `ArgumentException`, except for `Global`, whose identity is ignored;
- adding the same kind and identity again (case-insensitive) does nothing.
- `WithValue`:
- a blank key → `ArgumentException`;
- an existing key is replaced; keys are case-sensitive;
- values feed `[DwForceWhere(ContextValue = "key")]`, and an absent key → `MissingContextValue` (12).
- Use one context per request, built completely before `PrepareAsync`.
- A `User` subject added after `PrepareAsync` makes store-backed queries throw `PolicyContextNotPrepared`.
- Calling `PrepareAsync` again replaces what was pinned.
- A context serves the one snapshot pinned at `PrepareAsync`. Once it is older than `MaxSnapshotAge`, store-backed queries throw `StoreUnavailable`, so never cache contexts across requests.
- Queries may share one context concurrently, because the audit buffer is locked. Subjects and values are not synchronized: finish building before querying.
- There is no public correlation id or attachment API; `IsPrepared` (3.1.0) says only that `PrepareAsync` ran.
---
## 13. Guarded queries
### ApplyPolicy
```
PolicyExtensions static class, DynamicWhere.ex.Policies.Source
ApplyPolicy<T>(this IQueryable<T> query, DwPolicyContext context) -> PolicyQueryable<T> reads DwPolicy.Options + DwPolicy.Resolver
ApplyPolicy<T>(this IEnumerable<T> query, DwPolicyContext context) -> PolicyQueryable<T> in-memory, via AsQueryable()
ApplyPolicy<T>(this IQueryable<T> query, DwPolicyContext context,
DwPolicyOptions options, PolicyResolver resolver) -> PolicyQueryable<T> explicit posture; DwPolicy not read
where T : class; any null argument → ArgumentNullException
```
- `ApplyPolicy` builds the handle and resolves no policy. The two overloads that read `DwPolicy` refuse an unprepared context at the call, with `PolicyContextNotPrepared` (3.1.0), and write that refusal to the context's audit buffer when `DwPolicy.Options.AuditRefusals` is on; the four-argument overload checks nothing. Every other refusal comes from the method called on the handle.
- A resolver built with `new PolicyResolver(...)` for the four-argument overload does not include `AttributePolicyProvider`. Add `new AttributePolicyProvider()` yourself, or attributes are ignored.
### PolicyQueryable<T>
```
PolicyQueryable<T> where T : class sealed class; no public constructor
Terminal: sanitize, run, transform, set result.Policy (null under Strict unless IncludeTraceInResult)
ToList(Filter filter, bool getQueryString = false) -> FilterResult<T>
ToListAsync(Filter filter, bool getQueryString = false) -> Task<FilterResult<T>>
ToListDynamic(Filter filter, bool getQueryString = false) -> FilterResult<dynamic>
ToListAsyncDynamic(Filter filter, bool getQueryString = false) -> Task<FilterResult<dynamic>>
ToList(Summary summary, bool getQueryString = false) -> SummaryResult
ToListAsync(Summary summary, bool getQueryString = false) -> Task<SummaryResult>
ToListAsync(Segment segment) -> Task<SegmentResult<T>>
Terminal, cancellable (3.2.0): the same, with the token passed to the count and the read
ToListAsync(Filter filter, CancellationToken cancellationToken) -> Task<FilterResult<T>>
ToListAsync(Filter filter, bool getQueryString, CancellationToken cancellationToken) -> Task<FilterResult<T>>
ToListAsyncDynamic(Filter filter, CancellationToken cancellationToken) -> Task<FilterResult<dynamic>>
ToListAsyncDynamic(Filter filter, bool getQueryString, CancellationToken cancellationToken) -> Task<FilterResult<dynamic>>
ToListAsync(Summary summary, CancellationToken cancellationToken) -> Task<SummaryResult>
ToListAsync(Summary summary, bool getQueryString, CancellationToken cancellationToken) -> Task<SummaryResult>
ToListAsync(Segment segment, CancellationToken cancellationToken) -> Task<SegmentResult<T>>
Composable: sanitize one clause, apply the type's forced predicates, return a new handle
Select(List<string> fields) -> PolicyQueryable<T>
Where(Condition condition) -> PolicyQueryable<T>
Where(ConditionGroup group) -> PolicyQueryable<T>
Order(OrderBy order) -> PolicyQueryable<T>
Order(List<OrderBy> orders) -> PolicyQueryable<T>
Page(PageBy page) -> PolicyQueryable<T>
Filter(Filter filter) -> PolicyQueryable<T>
Composable, returning a query the caller materializes; each throws TransformRequiresMaterialization (16)
on a type whose values are transformed on the way out for this caller
SelectDynamic(List<string> fields) -> IQueryable
Group(GroupBy groupBy) -> IQueryable
FilterDynamic(Filter filter) -> IQueryable
Summary(Summary summary) -> IQueryable
Other
AsUnguardedQueryable() -> IQueryable<T>
LastTrace : PolicyTrace?
```
How this differs from the unguarded surface:
- Composables return `PolicyQueryable<T>`, not `IQueryable<T>`. Finish with a terminal method or `AsUnguardedQueryable()`.
- There are no `IEnumerable<T>` overloads on the handle; use `ApplyPolicy(IEnumerable<T>)` instead.
- There is no synchronous Segment method, same as core.
Rules:
- Every guarded query runs over `AsNoTracking()`. Returned EF Core entities are detached, so a masked value is never
saved back. Through a provider that wraps EF Core's, as LinqKit's `AsExpandable` and DelegateDecompiler's
`Decompile` do, EF Core's extension would hand the query back still tracking, so the call is put into the query
itself (3.2.0). An in-memory source has no such copy: without `Selects` and with nothing denied, the rows returned are
the source objects themselves, transformed in place. When a projection is synthesized the rows are new, and they
hold no member of an object type, so the source objects are left as they were.
- Chaining keeps decisions: each composable returns a new handle, and the terminal's trace includes the earlier links' decisions.
- The terminal's trace is on `result.Policy` only when `DwPolicyOptions.IncludeTraceInResult` allows it: null follows the tier (off under `Strict`, on under `Convenience`), and `true` or `false` overrides it (3.1.0). `LastTrace` holds it whatever the setting.
- `getQueryString: true` under `Strict` → `QueryStringDenied` (14). This is checked before sanitizing; dry run records it instead.
- `TransformRequiresMaterialization` depends on the type, not on the fields named.
- It is thrown even in dry run, and `FieldPath` lists the transformed paths — except under `Strict` outside a
dry run (3.3.0), and except where the policy names no transformed path at all, where it is `"*"` in both
tiers (3.3.0).
- It asks two questions since 3.3.0: whether the policy names a transformed path, and whether a row of the
type can hold a member that declares a transform attribute anywhere in what the type can reach — a member
only a subtype declares, one five segments down, one of an object a dictionary holds. The outbound walk
transforms those from their own attributes (section 18), so a type whose only transforms sit there is
transformed on the way out as well; reading the named paths alone handed such a type its query, and its rows
came back exactly as stored. The second question is asked only of a resolver that reads attributes.
- With no named column to list, the refusal names the clause: `FieldPath` is `"*"` in both tiers, where under
`Convenience` it otherwise lists the transformed columns. The `AuditRefusals` event records the same text.
- A type nothing transforms anywhere still gets its query.
- Use `ToListDynamic` or `ToList(Summary)`, or leave deliberately with `AsUnguardedQueryable()`.
- `Where`, `Order` and `Page` never fail with `AllSelectsDenied`: no projection is synthesized for a single clause.
- `Group(GroupBy)` runs its sanitized summary through the summary pipeline, so forced predicates and the
`MinGroupSize` floor both apply, as they would on `ToList(Summary)`. Core `Group` takes no `Having`, which is
why it is not the path used.
- `Summary(Summary)` applies the floor's HAVING too. Neither returns the floor's own `__dwGroupSize` column —
both project it back out — and both keep canonical column names, because alias renaming happens on
materialized rows.
- `AsUnguardedQueryable()` returns the handle's source as a plain `IQueryable<T>`.
- It keeps what earlier composable calls applied: forced predicates, clauses, `AsNoTracking`.
- It skips everything after: no gating, no transforms (values come back unmasked), no trace.
- On a fresh handle it is the original, tracked source with no predicates.
- Calling a DynamicWhere extension on it for a `RequirePolicy` type → `PolicyRequired`.
- `LastTrace` is set on the handle the method was called on, not on the handle it returns.
- `LastTrace` is assigned before the request is sanitized (3.3.0), so a refused request leaves its own trace
readable. It used to be assigned after, so a refusal left the previous request's trace, or null. A strict refusal
names no field, and the trace is where the real path and reason are.
- It holds the latest call the handle made: one that got past sanitizing, one sanitizing refused, or a
`QueryStringDenied` refusal.
- Refusals throw `PolicyException` (a `LogicException`) from the handle method.
- With `DwPolicyOptions.AuditRefusals` on, the terminal and composable methods also write the refusal to the context's audit buffer on its way out, once, without changing or catching it (section 22).
### Default order — `[DwEntity(DefaultOrder = ...)]` (3.1.0)
```csharp
[DwEntity(DefaultOrder = "CreatedAt desc, Id")] // DynamicWhere.ex.Policies.Attributes
public class Ticket { … }
```
- The order a guarded query takes when its caller sends none. Comma-separated entries, each a field path
(navigations allowed) optionally followed by `asc` or `desc` in any letter case; ascending when neither. A blank
entry, such as the one a trailing comma leaves, is ignored, and a field named twice is used once.
- Applied only under `ApplyPolicy`, when `Orders` is null or empty: `ToList`, `ToListAsync`, `ToListDynamic` and
`ToListAsyncDynamic` with a `Filter`; `ToListAsync(Segment)`; the composable `Filter` and `FilterDynamic`; and the
composable `Page` on a source nothing has ordered and whose projection hides no default field.
- Never applied:
- by an unguarded call. The core methods of section 6 on a plain `IQueryable<T>` or `IEnumerable<T>`, `Page`
included, never read the attribute and order only as their caller asks, exactly as in 3.0; so does a DynamicWhere
method called on what `AsUnguardedQueryable()` returns;
- when the caller sends orders: the default is not appended as a tiebreak;
- to a source already ordered, before it was guarded (`db.Tickets.OrderBy(t => t.Title).ApplyPolicy(ctx)`) or by
a composed `Order` earlier in the chain — even one whose every order the policy dropped, because the caller
still sent orders. A composed `Filter` that sent orders counts the same way (3.2.0). Only the query's expression
is read, so a sequence sorted in memory before `ApplyPolicy(IEnumerable<T>)` does not count as ordered: send
`Orders` for it;
- to a source whose projection could hide a default field. Only the outermost `Select` of the chain counts, because
it makes the rows the default orders. Since 3.2.0 it hides nothing when it builds T itself in an object
initializer, `Select(t => new Row { Code = t.Code, … })`, and assigns every field the default names, at every
level of a nested path (`"Owner.Name"` needs `Owner = new OwnerRow { Name = … }`), a column. On EF Core a column
is a member the model maps on the entity the `Select` reads, read directly, through reference navigations
(`t.Owner.Name`) or through `EF.Property` (a shadow property included); the default then applies, because EF
Core translates an order by a column the projection assigned. In memory any assigned field is ordered by. A
value the projection computes, by a method (`Regex.Replace`, `ToUpper`, the application's own) or an operator
(`t.First + " " + t.Last`), a member the model does not map, a constructor with arguments, a default field the
initializer does not assign, or a nested path through anything but an initializer leaves the query in its own
order, as every projection did in 3.1.0: ordering by it could fail where the unguarded query ran;
- after a projection composed on the handle: the guarded `Select`, or a guarded `Filter` whose `Selects` is set,
leaves the rest of the chain unordered even when it keeps every default field, so
`guarded.Select(["Id", "Title"]).Page(page)` pages as it did in 3.0.0, unordered. The default is for the rows the
caller's source makes;
- to a `Summary`, or by the composable `Where`, `Select` and `Order`.
- A type that declares no `DefaultOrder` is never ordered by the library, guarded or not; only a caller's own
orders apply. End a default with a unique field, such as the key, or rows sharing the leading values can still
change places between pages.
- `[DwEntity]` allows one attribute per type and is inherited the .NET way: a `[DwEntity]` on a derived type replaces
its base type's instead of merging with it. `[DwEntity(RequirePolicy = true)]` on a type whose base declares
`DefaultOrder` has no default order, and the reverse drops `RequirePolicy`. Repeat both on the derived type.
- An entry naming a field the type does not have, one that is not a field and a direction (`"Id sideways"`), one
the core refuses to order by, a path ending on a collection of entities (`"Tags"`), or one whose name the parser
keeps for itself (`"Null"`, section 5) is skipped, never refused, and `PolicyModelValidator` reports all four
(section 20). A path through a collection to a value (`"Tags.Value"`) is
kept and sorted as section 6 sorts it, by the smallest value ascending or the largest descending.
- The sanitizer applies it for this caller, after the caller's own orders are gated:
- a default field this caller may not order by is left out, never refused, and recorded as a `Dropped` decision
on `Order` whose reason starts `left out of the default order`. Ordering by it would rank rows by a value the
caller may not see;
- in a `Segment`, a default field this caller may not use in a segment is left out too, recorded the same way,
because a segment refuses that field in any clause. A `Filter` still takes it;
- a dry run keeps the field and still records the decision;
- a field the default keeps is a use of that field. One audited for `Order`, by `[DwAudit]` or a rule, is recorded
as a use, `Effect` `Allow`, each time a guarded query orders by it, as a caller's own order is. A field the
default leaves out is not recorded: the query does not order by it, and the caller never named it. A dry run
keeps the field, so it records it, with its `Order` effect (`Deny` for a field the caller may not order by) and
`DryRun` true (section 22);
- a caller whose sent orders were all dropped (Convenience) gets no default in their place;
- it is added after the caps and after the cost is counted, so it counts toward no cap and costs nothing.
### [DwEntity(RequirePolicy = true)] enforcement
- All 28 public methods of `DynamicWhere.ex.Source.Extension` (the `IQueryable<T>` and `IEnumerable<T>` overloads) run the guard first.
- They throw `PolicyException` with `ErrorCode = PolicyRequired` (10) when `T` requires a policy and the call is not running inside a `PolicyQueryable<T>` method.
- Composables such as `Select` and `Where` throw when called, not when enumerated.
- The exception carries:
- `FieldPath` = `typeof(T).Name`;
- `Feature` = `None`;
- `Tier` = `Strict` (whatever is configured);
- `SourceOrigin` = `"DwEntityAttribute(RequirePolicy = true)"`.
- Neither dry run nor `DwPolicy.IsConfigured` affects it.
- It checks `T`, the query's element type. The attribute is inherited by subclasses, and the answer is cached per type for the process lifetime.
- A subclass that declares a `[DwEntity]` of its own, for a `DefaultOrder` say, replaces the inherited one, and its `RequirePolicy` is false unless it says `true` again.
- Not guarded:
- plain EF Core or LINQ on the type (`db.Employees.ToListAsync()`, `.Where(e => ...)`);
- anything done with `AsUnguardedQueryable()` that avoids DynamicWhere extension methods.
---
## 14. Policy attributes
Namespace `DynamicWhere.ex.Policies.Attributes`; `[DwX]` is class `DwXAttribute`. There are 22
attributes plus the abstract base `DwPolicyAttribute`.
```
AttributeUsage (every attribute: Inherited = true)
DwEntity Class AllowMultiple = false
DwDeny DwDenied DwNoWhere DwNoSelect DwNoOrder DwNoGroup
DwNoAggregate DwOperators DwForceWhere Property|Field AllowMultiple = true
every other attribute Property|Field AllowMultiple = false
```
- Only public instance properties are read, including properties reached through navigations and
collection elements, up to paths of 4 segments. An attribute on a field compiles and does nothing.
- A request can name a longer path only where a host raises `Caps.MaxNavigationDepth` above 4. The attributes of
the member at the end of such a path are then read directly (3.3.0): the deny family, `[DwOperators]`, the
transform stages, `[DwCost]`, `[DwAudit]`, `[DwDescribe]` and `[DwAllowedValues]`. What is declared about the
queried entity itself is left out there, as it is around a cycle: `[DwAlias]`, `[DwRequireWhere]`,
`[DwForceWhere]`. Only a resolver that reads attributes does this, which every resolver `DwPolicy.Configure`
builds does. Until 3.3.0 no fragment reached such a path, so a `[DwDenied]` member at segment five was filtered
on, grouped by and returned, under `Strict`.
- A type first met at the fourth segment used to read as a type reached from itself wherever it was met again in
the same walk, so what a cycle leaves out — `[DwForceWhere]`, `[DwRequireWhere]`, `[DwAlias]` — was left out of a
shorter path reaching that type directly, and which of two members was declared first decided whether a forced
tenant scope applied. Fixed in 3.3.0: the three apply on every path within 4 segments that is not around a
cycle.
- Every attribute except `DwEntity` derives from `DwPolicyAttribute` and has `Overridable:bool = false`.
False places it at `PolicyLevel.SealedAttribute`; true places it at `PolicyLevel.OverridableAttribute`.
- A type's attributes are read once and cached. A malformed one throws `ArgumentException` on every
guarded query of that type, and of every type that navigates to it.
```
Access control
[DwEntity] RequirePolicy:bool = false DefaultOrder:string? = null (3.1.0)
[DwDeny(PolicyFeature features)] Features:PolicyFeature
[DwDenied] DwDeny(PolicyFeature.All)
[DwNoWhere] DwDeny(PolicyFeature.Where)
[DwNoSelect] DwDeny(PolicyFeature.Select)
[DwNoOrder] DwDeny(PolicyFeature.Order)
[DwNoGroup] DwDeny(PolicyFeature.Group)
[DwNoAggregate] DwDeny(PolicyFeature.Aggregate)
[DwOperators] Allow:Operator[]? = null Deny:Operator[]? = null
Resolve() -> IReadOnlyList<Operator>
Injection
[DwAlias(string name)] Name:string
[DwForceWhere(Operator op)] Operator:Operator Value:string? = null
ContextValue:string? = null AllowNull:bool = false (3.1.0)
[DwRequireWhere] Operators:Operator[]? = null Resolve() -> IReadOnlyList<Operator>
static DefaultOperators = { Equal, IEqual, In, IIn }
Transformation (each also has AllowAggregate:bool = false MinGroupSize:int = 0)
[DwMask(MaskStrategy strategy)] Strategy KeepStart:int = 0 KeepEnd:int = 0
MaskChar:char = '*' PreserveLength:bool = true
Pattern:string? = null Replacement:string? = null
Text:string? = null TokenScope:string? = null
[DwMutate(Type transformer)] Transformer:Type
[DwDefault] [DwDefault(string value)] Value:string? HasValue:bool (true only via the string ctor)
[DwGeneralize(GeneralizeMode mode)] Mode Step:int = 0 Part:DatePart = DatePart.Year
Decimals:int = 0
[DwTruncate(int length)] Length:int Ellipsis:string? = null
[DwFormat(string format)] Format:string
Discovery, budget, audit
[DwDescribe] Label:string? Description:string? Group:string? Order:int
[DwAllowedValues(params string[] values)] Values:string[]
[DwCost(int weight)] Weight:int
[DwAudit] [DwAudit(PolicyFeature features)] Features:PolicyFeature (no-arg ctor = PolicyFeature.All)
```
No named attribute refuses `Segment`: write `[DwDeny(PolicyFeature.Segment)]`.
### DwEntity
- `RequirePolicy = true`: any DynamicWhere.ex extension method on this `T` throws `PolicyException`
`PolicyRequired` when called outside a guarded call. That covers `Select`, `SelectDynamic`, `Where`,
`Order`, `Page`, `Group`, `Filter`, `FilterDynamic`, `Summary` and every `ToList*`, on `IQueryable<T>`
and `IEnumerable<T>`. The exception's `FieldPath` is the type name and `Tier` is `Strict`.
- It is checked whether or not `DwPolicy.Configure` ran. Plain LINQ or EF on the `DbSet` is not
intercepted.
- `DefaultOrder` (3.1.0), e.g. `[DwEntity(DefaultOrder = "CreatedAt desc, Id")]`: the order a guarded query
takes when its caller sends none, less the fields this caller may not order by. Unguarded calls ignore it.
Entry syntax, entry points and exclusions are in section 13; `ValidateModel` checks it (section 20).
- A derived type reads its own `[DwEntity]` when it has one, and that attribute replaces the base type's whole: set
`RequirePolicy` and `DefaultOrder` again on it.
### DwDeny family
- Each attribute refuses its features on that exact path. Several on one member all apply.
- A denial on a navigation (`Contact`) covers only `Contact`. `Contact.Email` is a separate field: it is declared
by an application's own type, so an attribute can be placed on it. Decorate it, or deny its path.
- A path that continues beneath a member whose type the framework declares is not a separate field (3.3.0). No
attribute can be placed on `Salary.Value` or `Salary.HasValue` on a `decimal?`, `Secret.Length` on a `string`,
`Born.Year` or `Born.Date.Year` on a `DateTime`, `Bag.Count` on a dictionary, or `Lines.Count` on an
application's own collection class — the collection's own member, not an element's — and no fragment named such
a path, so it resolved as allowed: a `[DwDenied] decimal?` was filtered on, sorted by, grouped by with its
values as the group keys, aggregated (`MAX(Salary.Value)`) and handed back by a dynamic projection, under
`Strict`. Such a path now takes every fragment of the member it reads, whichever provider supplied it — an
attribute and a store rule on `Salary` both cover `Salary.Value`: the deny effects per feature, the
`[DwOperators]` restriction (intersected), the `[DwCost]` weight and the audited features. A rule naming the
sub-path itself still applies alongside.
- One feature is one feature: `[DwNoWhere] Born` refuses `WHERE Born.Year` and still allows
`GROUP BY Born.Year`. A member nothing denies is read beneath exactly as before, so `Name.Length` still runs.
- It does not take what is said to the caller about the member: the alias, the required filter
(`[DwRequireWhere]` on `TenantId` is not satisfied by a filter on `TenantId.Value`), the forced scope, and the
descriptive facts — label, description, group, order, allowed values.
- Where the member it reads is transformed, `Select`, `Group` and `Aggregate` on the sub-path are refused
(section 18): there is nothing beneath the member to apply the transform to.
- A member only a subtype of the navigated type declares is not such a path: `Zone.Parent`, where a subclass of
Zone's type declares `Parent`, is decided by the fragments naming it, as before, so a grant of `Zone` under a
`"*"` deny does not grant it.
- `[DwDeny(PolicyFeature.None)]` refuses nothing, and nothing reports it.
- `[DwNoSelect]` leaves the field filterable, sortable, groupable and aggregatable, so it can still be
counted.
- One on another declaration of the member applies to the path too (3.2.0): on the interface member a class, or a
loaded subtype of it, implements with the member; on an override of either accessor a loaded subtype declares; on
a public member a loaded subtype hides with `new`; and on the implementation a loaded type gives an interface
member, explicit, inherited from a base class or declared by an open generic class, through that interface or an
instantiation variance lets stand for it (`IFeed<VisaCard>` for a member typed `IFeed<Card>`, with `out T`). A row read through the base type or
the interface is still that subtype, and its member returns what the subtype's declaration returns, so the denial
holds for the path on every row, Where, Order, Group and Select alike. A member hidden with `new` counts because
whether it reads the member it hides cannot be told from outside, and a row serialized as its own type writes it
under the same name. Only the deny family is read this way; aliases, transforms and the other attributes are read
from the declaration walked.
### DwOperators
- `Resolve()` returns `Allow` (every `Operator` when null) minus `Deny`. An operator in both lists is
refused. `Allow = new Operator[0]` permits nothing, so the field cannot be filtered at all.
- Every restriction matching the field is intersected: repeated attributes, rules at any level, and `"*"`
rules. A rule can only narrow the set, so `Overridable` changes nothing. A path beneath a member whose type the
framework declares intersects that member's restriction too (3.3.0), so a restriction on `Salary` holds for
`Salary.Value`.
- It is checked on every WHERE condition at any depth and in every Segment set. It is also checked on a
`Having` condition that reaches the field through a group key or an aggregate alias.
- A failure throws `OperatorNotAllowed` in both tiers. If the field is also denied for Where,
`FieldDeniedForWhere` is raised instead.
- It is never applied to forced predicates.
### DwAlias
- `Name` is trimmed. A blank, dotted (`a.b`) or `"*"` name throws `ArgumentException`.
- Two members of one type with the same alias (case-insensitive) is a `ValidateModel` error.
- The alias is accepted wherever a path is: condition `Field`, `Selects`, `Orders`, `GroupBy.Fields` and
`AggregateBy.Field`. It is also accepted in `Having` and summary `Orders` names that refer to an aliased
group key.
- Matching is case-insensitive, and the real path still works.
- A name that could mean two fields throws `AmbiguousFieldName` under `Convenience` and in a dry run. That
happens when an alias equals another real path or another alias. Under `Strict` outside a dry run it is
refused as an unknown name is — the clause's own code, `FieldPath` `"*"` — because a name matching two
fields matches at least one (3.3.0); the trace records which fields it matched.
- Exception: one member reached both at the root and through navigations (`Code`, `Manager.Code`)
resolves to the root.
- `PolicyException.FieldPath` carries the name the caller wrote. The trace records the canonical path,
with the alias in the reason. Under the Strict tier a `FieldDeniedFor*` or `CapExceeded` refusal names no
field at all, alias or path: its `FieldPath` is `"*"` (3.1.0, section 17).
- Output renaming happens only in `ToListDynamic`, `ToListAsyncDynamic` and
`ToList`/`ToListAsync(Summary)`.
- A row holding an aliased column is rebuilt as an `ExpandoObject` with that column under the alias.
- A nested path's column is matched by its path without dots.
- Typed `FilterResult<T>` and `SegmentResult<T>` rows keep their member names.
- An alias that stands for more than one path is not renamed.
- An alias is not applied on a type reached from itself (`Employee.Manager.Code`). Until 3.3.0 a type first met
at the fourth segment counted as reached from itself wherever it was met again in the same walk, which dropped
the alias from a shorter path reaching that type directly.
- A path beneath a member whose type the framework declares does not take that member's alias (3.3.0): an alias
is a public name for the member, and `Salary.Value` is not the member.
### DwForceWhere
- Set exactly one of `Value` or `ContextValue`. `Operator.IsNull` and `Operator.IsNotNull` take neither.
Any other combination throws `ArgumentException` on every guarded query of the type. Since 3.1.0
`ValidateModel` reports it at startup, and so it does an unsupported member type and a misused
`AllowNull`.
- `DataType` comes from the member's CLR type:
- enum → `Enum`
- `string`, `char` → `Text`
- `Guid` → `Guid`
- `bool` → `Boolean`
- `DateOnly` → `Date`
- `DateTime`, `DateTimeOffset` → `DateTime`
- any numeric type → `Number`
- any other type throws `ArgumentException`.
- `Value` is the injected condition's only value, parsed by the query pipeline.
- `ContextValue` is a key looked up with `DwPolicyContext.TryGetValue`. Keys are set by `WithValue` and
are case-sensitive.
- A missing or null context value throws `MissingContextValue` in both tiers. In dry run it is recorded
and nothing is injected.
- Convenience: `FieldPath` is the scoped field and `SourceOrigin` names it and the context key.
- Strict (3.1.0): `FieldPath` is `"*"` and `SourceOrigin` is null, so the message names neither the scope's
column nor the key it reads, which together describe how rows are partitioned. The trace keeps both, and an
`AuditRefusals` event records the scoped field's canonical path.
- Each predicate carries one value, or none for a null check, so `Between` and `NotBetween` cannot be
forced.
- `AllowNull = true` (3.1.0) lets a row whose member is null through as well: the injected term is
`(field op value OR field IS NULL)`. It is for a row that belongs to one tenant or to none, such as a
system role no institution owns. Two forced predicates on one member are joined by `And`, so no
combination of them can say "or null".
- `[DwForceWhere(Operator.Equal, ContextValue = "TenantId", AllowNull = true)] public int? InstitutionId`
- It works with every operator that takes a value. With `Operator.IsNull` or `Operator.IsNotNull`, or on
a member that can never be null (a non-nullable value type), it throws `ArgumentException` when the
type's policy is resolved, and `ValidateModel` reports it.
- A `ContextValue` is still required: a missing or null one throws `MissingContextValue`. The rows that
pass widen; the caller's own scope does not.
- The term is a disjunction, so it does not satisfy a `[DwRequireWhere]` on the same member.
- Dry run injects nothing, as for every forced predicate.
- The predicate is added after gating. It is never checked against the caller's own policy and never
counts toward caps or cost. Shape:
`ConditionGroup { Sort = 0, Connector = And, Conditions = forced (Sort 0,1,…), SubConditionGroups =
[caller's group, Sort reset to 0, then one Or group per AllowNull predicate (Sort 1,2,…)] }`. With no
caller group, only the forced terms are sent. The result reads `(A OR B) AND TenantId = 5`, or with
`AllowNull` `(A OR B) AND (InstitutionId = 5 OR InstitutionId IS NULL)`: a caller's `Or` never merges
with a forced term.
- It applies to `Filter`, to `Summary.ConditionGroup` (not `Having`), to every Segment condition set, and
to the guarded composable methods.
- A Segment with no sets gets one set holding the scope.
- The trace records `PolicyAction.Injected` on `Where`, with reason `forced predicate (Equal)`, or
`forced predicate (Equal, or null)` for a term that admits null (the operator name varies).
- Predicates are collected and ANDed, never elected: repeated attributes, rules at every level, `"*"`
rules. No rule can remove one, so `Overridable` changes nothing.
- A predicate declared on a navigated type is injected through the navigation, up to 4 segments, except
onto a type reached from itself. For example, a query on `Order` injects `Buyer.TenantId`; through a
collection the condition becomes `Any`.
- Until 3.3.0 a type first met at the fourth segment read as a type reached from itself wherever it was met
again in the same walk, so its forced scope was dropped from a shorter path reaching it directly, and which
of two members was declared first decided whether the scope applied. It applies on every path within 4
segments that is not around a cycle now. A query that ran unscoped is scoped.
- A path beneath a member whose type the framework declares takes no forced scope from that member (3.3.0): the
scope is about the rows, not about the value one segment down.
### DwRequireWhere
- `Operators = null` means `DefaultOperators` (Equal, IEqual, In, IIn). An empty array means nothing
satisfies the requirement, so every query on the type is refused.
- It is satisfied only by a WHERE condition on the field whose operator is in the set and that is in a
narrowing position. The condition's group and every ancestor group must use `Connector.And` or hold at
most one child (conditions plus subgroups).
- `Status = A OR TenantId = 5` does not satisfy it.
- A `Having` condition never does.
- It is checked after injection, so a `[DwForceWhere]` on the same field satisfies it when its operator
is in the set. Dry run injects nothing. A forced predicate with `AllowNull = true` never satisfies it:
its term sits in an `Or` group, which is not a narrowing position, so the caller must still filter on
the field (3.1.0).
- A missing filter throws `RequiredFilterMissing` in both tiers. `FieldPath` is the field's alias when it
has one.
- In a Segment, every condition set must satisfy it.
- It is checked on every guarded call, so a lone `.Order(...)` or `.Page(...)` on the handle also throws.
- It follows navigations the same way `[DwForceWhere]` does: a query on `Order` can require
`Buyer.Division`, and since 3.3.0 on every path within 4 segments that is not around a cycle, so a type first
met at the fourth segment no longer drops it from a shorter path. A requirement a caller was never asked for
may now be demanded.
- A filter on a path beneath the member does not satisfy it (3.3.0): `[DwRequireWhere]` on `TenantId` asks for a
filter on `TenantId`, which `TenantId.Value` is not.
- It is elected: a rule cannot lift a sealed requirement but may add one where none exists. A `"*"` rule
cannot carry one.
### DwDescribe, DwAllowedValues
- Both are schema metadata and decide nothing. `[DwAllowedValues]` is not enforced: a filter on an
unlisted value is allowed.
- `Label`, `Description`, `Group`, `Order` and `AllowedValues` are each elected separately. A rule that
sets only a label keeps the attribute's other values.
- An unset `Order` reads as 0 but counts as not set.
- `[DwDescribe]` with nothing set, or `[DwAllowedValues]` with no values, is a `ValidateModel` error. A
blank text value or blank list entry throws `ArgumentException`.
- A `"*"` rule cannot carry either.
### DwCost
- `Weight = 0` means the field is free. A negative weight is a `ValidateModel` error and throws
`ArgumentOutOfRangeException`.
- Every reference the caller writes is charged before gating, so a denied field costs the same as an
allowed one.
- Filter: each condition at any depth, and each `Orders` and `Selects` entry.
- Summary: `ConditionGroup` conditions, `GroupBy.Fields`, and each `AggregateBy` entry — its `Field`, or
`Caps.DefaultFieldCost` for an aggregate with no field, such as a `Count` (3.1.0; before, it was free).
- Segment: every field in every clause.
- Not charged: a synthesized projection, `Having`, summary `Orders`, forced predicates and the group-size
count.
- An unweighted field costs `Caps.DefaultFieldCost` (default 1, may be 0).
- A path beneath a member whose type the framework declares costs that member's weight (3.3.0): `Notes.Length`
costs what `Notes` costs, where it used to cost the default.
- A total above `Caps.MaxQueryCost` (default 1000) throws `QueryCostExceeded` in both tiers. In dry run
it is only recorded.
- The Convenience tier checks the total before gating. The Strict tier checks it after every field gate
(3.1.0), so a weighted field the caller may not use is refused as denied, exactly as a name that matches
nothing is, before its weight could set the two apart.
- The weight is elected: the top-ranked one wins, and among fragments tied at that rank the largest wins.
A `"*"` rule may set a weight for every field.
### DwAudit
- `PolicyFeature.None`, or a value with an undefined bit, is a `ValidateModel` error and throws
`ArgumentException`.
- One `DwAuditEvent` is recorded per use of an audited feature on a field the request names, whether the
use is allowed or refused. A field named twice is recorded twice. Events are buffered on the context
until drained.
- A field the type's `DefaultOrder` adds is recorded too (3.1.0): audited for `Order`, it is an `Order` use,
`Effect` `Allow`, each time a guarded query orders by it (section 13).
- Not recorded:
- a `DefaultOrder` field left out for this caller, outside a dry run
- forced predicates
- applying an output transform, which is no use of its own
- columns returned because the request had no `Selects` — until 3.3.0, which records every audited member the
query hands back (below)
- If the context already holds `Caps.MaxAuditEvents` (default 10000) events, the query is refused, in both tiers
and in dry run. `CapExceeded`, except under `Strict` outside a dry run, where it is the clause's own field
refusal instead (3.3.0): only a real, audited field can reach the cap, so answering with the cap's own code
would set such a field apart from a name that matches nothing.
- A use is what the request reads: a field it names, a default-order field the query adds, and — since 3.3.0 —
every audited member a projection the caller did not name hands back, recorded for `Select`. One event per query,
not per row. Until 3.3.0 an empty `Selects` returned the value with nothing written down.
- A path beneath a member whose type the framework declares is audited as that member is (3.3.0): reading
`Salary.Value` records what reading `Salary` records, where it used to record nothing.
- A member the rows hand back where no path of the policy names it is recorded once the rows show it (3.3.0).
The gate records a use by path, before the query runs, and a member only a subtype of the row's type declares,
or one past the four segments the attribute walk reads, has no path it could ask about: handed back inside a
row returned whole or a navigation kept whole, it was read with nothing written down. The outbound walk's
second pass reports each one it meets and the terminal records it (section 22): one event per path per query,
`Feature` `Select`, `Effect` `Mask` where the member is transformed as well and `Allow` otherwise.
- Only a member its own `[DwAudit]` audits for Select, and only where the projection carries it: a member the
projection left out is not a read.
- A member the declared types hold within four segments is the gate's and is left to it, and so is a path the
projection spells out, however long — neither is recorded twice.
- Read only by a resolver that reads attributes. A model that declares neither an audit for Select nor a
transform anywhere pays for no second pass.
- What is handed back is read strictly: a member kept whole records the audited paths inside it; a navigation
nothing loads records nothing, since the caller receives null for it; a value the source does not carry records
nothing. A dry run applies no projection, so everything the row carries is recorded, a denied member included.
- A refusal becomes an event of its own, with an `ErrorCode`, only when `DwPolicyOptions.AuditRefusals` is
on, and then for every refused guarded query, audited field or not (3.1.0, section 22).
- It is elected: the top-ranked audit wins outright, and fragments tied with it are unioned. A rule can
neither widen nor narrow a sealed `[DwAudit]`. A `"*"` rule may audit every field.
---
## 15. Policy enums
Namespace `DynamicWhere.ex.Policies.Enums`, except `TransformKind`, which is in
`DynamicWhere.ex.Policies.DTOs`. Stored rules name enum members by name, never by number.
```
PolicyFeature [Flags] None=0 Where=1 Select=2 Order=4 Group=8 Aggregate=16 Segment=32 All=63
PolicyLevel SealedAttribute=1 DynamicUser=2 DynamicRole=3 DynamicTenant=4 DynamicGlobal=5
OverridableAttribute=6
PolicyEffect Allow=0 Mask=1 Deny=2
PolicyAction Allowed=0 Denied=1 Dropped=2 Masked=3 Injected=4 Mutated=5 Defaulted=6 Generalized=7
DwTier Convenience=0 Strict=1
DwSubjectKind Global=0 Tenant=1 Role=2 User=3 Custom=4
MaskStrategy Full=0 Partial=1 Email=2 Phone=3 Regex=4 Fixed=5 Hash=6 Null=7 Tokenize=8
GeneralizeMode Round=0 Bucket=1 DatePart=2 Truncate=3
DatePart Year=0 Quarter=1 Month=2 Day=3
TransformKind Mutate=0 Generalize=1 Format=2 Mask=3 Truncate=4 Default=5
StoreFailureMode LastKnownGood=0 FailClosed=1 StaticOnly=2
```
```
Where / Select / Order conditions (plus Having through a key or alias) / Selects / Orders
Group / Aggregate GroupBy.Fields / AggregateBy.Field
Segment any use inside a Segment
PolicyFeature.None fragment that only carries something: alias, operators, predicate,
requirement, facts
PolicyEffect.Mask allowed and transformed on output; emitted on Select by transform attributes
PolicyAction.Allowed untouched; also recorded when an alias renames an output column
PolicyAction.Denied refused: thrown, or recorded in dry run
PolicyAction.Dropped removed quietly: Convenience Order/Select, synthesized projection, group floor
PolicyAction.Injected forced predicate added
Masked/Mutated/ one per transformed path; Defaulted > Mutated > Masked > Generalized;
Defaulted/Generalized a chain of only Format/Truncate records Masked
DwTier.Convenience the default; refused Order and Select entries are dropped
DwTier.Strict refused Order/Select throw; getQueryString and Segment filters on
select-denied fields are refused; results carry no trace unless
IncludeTraceInResult; an unknown path is refused as a denied field is,
and field and cap refusals name no field (3.1.0)
DwSubjectKind rule level: User→DynamicUser, Role→DynamicRole, Tenant and Custom→DynamicTenant,
Global→DynamicGlobal (takes no key)
StoreFailureMode LastKnownGood (default) serves the last snapshot up to MaxSnapshotAge;
FailClosed refuses guarded queries (StoreUnavailable); StaticOnly enforces
attributes alone
```
---
## 16. Precedence
```
Level PolicyLevel Comes from
1 SealedAttribute attribute with Overridable = false (the default)
2 DynamicUser rule for DwSubjectKind.User
3 DynamicRole rule for DwSubjectKind.Role
4 DynamicTenant rule for DwSubjectKind.Tenant or DwSubjectKind.Custom
5 DynamicGlobal rule for DwSubjectKind.Global
6 OverridableAttribute attribute with Overridable = true
```
`DwPolicy.Configure` always adds `AttributePolicyProvider`. No rule can reach level 1.
Resolution for one field path and one feature (`Where`, `Select`, `Order`, `Group`, `Aggregate`,
`Segment`):
1. A fragment matches when its path equals the field path, compared case-insensitively after trimming
each segment and dropping empty ones. A path of `"*"` also matches: it means every field of the entity,
nested paths included. `"Contact.*"` is a literal path, not a wildcard.
2. Among matching fragments whose `Features` include the feature, the winner is decided by, in order:
- the lowest level;
- an exact path over `"*"`;
- the higher `Priority` (rules only; attributes are 0);
- the stronger `PolicyEffect`: `Deny` > `Mask` > `Allow`.
A complete tie keeps the first fragment found, which has the same effect anyway.
3. Weaker levels are discarded, not merged. When nothing matches, the feature is allowed.
4. Afterwards, if the field has a transform stage and any stage lacks `AllowAggregate`, `Aggregate` is set
to `Deny`, overriding whatever won.
```
Fragment from Enters the feature contest as Also carries
DwDeny family Deny on Features -
transform attribute Mask on Select one transform stage
every other attribute PolicyFeature.None (no contest) operators / predicate / alias / requirement / fact
```
```
Carrier Combined how
AllowedOperators intersected across every match at every level (empty = none)
forced predicates all kept, ANDed
Alias, RequiredOperators elected by the ranking above
each TransformKind stage elected separately per stage kind
Label Description Group Order AllowedValues elected separately per fact
CostWeight elected; ties at the winning rank take the largest
AuditedFeatures elected; ties at the winning rank are unioned
```
- Sealed means no rule can win anything a sealed attribute decides:
- its Deny;
- the Mask on Select that a transform attribute adds, so no rule can deny Select on a field with a
sealed transform;
- its stage kind, alias, requirement, fact, weight and audit.
A rule can still decide features the attribute does not touch, and can add stages of other kinds.
- An `Overridable` attribute loses to any rule, at any dynamic level, that decides the same feature or
carries the same stage kind or fact.
- `Overridable` does nothing on `[DwOperators]`, which are intersected, or `[DwForceWhere]`, which are
collected.
- A rule never removes a transform stage. It can only win that stage kind with a stage of its own, and a
stored rule cannot carry `Mutate`. An Allow rule on Select leaves the mask running.
- Level is compared first. A user rule beats a role rule. A `"*"` rule at a stronger level beats an exact
rule at a weaker one.
- Within one level, an exact-path Allow beats a `"*"` Deny, and a higher-Priority Allow beats a Deny.
Conflicting role rules land on the stricter effect only when they tie on specificity and Priority.
- If a field is both denied and transformed at the same level, Deny wins: the field is dropped or refused,
not masked.
- Every store and the admin endpoint call `SealedFields.Refuse` before saving a rule. It rejects a rule
whose path and features overlap a sealed attribute, provided the store can resolve the entity type.
Resolution enforces the ceiling regardless.
The resolver, fragment and provider types are listed in section 21.
---
## 17. Enforcement by tier and dry run
Under `Strict` outside a dry run, a path that exists on the type but names no value the query can compute is refused
as an unknown name is (3.3.0), with the clause's own code and `FieldPath` `"*"`, in every clause the database has to
compute: a filter, an order, a grouping key, an aggregated field, and a filter or an order inside a `Segment`.
`Selects` is not one of them. A projection is the last thing the provider builds, and EF Core evaluates that one on
the client when it cannot translate it, so `Selects` naming such a member returns its value exactly as before.
What decides:
```
Source of the rows What the set of producible members is read from Name.IsEmpty
entity query EF Core model of the queried type: columns
(shadow included), owned, complex, navigations refused
Select(...) before ApplyPolicy building the row that initializer's assignments, at every level,
both branches of a conditional included refused
... a member assigned from something else: a nothing; the assignment is not one this shape
method call, a captured value, a subquery, reads
two branches building it two ways left alone
... a member it copies whole: Name = role.Name the model, beneath the copied member refused
rows in memory (EnumerableQuery) nothing; the getter runs runs
beneath a column, converted or not nothing; the converter decides left alone
a framework member: Length, Year, HasValue nothing; the provider translates it runs
a source the library cannot read into nothing left alone
a provider in front of EF Core nothing; it rewrites what EF Core cannot
translate left alone
a column only a subtype maps, through the base the queried type's model, which EF Core
translates against refused
a projection another provider ran nothing; its rules are its own left alone
```
- The `Length, Year, HasValue` row says whether such a path is computable, and it is. Its *policy* is the
member's own since 3.3.0 (section 14), so it runs only where that member may be used for that feature.
- A projection is read only as far as its initializer can be read: a nested initializer, a member copied from the
entity, a value built and left empty, and a conditional over those, a null branch included. A member assigned
nothing but a null is left alone. Past `MaxComplexDepth` (eight levels) it stops reading and stops speaking. Such
a path reaches the provider and fails there, as before 3.3.0.
- `left alone` is not a promise that the path runs: the policy does not refuse it, so it behaves
exactly as it does unguarded. `Name.IsEmpty` beneath a column mapped through a value converter
still fails inside the provider.
- An unmapped getter on the entity itself (`Display => $"{Code}:{Id}"`) is refused for the same reason.
- The rule is EF Core's own provider's — that exact type, from EF Core's own assembly. A provider that wraps EF
Core — LinqKit's `AsExpandable()`, DelegateDecompiler's `Decompile()` — rewrites what EF Core cannot translate, so
its rows are left alone, over a projection and over an entity alike. Every other provider is left alone for the
same reason turned around, a host's own through `ReplaceService<IAsyncQueryProvider, …>` included: the library
cannot tell one that rewrites from one that passes straight through, and refusing on that guess would take back a
query the rewriting host answers today. A rewrite inside EF Core's own pipeline — a member-translator plugin, a
replaced query preprocessor — leaves EF Core's own provider in place, so such a member is refused.
- A projection that does not build its rows with an object initializer — an anonymous type, a constructor with
arguments — says nothing about which member each value sets, so no member of such a row is refused here.
- A member assigned through a sequence operator — `Lines = o.Lines.ToList()`, `o.Lines.Where(…).ToList()`, a
subquery building rows of its own — is left alone as well: what the row holds is not what the navigation holds,
and reading it as the navigation would refuse a member the row carries.
- A row the library itself projected reads like any other: the core's typed `Select` null-guards every nested node
it builds, and both branches of that guard are read, so `Select` composed and then filtered refuses what the bare
handle refuses.
- The refusal raises no `[DwAudit]` event, as an unknown name raises none: no field was read and the refusal names
none. `AuditRefusals` records it, and so does the trace.
- `Convenience` and a dry run are unchanged: the provider throws, as it does unguarded.
- The rule is the model's: a member it maps nowhere is refused, whatever a provider extension could compute.
- The trace records it: `the member exists on the type and the query cannot compute it, so it is refused as an unknown name is`.
The count caps are checked first, before any name is resolved (3.1.0). Names are then resolved to canonical paths,
`MaxNavigationDepth` is checked, and the cost budget is checked before field policies — except under `Strict` outside
dry run, where it is checked after them (3.1.0). Nothing is clamped.
```
Request Convenience Strict Error code
WHERE on a field denied for Where (any depth, any set) throw throw FieldDeniedForWhere
WHERE with an operator the field does not allow throw throw OperatorNotAllowed
HAVING through a key or alias of such a field or operator throw throw one of the two above
ORDER BY a field denied for Order (also a summary key/alias) drop throw FieldDeniedForOrder
SELECT a field denied for Select drop throw FieldDeniedForSelect
SELECT a navigation with a denied field beneath it allowed leaves throw FieldDeniedForSelect
SELECT a navigation that cannot be narrowed around a denied
field: its key a.Id is denied, or a path through it is one
the core cannot project (3.2.0) throw throw FieldDeniedForSelect
SELECT a.b when the key a.Id is denied throw throw FieldDeniedForSelect
SELECT a member that can carry a denied field no path names:
past four segments, in a framework generic, on a subtype,
or unasked under a "*" deny (3.2.0) allowed leaves, throw FieldDeniedForSelect
or throw where it
cannot be narrowed
every requested SELECT dropped throw - AllSelectsDenied
no Selects while a denied field can reach the result (3.2.0) allowed members allowed members
GROUP BY a denied field throw throw FieldDeniedForGroup
AGGREGATE a denied field, or a transformed field lacking
AllowAggregate on a stage throw throw FieldDeniedForAggregate
any clause on a path the attribute walk cannot name:
Salary.Value beneath a framework-typed member, or a
path past four segments with MaxNavigationDepth
raised (3.3.0) the member's same the member's own code
own decision
SELECT a path beneath a transformed member, Bonus.Value:
no member there to apply the chain to (3.3.0) drop throw FieldDeniedForSelect
SELECT a transformed member past four segments (3.3.0) transformed transformed
GROUP BY or AGGREGATE a path of either kind whose member
is transformed (3.3.0) throw throw FieldDeniedForGroup /
FieldDeniedForAggregate
field denied for Segment used anywhere in a Segment throw throw FieldDeniedForSegment
Segment condition on a field denied for Select allowed throw FieldDeniedForSegment
any other field refusal inside a Segment (3.1.0) the rows above throw Strict: FieldDeniedForSegment
[DwRequireWhere] not satisfied throw throw RequiredFilterMissing
ContextValue missing or null throw throw MissingContextValue
MaxConditions / MaxConditionDepth / MaxConditionSets /
MaxConditionValues / MaxAggregates (3.1.0) /
MaxOrderFields / MaxPageSize /
MaxNavigationDepth exceeded throw throw CapExceeded
total cost above MaxQueryCost throw throw QueryCostExceeded
getQueryString: true SQL returned throw QueryStringDenied
a path that matches nothing on T (3.1.0) throw throw Convenience: ConditionMustHasValidFieldName
Strict: that clause's FieldDeniedFor* code
DefaultOrder field the caller may not order by (3.1.0) left out left out
DefaultOrder field denied for Segment, in a Segment (3.1.0) left out left out
```
- "drop" removes the entry and records `Dropped`. A Strict throw records `Denied`.
- The guarded composable `Order(...)` and `Select(...)` drop and throw the same way.
- "allowed leaves": when `Selects` names a navigation with a denied field beneath it, it is replaced by the allowed
leaf paths beneath it. A navigation with nothing denied beneath it is kept as written, unless a transform beneath it
lands on a property with no setter, which the outbound walk could not write back: it is then narrowed around that
property (3.2.0).
- The fields beneath are read the way the attribute walker reads them (3.2.0): through any collection type, so a
member typed `IReadOnlyList<T>` or an application's own collection no longer hides its denied fields, and no
deeper than the walker's four segments. The providers' fragments are asked too, so a denied property with no
setter and a rule on a path reached through a cycle count (3.2.0).
- A named member can also carry a field denied for Select that no path names (3.2.0): deeper than four segments,
inside a framework generic such as `Dictionary<string, T>`, or declared by a subtype of the member's type (a
derived entity, a subclass, an interface's implementation). What it can carry is read from the source. On an
entity it is read from the EF Core model, so only what loads counts: the navigation's columns, a converted one
included, its owned chain at any depth, the navigations beneath it that an include, an automatic include or a
lazy loader fills, and each member the model does not map, read as its type, since its getter can hand out what
EF Core loaded; for its type and every type the model derives from it. On a projected row it is the type the
initializer constructs the member as, when it says (`Contact = new ContactRow { … }`), and otherwise the
member's type and every loaded subtype of it, as on a row in memory. Under a policy with a `"*"` deny, a path
the walk never asks about (past four segments, with no setter, or on a subtype) is a denied one unless the
policy names it; around a cycle, where the paths never end, it always is.
- Such a member is refused with `FieldDeniedForSelect` under Strict. Under Convenience it is narrowed to the
allowed leaves where the core can narrow the path's first member, which builds the declared type and so drops a
subtype's fields; a path that names a framework generic itself narrows to nothing and is dropped. Where the core
cannot narrow it (a column, complex or JSON member at the top of T, or a member of a row in memory) it is
refused in both tiers.
- A narrowing that cannot be built as gated is refused with `FieldDeniedForSelect`, in both tiers (3.2.0). The
core's typed projection adds the key (`Id`) of every nested node it builds, so a narrowing whose nodes carry a
denied key would return it; a path through a collection the core does not unwrap fails its validation; and a
column, complex property or JSON-stored member, a member of a row in memory, or one a projection builds some way
the core cannot narrow, cannot be narrowed at all.
- A navigation named through another, `Main.Lead`, gates the key of every node it passes through, which the
builder adds, as a dotted path to a value always did (3.2.0). A denied one refuses the projection.
- "allowed members" (3.2.0; "allowed scalars" before) applies when `Selects` is null or empty and a field is denied
for Select where its value can reach the result. It runs for a whole-`Filter` terminal and for a Segment, not for a
single-clause composable call.
- A denial at the top of T always counts, whatever the member holds: a scalar, a blob, a list, an owned object, a
JSON column (3.2.0; 3.1.0 asked only about simple members, so a denied `byte[]` or owned member came back).
- A denial beneath a member counts when its value can reach the result. A rule may spell its path in any letter
case.
- On an entity: beneath a column, an owned or complex member, or a navigation something loads. That is an
`Include` or `ThenInclude` on the query, an automatic include, or a lazy loader: proxies, an injected
`ILazyLoader`, a loader delegate or `ILazyLoader` the constructor takes, kept in a field or any property, the
asynchronous loader delegate EF Core 7 added, or an injected `DbContext`, any of which fills a navigation
after the query. Every navigation counts as loaded when the library cannot read which the query loads: an
include in a form it cannot read, an include off the query's own chain (on a join's inner source, say), and a
chain that reaches its rows through anything but the root's own rows (`Select(o => o.Customer)`, a
`SelectMany`, a `Join`, a `GroupBy`) when it also has an include, which EF Core applies from the root to the
entities it reaches, or when one of its lambdas hands its rows an object: one it builds (a projection behind an
identity `Select`, or an object built inside an anonymous row or a conditional), which loads whatever it
assigns; one an application's method returns from what the lambda gives it; or one it captured, another query
with its own include or projection, or an object in memory. A call that reads nothing of the lambda's and returns
a query or an expression (a specification, a repository's query, `FromSql`, a context's `Set` through an
interface) is evaluated as EF Core evaluates it, and what it returns is read; a context's own query function is a
query root; an anonymous object that only carries what the rows hold (query-syntax range variables, a
composite key) builds nothing; and what only feeds a predicate or a key is a value and hands a row nothing. Such
a chain with none of these is read from the model. A denial beneath a navigation nothing loads never leaves the database
and needs no projection,
so a connected model is read as it was in 3.1.0. A member EF Core does not map counts as loaded: its getter can
hand out a mapped field or a private navigation, so its type is read whole.
- On a row a projection builds: beneath a member its initializer assigns. A constructor with arguments counts
every member as assigned; an initializer after it still says what its own bindings hold.
- On a row in memory: beneath any member.
- So does a member whose value can hold a field denied for Select that no path names (deeper than the walker's
four segments, inside a framework generic such as `Dictionary<string, T>`, or declared by a subtype of its
type), read as for a named member above, so on an entity only what loads counts. Under a policy with a `"*"`
deny, so does a member whose value can hold a path the walk never asks about and the policy does not name.
- A member that can hold an object of any type, one typed `object`, a framework interface such as `IComparable`,
an unbound type parameter, or a collection that is not generic (`IEnumerable`, `ArrayList`, `Array`, an
application's own), asks for nothing on its own: the policy cannot see into it whether or not a projection is
built. An application's own such collection still has its own members read, as any type's are.
- A row can be a subtype of T. On an entity, each member a type the model derives from T declares, and what loads
beneath it, counts as one of T's own would; on a row in memory, each member any loaded subtype declares; on a
row a projection builds, each member the type its initializer constructs declares below T. The projection builds
T, so it leaves them all out, recorded as `Dropped` with a reason starting `left out: a type derived`.
- A subtype is any type loaded outside the framework's own assemblies that derives from the type or implements it:
an open generic one, `Tagged<T> : Creature`, and an application's subclass of a framework class, an `Exception`
or a `Stream`, included. A rule on a path through a subtype's member (`Org.Swift`, where `Swift` is the bank's),
or through one of two members whose names differ only in letter case, counts beneath a member as a rule on the
declared type's own path does.
- The denials beneath a member come from the providers' fragments as well as from walking the type, so a denied
property with no setter, a rule on a path reached through a cycle, and a rule deeper than the walk all count.
- A forced scope beneath a member asks for no projection on its own: it filters the rows that hold the member, as it
always has. When a projection is needed anyway, the member is left out whole (below).
- The projection is what an unguarded call would return, less what the policy withholds:
- every allowed member holding a value, a simple type (primitive, enum, `string`, `decimal`, `DateTime`,
`DateOnly`, `TimeOnly`, `DateTimeOffset`, `TimeSpan`, `Guid`) or a collection of one (`byte[]`, `string[]`,
`List<string>`), that the source carries: every one a projection assigns, every one of a row in memory, and
every one EF Core maps on an entity. A value EF Core does not map is left out: computing it would make EF Core
read the whole entity, the denied columns included, and it holds only its initial value anyway;
- every allowed member holding an object or a list of them that the source carries: a member a projection's
initializer assigns, and an entity's columns (converted or JSON), owned and complex members. It is kept whole
when nothing beneath it is denied, nothing its value can hold is denied, it cannot hold an object of any type
(asked of a projected row, a row in memory, and an entity's column a value converter hands back, directly or
inside a complex property: what EF Core materializes itself never holds one), under a
`"*"` deny every path beneath it the walk skips is one the policy names, no forced scope is beneath it, and no
transform beneath it lands on a property with no setter;
- otherwise it is narrowed to the allowed leaves beneath it, to four segments, as a caller naming it would get,
when the core's narrowing translates: an object the projection's initializer builds, a list a subquery reads
into a type the core can bind (not an array or a set), a navigation that is neither complex nor stored as JSON,
or an entity's owned member not stored as JSON. The narrowing builds the member's declared type, so a
subtype's fields are dropped. A leaf that can hold what the policy cannot name is left out;
- otherwise it is left out whole, recorded as `Dropped` on `Select` with a reason starting `left out whole`: a
scope forced beneath it; a column, complex property or JSON-stored member, which EF Core reads whole; one the
projection builds some other way, by a constructor with arguments, a conditional or an unassigned member; a
denied key the core's projection would add back; a path the core cannot project; a node type the core cannot
construct (an interface, an abstract class, one with no public parameterless constructor); nothing beneath it
left to select;
- Never kept: an entity's navigation, included or not, since projecting it would load it (under Convenience, name
it in `Selects` to get it narrowed; under Strict, name its allowed fields); an object held by a row in memory,
since a kept object is the caller's own and a transform would change it in place; a member with no setter; a
member named with one of the parser's words. Each one the unguarded call would have returned, an included
navigation or an object in memory, is recorded as `Dropped` with a reason starting `left out:` (3.2.0).
- A `Select` handing back an entity, `Select(o => o.Customer)`, builds no row and is read as an entity query, with
every navigation counted as loaded when the query has an include (above).
- A narrowed reference that is null in the source comes back as an empty object, as it does for a caller's own
dotted `Selects`.
- A narrowed member carries every allowed leaf beneath it. An entity reached beneath it has its own navigations
projected and so loaded, which the source may not have included, exactly as `Selects` naming the member does.
- Each denied field whose value can reach the result, at the top or beneath, is recorded as `Dropped` on `Select`.
- It never throws for the denied field. It throws `AllSelectsDenied` only when no field is left. A typed terminal
projects into T, so a T with no public parameterless constructor, an abstract one among them, fails there with
`SelectTypeMustHaveParameterlessConstructor`; the dynamic terminals build their own class.
- With nothing counted, `Selects` stays null and the query is the one an unguarded call runs.
- A path the attribute walk cannot name takes the policy of the member it reads (3.3.0, section 14): one that
continues beneath a member whose type the framework declares — `Salary.Value` and `Salary.HasValue` on a
`decimal?`, `Secret.Length` on a `string`, `Born.Year` on a `DateTime`, `Bag.Count` on a dictionary,
`Lines.Count` on an application's own collection class — and, where a host raised `MaxNavigationDepth`, one
longer than the walk's four segments.
- The deny effects per feature, the `[DwOperators]` restriction, the `[DwCost]` weight and the audited features
come from the member; its alias, its required filter, its forced scope and its descriptive facts do not. A
rule naming the sub-path itself still applies alongside, and `Explain` names the member's fragment as the
source.
- Where the member it reads is transformed, a path beneath it has no member to apply the chain to, so every
way it hands a value back is refused: `Select`, `Group` and `Aggregate`, with a `Selects` entry dropped under
`Convenience` as any field denied for Select is. A transformed member past the walk is itself a member, so
`Selects` naming it returns it transformed (section 18); only `Group` and `Aggregate` are refused there,
because a summary's own transform finds a generated row's columns by the type's list, which stops at four
segments. `Where` and `Order` follow the member's own decision and run on the stored value, as they do along
a named path.
- A member only a subtype of the navigated type declares is not such a path: it is decided by the fragments
naming it, as before.
- The member a path reads is found whatever the providers are, an attribute and a store rule alike. The
attributes past the walk's depth are read only by a resolver that reads attributes.
- A path that matches nothing on `T` (3.1.0):
- Convenience, and dry run in either tier: validation throws `LogicException`
`ConditionMustHasValidFieldName` before any policy decision, as it does unguarded.
- Strict, outside dry run: the name is kept and gated as a field denied for every feature, at the step a
denial is raised, after the caps. It throws the code a `[DwDenied]` field throws in the same clause:
`FieldDeniedForWhere`, `FieldDeniedForSelect`, `FieldDeniedForOrder`, `FieldDeniedForGroup` or
`FieldDeniedForAggregate`, and `FieldDeniedForSegment` in any clause of a Segment.
- A name padded with dots or blank segments is normalized as a real path is, empty segments dropped (3.1.0):
`NoSuchColumn....`, `....X` and `. . . . X` are gated as `NoSuchColumn` and `X`, and refused as a padded real
field is. Kept whole, it would fail `MaxNavigationDepth` where a padded real field passes.
- It covers condition fields, `Selects`, `Orders`, `GroupBy.Fields` and `AggregateBy.Field`. `Having` and
summary `Orders` name aliases and group keys, which validation still checks. A null or blank name is
refused the same way in both tiers.
- Under `Strict`, field and cap refusals name no field (3.1.0). Every `FieldDeniedFor*` refusal carries
`FieldPath = "*"`, `RuleId = null` and `SourceOrigin = null`, for a denied field, an alias and an unknown name
alike, so their messages are identical. Every `CapExceeded` refusal carries `FieldPath = "*"` as well, and
keeps its `SourceOrigin`. `MissingContextValue` carries `FieldPath = "*"` and `SourceOrigin = null`, naming
neither the scope's column nor the context key it reads. `OperatorNotAllowed` and `RequiredFilterMissing` still name
their field. `AmbiguousFieldName` is not raised under this tier at all since 3.3.0: an ambiguous name is
refused as an unknown name is.
- Inside a `Segment` every field refusal is `FieldDeniedForSegment` with `Feature = Segment`, whichever clause
refused it: a condition in any set, an order, a select, or taking part at all (3.1.0). Answered by clause, a field
denied for every clause but not for `Segment` would say `FieldDeniedForOrder` where a name that matches nothing
says `FieldDeniedForSegment`. The Convenience tier keeps per-clause codes, and filters and summaries keep them in
both tiers.
- The cost budget is checked after every field gate (3.1.0). A weighted field the caller may not use is refused
as denied before its weight counts, exactly as a name that matches nothing is; an allowed weighted field still
gets `QueryCostExceeded`.
- The trace keeps the real path and reason; an unknown name is recorded as `Denied` with reason
`names nothing on <TypeName>`. A refusal event under `AuditRefusals` names the field too (section 22).
- Convenience names the field as the caller wrote it, with `RuleId` and `SourceOrigin`.
- "left out" (3.1.0): a guarded query that sends no orders takes the type's `DefaultOrder` less every field
this caller may not order by, in both tiers, and in a `Segment` less every field this caller may not use in a
segment. Each field left out is recorded as `Dropped` on `Order`, with a reason starting
`left out of the default order`, and never refused. Dry run keeps it and still records it (section 13).
- These throw regardless of tier and dry run: `GroupTooSmall`, `AmbiguousGroupKey`,
`CapExceeded` from a full audit buffer, `MissingHashSalt`, `MissingTokenVault`,
`TransformRequiresMaterialization`, `StoreUnavailable` and `PolicyContextNotPrepared`. `PolicyRequired`
is raised on unguarded calls only. `AmbiguousFieldName` throws under `Convenience` and in a dry run; under
`Strict` the name is refused as an unknown one instead (3.3.0).
- Dry run is on when `DwPolicyOptions.DryRun` or `DwPolicyContext.DryRun` is true. It records every other
throw and drop in the trace, applies none of them, and leaves the clause as written. In dry run:
- no forced predicate is injected and no projection is synthesized;
- the group-size column is added without its predicate;
- a default order keeps the fields the caller may not order by;
- a path that matches nothing fails validation with `ConditionMustHasValidFieldName`, in both tiers;
- output transforms still run.
- a dry run therefore returns what the policy would have withheld: denied fields, and rows a forced
predicate would have excluded. Only the transforms still apply.
---
## 18. Transforms
Transforms run in memory after rows materialize, on the returned instances. Every guarded query runs
`AsNoTracking`. WHERE, ORDER BY, GROUP BY and aggregates run in SQL on the real values.
```
Order TransformKind Attribute Output Valid member type
alone Default [DwDefault] constant or type default, converted to member any
1 Mutate [DwMutate] whatever Transform returns any it fits
2 Generalize [DwGeneralize] Round/Truncate/DatePart keep the value's type; numeric; DateTime,
Bucket returns string DateOnly, DateTimeOffset;
Bucket: string
3 Format [DwFormat] string string
4 Mask [DwMask] string; Null strategy returns null string; Null: reference or T?
5 Truncate [DwTruncate] string string
```
- `[DwDefault]` short-circuits the chain: no other stage runs. Declaring it with any other transform
attribute is a `ValidateModel` error.
- The chain's result must be assignable to the member's type, and null only fits a reference type or
`Nullable<T>`. Otherwise the query throws `InvalidOperationException`, not `PolicyException`.
- So a mask, format, truncate or `Bucket` on a `decimal` or `DateTime` member fails.
- On such a member, use `[DwGeneralize]` (not `Bucket`) or `[DwDefault]`.
- Mask and Truncate read the value as text: a `string` as-is, an `IFormattable` via
`ToString(null, CultureInfo.InvariantCulture)`, anything else via `ToString()`. Null stays null, except
under `Fixed`.
- Only paths the result carries are transformed: every path when `Selects` is null, otherwise the
selected paths and every path beneath a selected navigation.
- Collections and navigations are followed, and null navigations are skipped.
- An object shared by several rows is transformed once.
- The rows are also walked by run-time type (3.3.0), so a member that declares a transform attribute and was not
transformed along a path the policy names is transformed by its own attributes, exactly once: an object reached
both ways is not transformed twice. The walk of the declared types names the paths of four segments, and a
value can sit where none of them goes — a `[DwMask]` member five segments down an included or in-memory graph
(`B.C.D.E.Card`, while `B.C.D.Pin` four segments down was masked), a masked member only a subtype of the row's
type declares (`Dog.Chip` on rows typed `Animal`, in memory or in a TPH hierarchy), a masked member of an
object a dictionary holds, and the far side of a cycle — with no `Selects`, with a navigation named whole in
`Selects`, and in a dynamic projection holding a real object. Each came back exactly as stored, at the default
caps, under `Strict`.
- Only members that declare a transform or an audit for Select, or that can lead to one, are read, so a
navigation whose type can reach neither is never touched and a lazy loader behind it is not woken. A model
that declares neither anywhere pays for no second pass at all.
- The same pass reports each audited member it meets where the policy names no path to it, which the terminal
records (3.3.0, section 22).
- The transform is the member's own attributes. No rule can speak to such a member, since no path names it,
which is the same answer as "no runtime rule can unmask a field" (section 32). A resolver built over no
`AttributePolicyProvider` reads no attribute here either.
- It carries only what the result carries, as the first pass does; it runs in a dry run, as transforms always
have; and a transformed member with no setter fails the query with `InvalidOperationException`, as one along
a named path does.
- The trace records the path with its stages and the note
`(declared on the member; no path of the policy names it)`.
- Still a limit: a member typed `object`, or a collection that is not generic, says nothing about what it
holds and is not read into.
- A transform is applied to a member. Beneath that member the value is another type — `Bonus.Value` is a decimal
where `Bonus` is the member that is rounded — so there is no member to apply the chain to, and `Select`, `Group`
and `Aggregate` on such a path are refused (3.3.0): `FieldDeniedForSelect`, `FieldDeniedForGroup`,
`FieldDeniedForAggregate`, with a `Selects` entry dropped under `Convenience` as any field denied for Select is.
- A transformed member past the walk's depth is itself a member, so a row that carries it carries it transformed
(3.3.0). `Selects` naming it returns it transformed, in a typed projection and in a generated row alike: the
chains of the members a projection names past the walk are handed to the outbound walk beside the type's own
list and applied by path. A grouping key and an aggregated field are columns of a generated row, which a
summary's own transform finds by the type's list, and that list stops at four segments, so those two are
refused: `FieldDeniedForGroup`, `FieldDeniedForAggregate`.
- `Where` and `Order` run on stored values wherever a member is reached from, so on both kinds of path they
follow the member's own decision, as they do along a named path.
- A transformed member with no setter, or a path missing on the runtime type, throws
`InvalidOperationException`.
- Segment results are transformed like Filter results.
- In a summary, each transformed grouping-key column and each aliased aggregate of a transformed field
gets that field's chain.
- The trace gets one `PolicyDecision` per transformed path. Its feature is `Select`, or `Aggregate` for
summary columns, and its reason names the stages, e.g. `"Generalize then Mask"`.
- Aggregating a transformed field throws `FieldDeniedForAggregate` unless every declared stage has
`AllowAggregate = true`. That includes a short-circuiting `[DwDefault]`, and the check overrides every
Allow, sealed ones included. The field's own floor is the largest `MinGroupSize` among its stages.
- On the guarded handle, `SelectDynamic`, `FilterDynamic`, `Group` and `Summary` throw
`TransformRequiresMaterialization` on a type whose values are transformed on the way out for this caller: one
the policy names a transformed path on, and, since 3.3.0, one a row of which can hold a member that declares a
transform attribute anywhere in what the type can reach, which only a resolver that reads attributes is asked.
Until 3.3.0 a type whose only transforms sit off the named paths — a member only a subtype declares, one five
segments down, one of an object a dictionary holds — was handed the query, and its rows came back exactly as
stored. With no named column to list the refusal names the clause, `FieldPath` `"*"`, in both tiers. A type
nothing transforms anywhere still gets its query.
- `AsUnguardedQueryable()` output is never transformed.
- Transforms also run in dry run.
### DwMask
```
Strategy Output for non-null text v null input
Full MaskChar repeated v.Length times (8 times when PreserveLength = false) null
Partial v[..KeepStart] + one MaskChar per hidden char + v[^KeepEnd..] null
if KeepStart + KeepEnd >= v.Length: Full
Email local[0] + n MaskChar + "@" + domain[0] + m MaskChar + domain from its last "." null
PreserveLength: n = max(local.Length - 1, 1), m = max(lastDotIndex - 1, 1)
PreserveLength = false: n = m = 3
Full when: no "@", "@" first or last, no "." after the domain's first char, "." last
Phone every digit masked except the last K (K = KeepEnd, or 4 when KeepEnd = 0) null
non-digits kept; K or fewer digits: Full
Regex Regex.Replace(v, Pattern, Replacement ?? "", RegexOptions.None, 1-second timeout) null
Fixed Text Text
Hash lowercase hex of HMAC-SHA256(key = UTF-8 HashSalt, data = UTF-8 v), 64 chars null
Null null null
Tokenize TokenVault.GetOrCreate(TokenScope ?? field path, v); built-in vaults: 32 lowercase hex null
```
```
Partial KeepEnd = 4 "999999991234" -> "********1234"
Partial KeepEnd = 4 "1234" -> "****"
Partial KeepStart = 2, KeepEnd = 2 "abc" -> "***"
Email "ada@example.com" -> "a**@e******.com"
Email PreserveLength = false "ada@example.com" -> "a***@e***.com"
Email "ada@example" -> "***********"
Phone "+964 770 12 1234" -> "+*** *** ** 1234"
Phone "123" -> "***"
Regex Pattern = @"\d", Replacement = "#" "A1-23" -> "A#-##"
Fixed Text = "[REDACTED]" null -> "[REDACTED]"
Full "Secret123" -> "*********"
Full PreserveLength = false "Secret123" -> "********"
Email "john.doe@example.com" -> "j*******@e******.com"
Phone "07701234567" -> "*******4567"
Hash any value -> 64 lowercase hex characters
Tokenize any value -> 32 lowercase hex characters
```
- `PreserveLength` affects `Full`, `Email`, and the Full fallback of every strategy. Partial's hidden
middle and Phone's digits are always masked one character for one.
- Refused when the stage is built (a `ValidateModel` error, or `ArgumentException` at query time):
- a negative `KeepStart` or `KeepEnd`;
- `Regex` with a null or empty `Pattern`;
- `Fixed` with `Text = null` (`""` is allowed);
- a blank `TokenScope`.
Invalid regex syntax surfaces only when a query runs.
- `Hash` without `DwPolicyOptions.HashSalt` throws `MissingHashSalt`. The salt is checked before the value,
so even a null value throws.
- The `HashSalt` setter refuses 1–15 characters. Blank means not configured.
- The same value under the same salt always gives the same digest.
- `Tokenize` without `DwPolicyOptions.TokenVault` throws `MissingTokenVault`, also checked before the
value. A null value is not given a token.
- The default token scope is the canonical path relative to the queried entity, e.g. `"NationalId"` or
`"Contact.NationalId"`. The entity type is not part of it. The same scope and value always give the same
token.
- `DwToken.New()` returns 16 cryptographic random bytes as 32 lowercase hex characters.
- `DwToken.KeyFor(string scope, string value)` returns `scope + ":" + lowercase hex SHA-256(UTF-8 value)`.
A vault given a key stores under `DwToken.KeyFor(scope, value, key)` instead (3.3.0), which is
`"hmac:" + scope + ":" + the lowercase hex HMAC-SHA256`, so a copy of the store gives no value back.
There is no reverse lookup either way. Section 24.
- `InMemoryTokenVault` lasts for the process and is unbounded. It exposes `Count` and `Clear()`, and a
restart issues new tokens. It draws a key of its own (3.3.0), so its mappings are keyed with nothing to
configure.
- `Hash` and `Tokenize` both preserve equality, so the column still groups and joins, given the same salt
or the same scope.
### DwGeneralize
```
Mode Requires Output Valid on
Round Step > 0 Math.Round(v / Step, MidpointRounding.AwayFromZero) * Step, same type numeric
Bucket Step > 0 "{lo}-{lo+Step-1}" with lo = Math.Floor(v / Step) * Step, invariant string holding a number
DatePart Part start of the Year / Quarter / Month / Day DateTime, DateOnly,
DateTimeOffset
Truncate Decimals >= 0 Math.Truncate(v * 10^Decimals) / 10^Decimals, same type, no rounding numeric
```
```
Round Step = 10000 118500 -> 120000
Round Step = 10 23 -> 20 -14 -> -10
Bucket Step = 10 27 -> "20-29" 30 -> "30-39" -1 -> "-10--1"
Truncate Decimals = 2 33.199999 -> 33.19
DatePart, 1987-06-15 Year 1987-01-01 Quarter 1987-04-01 Month 1987-06-01 Day 1987-06-15
```
- A null value stays null. Arithmetic runs in `decimal` and converts back to the value's own type, so an
`int` stays an `int`.
- `DatePart` keeps a `DateTime`'s `Kind`, returns a `DateOnly` for a `DateOnly`, and keeps a
`DateTimeOffset`'s `Offset`.
- Round, Bucket and Truncate use `Convert.ToDecimal(value, InvariantCulture)`. DatePart uses
`Convert.ToDateTime` for anything that is not a `DateOnly` or `DateTimeOffset`.
- A value of the wrong kind throws when the query runs. `ValidateModel` checks only Bucket's
string-member rule.
### DwFormat, DwTruncate, DwDefault
- `[DwFormat]` renders an `IFormattable` value with `value.ToString(Format, CultureInfo.InvariantCulture)`.
Any other value uses `ToString()` and ignores `Format`, so a string passes through unchanged.
- On a string member it therefore changes only a value that `[DwMutate]` or `[DwGeneralize(DatePart)]`
turned into a non-string. `Round` and `Truncate` convert back to string.
- A blank format is refused.
- `[DwTruncate]` leaves a null value, or a value of at most `Length` characters, unchanged. A longer value
becomes `v[..Length]` followed by `Ellipsis` when `Ellipsis` is not null.
- `Ellipsis` is not counted in `Length`: the output can be `Length + Ellipsis.Length` characters.
- `Length = 0` is allowed; a negative length is refused. Truncation runs after the mask.
- `[DwDefault]` and `[DwDefault(null)]` give the member type's default: `0`, `false` or a default struct
for a non-nullable value type, otherwise null.
- `[DwDefault("v")]` converts the text to the member type, after unwrapping `Nullable<T>`:
- `string` → `"v"` unchanged;
- `""` on any other type → the type's default;
- enum → `Enum.Parse(ignoreCase: true)`;
- `Guid` → `Guid.Parse`;
- `DateOnly`, `TimeOnly`, `DateTimeOffset`, `TimeSpan` → `.Parse` with `InvariantCulture`;
- anything else → `Convert.ChangeType` with `InvariantCulture`.
A constant that cannot convert is a `ValidateModel` error; at query time the parse exception is thrown.
### DwMutate
```csharp
public interface IValueTransformer
{
object? Transform(object? value, DwTransformContext context);
}
public readonly struct DwTransformContext : IEquatable<DwTransformContext>
{
public DwTransformContext(object entity, string fieldPath, DwPolicyContext policy); // null -> ArgumentNullException
public object Entity { get; } // object holding the member: nested owner for "Contact.Email", generated row in a summary
public string FieldPath { get; } // canonical path, never the alias
public DwPolicyContext Policy { get; } // the caller: Subjects, Identities(DwSubjectKind), Purpose, TryGetValue
}
```
- One instance per transformer `Type` is cached for the life of the process. It comes from
`DwPolicyOptions.Services?.GetService(type)`, falling back to `Activator.CreateInstance(type)`, which
needs a parameterless constructor. The implementation must be stateless and thread-safe.
- `[DwMutate]` runs first and sees the real value. Exceptions it throws are not caught, so the query fails.
- A type that does not implement `IValueTransformer` throws `InvalidOperationException` at query time, and
is a `ValidateModel` error.
- The transformer's result must fit the member. `ValidateModel` skips the output-type check for any chain
containing `[DwMutate]`.
- It is available from source only: a stored rule cannot carry a Mutate stage.
### Stage classes
For runtime rules and custom providers. Each constructor throws `ArgumentException` for the same
misconfigurations as the matching attribute.
```
TransformStage (abstract) Kind:TransformKind AllowAggregate:bool MinGroupSize:int
MutateStage(Type transformer, bool allowAggregate = false, int minGroupSize = 0)
GeneralizeStage(GeneralizeMode mode, int step = 0, DatePart part = DatePart.Year, int decimals = 0,
bool allowAggregate = false, int minGroupSize = 0)
FormatStage(string format, bool allowAggregate = false, int minGroupSize = 0)
MaskStage(MaskStrategy strategy, int keepStart = 0, int keepEnd = 0, char maskChar = '*',
bool preserveLength = true, string? pattern = null, string? replacement = null, string? text = null,
bool allowAggregate = false, int minGroupSize = 0, string? tokenScope = null)
Replacement:string (a null replacement becomes "")
TruncateStage(int length, string? ellipsis = null, bool allowAggregate = false, int minGroupSize = 0)
DefaultStage(string? value, bool hasValue, bool allowAggregate = false, int minGroupSize = 0)
ValueTransform(MutateStage? mutate = null, GeneralizeStage? generalize = null, FormatStage? format = null,
MaskStage? mask = null, TruncateStage? truncate = null, DefaultStage? @default = null)
Mutate Generalize Format Mask Truncate Default AllowsAggregate MinGroupSize IsEmpty
HasConflictingDefault Stages (in run order; a Default runs alone) Action
```
---
## 19. Group floor and transformed summaries
**The floor is on by default.** `Caps.MinGroupSize` ships at 5, so a guarded summary drops every group with fewer
than five rows, and nothing in the answer says a group was dropped. Right for anonymised reporting, surprising for
an operational count. `Caps.MinGroupSize = 1` switches it off, deliberately. An unguarded summary is never floored.
```
effective floor = max(Caps.MinGroupSize, MinGroupSize of the chain of each AggregateBy.Field)
Caps.MinGroupSize unset -> DwCaps.DefaultMinGroupSize = 5 (IsMinGroupSizeSet = false); 1 = no global floor
```
- The floor applies to every guarded `Summary` that has a `GroupBy`, once it is above 1, whether or not
any field is transformed. Only aggregated fields raise it; grouping keys do not.
- It is added after gating and is not charged against cost:
- `AggregateBy { Aggregator = Count, Alias = "__dwGroupSize" }` is appended.
- `Having` becomes `And [ __dwGroupSize >= floor ]`, with the caller's `Having` as a subgroup.
- The database removes the small groups, so `Data`, `TotalCount` and `PageCount` cover only the groups
that remain.
- A summary whose every group is too small returns an empty `Data`, never an error.
- The trace gets one decision: `__dwGroupSize`, `Aggregate`, `Dropped`,
`"groups below the group floor of {floor} are excluded by the query"`.
- `ToList`/`ToListAsync(Summary)` return each row as an `ExpandoObject` without `__dwGroupSize` whenever
the floor applied.
- The composable `Group(GroupBy)` and `Summary(...)` apply the floor as well, and project the column back out,
so no caller receives the library's own count. `Group` reaches the floor by running the summary pipeline.
- `GroupTooSmall` means the caller already used `"__dwGroupSize"` (case-insensitive) as an aggregate
alias, a `Having` field or an `Orders` field. It throws in both tiers and in dry run. A small group never
raises it.
- The walk over `Having` that looks for the name reads a null `Conditions` or `SubConditionGroups` as an empty
list (3.3.0). A request body sending `"conditions": null` or `"subConditionGroups": null` overwrites the
list's initializer, and such a summary used to fail guarded with a `NullReferenceException` wherever the
floor is on, which is the default, though it ran unguarded. It runs, and the floor still applies.
- Dry run adds the column without the predicate. Each group below the floor gets its own decision
(`"a group of {n}, below the group floor of {floor}"`) and stays in `Data`.
- Transformed keys are handled in this order:
- Groups form in SQL on real values, and small groups are removed.
- Each transformed grouping-key column (named as the key path without dots) and each aliased aggregate
of a transformed field gets the field's chain. `MAX(Age) = 41` with `Round, Step = 10` returns `40`.
- If two rows now share the same values across the transformed keys, the query throws
`AmbiguousGroupKey`, in both tiers and in dry run. `FieldPath` is the transformed key paths joined by
`", "` and `Feature` is `Group` — under `Strict` outside a dry run it is `"*"` with no `SourceOrigin`
(3.3.0), because the key's canonical path is the column behind the caller's alias.
- Only the transformed keys are compared. Grouping by `[Department, Salary]` with `Salary` rounded throws
as soon as two departments share a rounded salary.
---
## 20. Model validation
```
DwPolicy.ValidateModel(params Type[] types) -> PolicyModelReport
DwPolicy.ValidateModel(DwPolicyOptions? options, params Type[] types) -> PolicyModelReport
throws InvalidOperationException listing every error, if there is any
PolicyModelValidator.Inspect(IEnumerable<Type> types) -> PolicyModelReport (never throws)
PolicyModelValidator.Inspect(IEnumerable<Type> types, DwPolicyOptions? options)
PolicyModelReport Errors:IReadOnlyList<string> Warnings:IReadOnlyList<string> IsValid (no errors)
```
```csharp
DwPolicyOptions options = new() { HashSalt = secret, TokenVault = vault };
options.Entities.Expose<Employee>("Employee");
PolicyModelReport report = DwPolicy.ValidateModel(options, options.Entities.ToArray()); // throws when invalid
foreach (string warning in report.Warnings) logger.LogWarning("{Warning}", warning);
DwPolicy.Configure(options, providers);
```
- Validation never runs automatically.
- It reads only the public instance properties of the listed types, so pass DTO and navigated types too.
- It does not check runtime rules.
- Each message starts with `Type.Member: `, except the `DefaultOrder` messages, which start with `Type: `.
Errors:
- two members of one type with the same `[DwAlias]`, compared case-insensitively
- `[DwDescribe]` that sets nothing
- `[DwAllowedValues]` with no values
- a negative `[DwCost]`
- `[DwAudit]` with `None` or an undefined bit
- a `[DwForceWhere]` that resolving the type's policy would refuse, with that refusal's message (3.1.0; before,
it surfaced only on the first guarded query):
- neither or both of `Value` and `ContextValue`, or either one with `IsNull` / `IsNotNull`
- a member type with no `DataType` (section 14)
- `AllowNull = true` with `IsNull` / `IsNotNull`, or on a member that can never be null
- `[DwEntity(DefaultOrder = ...)]` (3.1.0):
- an entry that is not a field optionally followed by `asc` or `desc`:
`"{Type}: DefaultOrder entry '{entry}' is not a field optionally followed by asc or desc, so guarded queries skip it."`
- a field whose name begins with one of the parser's own words (section 5), judged before the type is asked
whether it has the member, because the type may well have it:
`"{Type}: DefaultOrder names '{field}', which starts with a name the expression parser keeps for itself, so no query can use it. Rename the member."`
- a field no query can order by, a path ending on a collection of entities:
`"{Type}: DefaultOrder names '{field}', which no query can order by, so guarded queries skip it."`
- a field the type's own attributes deny for ordering, unless every one of those denials is `Overridable` (then a
warning, below). The attributes include those of the member's other declarations, an interface member it
implements, a subtype's override and a public member a subtype hides with `new` (3.2.0):
`"{Type}: DefaultOrder names '{field}', which its attributes deny for ordering, so every guarded query leaves it out."`
- a stage that cannot be built:
- `Round` or `Bucket` with `Step <= 0`, or `Truncate` with `Decimals < 0`
- a blank `[DwFormat]`
- a negative `KeepStart` or `KeepEnd`
- `Regex` without `Pattern`, or `Fixed` without `Text`
- a blank `TokenScope`
- a negative `[DwTruncate]` length
- `[DwDefault]` together with another transform attribute
- `[DwMutate]` naming a type that does not implement `IValueTransformer`
- when `options` is passed: `Hash` with an empty `HashSalt`, or `Tokenize` with a null `TokenVault`
- output type (skipped for chains with `[DwMutate]`):
- a `[DwDefault("v")]` that cannot convert
- `MaskStrategy.Null` on a non-nullable value type
- `Format`, `Truncate`, `Bucket`, or any mask other than `Null`, on a member that is not `string`
(after unwrapping `Nullable<T>`)
Warnings:
- the member has a transform other than `[DwDefault]`, and no DwDeny-family attribute on it covers `Order`.
The message ends "Add [DwNoOrder] unless that is intended."
- `DefaultOrder` names a field the type does not have (3.1.0):
`"{Type}: DefaultOrder names '{field}', which {Type} does not have, so guarded queries skip it."`
- `DefaultOrder` names a field denied for ordering only by overridable attributes on the member its path ends on,
such as `[DwNoOrder(Overridable = true)]`, which a rule can lift for some callers (3.1.0):
`"{Type}: DefaultOrder names '{field}', which its attributes deny for ordering unless a rule allows it, so guarded queries leave it out until one does."`
- `DefaultOrder` names a field the attributes allow ordering but deny for segments (3.1.0):
`"{Type}: DefaultOrder names '{field}', which its attributes deny for segments, so guarded segments leave it out."`
Not checked; these throw at query time:
- a blank, dotted or `"*"` alias
- a blank describe value or a blank listed value
- an invalid regex
- `Round`, `Truncate` or `DatePart` on a value of the wrong kind
Attributes on fields are neither checked nor applied.
---
## 21. Trace, explain and custom providers
### PolicyTrace
```
PolicyTrace sealed class, DynamicWhere.ex.Policies.DTOs
PolicyTrace(DwTier tier, bool dryRun)
Tier : DwTier
DryRun : bool true when this query ran in dry run
Decisions : IReadOnlyList<PolicyDecision> in the order taken; read-only view
Add(PolicyDecision decision) null → ArgumentNullException
PolicyDecision sealed class
PolicyDecision(string fieldPath, PolicyFeature feature, PolicyAction action, string? reason) blank path → ArgumentException
FieldPath : string canonical path, never the alias typed; "*" = whole request; "__dwGroupSize" = group floor;
under Strict, a name that matches nothing as the caller sent it, trimmed and with
empty segments dropped (3.1.0)
Feature : PolicyFeature
Action : PolicyAction
Reason : string?
PolicyAction Allowed=0 Denied=1 Dropped=2 Masked=3 Injected=4 Mutated=5 Defaulted=6 Generalized=7
PolicyFeature None=0 Where=1 Select=2 Order=4 Group=8 Aggregate=16 Segment=32 All=63 [Flags]
```
Where to read a trace:
- `FilterResult<T>.Policy` or `SummaryResult.Policy` (`SegmentResult<T>` inherits it). It is null on unguarded calls, and on guarded ones when `DwPolicyOptions.IncludeTraceInResult` withholds it: by default under `Strict` (3.1.0).
- `PolicyQueryable<T>.LastTrace`.
- `PolicySimulation<TClause>.Trace`.
A refused call throws, and `PolicyException` carries no trace. To see a refusal's decisions, run it in dry run or through `PolicySimulator`. A Strict refusal of a name that matches nothing shows only through `PolicySimulator`, because dry run fails that name in validation instead.
```
What is recorded
Dropped a field removed from a Convenience request (Select, Order); one per field
Dropped a DefaultOrder field left out for this caller (Order), both tiers, and in a Segment a field
denied for Segment too; Reason starts "left out of the default order" (3.1.0)
Dropped a denied field a synthesized projection leaves out (Select), at the top or beneath a member;
Reason names the policy's sources, such as "DwDeniedAttribute (sealed)" (3.2.0)
Dropped a member a synthesized projection leaves out whole (Select); Reason starts "left out whole" (3.2.0)
Dropped a member a type derived from T declares, which a synthesized projection building T leaves out
(Select); Reason starts "left out: a type derived" (3.2.0)
Dropped a member a synthesized projection cannot keep that the unguarded call would have returned, an
included navigation or an object of a row in memory (Select); Reason starts "left out:" (3.2.0)
Dropped a member a Convenience caller named that can hold a denied field no path names (Select); Reason
"it holds a field denied for Select where no path can name it" (3.2.0)
Denied an audited member no path of the policy names that the audit buffer has no room for (Select),
before the refusal that withholds the rows; Reason "MaxAuditEvents cap (<n>) reached with the
buffer undrained" (3.3.0, section 22)
Denied a refusal: Strict denial, cap, cost, query string ("*"), required filter, segment inference,
MaxAuditEvents; the throw follows unless dry run. Under Strict a name that matches nothing
is Denied under that name, Reason "names nothing on <TypeName>" (3.1.0)
Injected a forced predicate added (Where); Reason "forced predicate (<Operator>)", or
"forced predicate (<Operator>, or null)" when the term admits null (3.1.0)
Masked | Mutated | Defaulted | Generalized
once per transformed path per query (Select); Reason lists the stages ("Mask then Truncate");
summary key and aggregate columns use Feature Aggregate
Masked | Mutated | Defaulted | Generalized
a member the run-time-type pass transformed because no path of the policy names it (Select),
once per path per query; Reason is the stages followed by
" (declared on the member; no path of the policy names it)" (3.3.0, section 18)
Allowed an aliased column renamed on output (dynamic and Summary rows); Reason "emitted as '<alias>'"
Dropped "__dwGroupSize", Aggregate: the floor was applied; in dry run, one record per group below it
```
- A chain records `Defaulted` if it has a `[DwDefault]` stage, else `Mutated`, else `Masked`, else `Generalized`. A Format/Truncate-only chain records `Masked`.
- A Convenience drop leaves no marker in `Data`. The trace is the only way to tell a dropped field from a null value.
### PolicyResolver and explanations
```
PolicyResolver sealed class, DynamicWhere.ex.Policies.Resolution
PolicyResolver(IEnumerable<IDwPolicyProvider> providers) null → ArgumentNullException; null element → ArgumentException
Resolve(Type entityType, string fieldPath, DwPolicyContext context) -> FieldPolicy
Explain(Type entityType, string fieldPath, DwPolicyContext context) -> PolicyExplanation
ResolveType(Type entityType, DwPolicyContext context) -> TypePolicy
```
```csharp
PolicyExplanation why = DwPolicy.Resolver.Explain(typeof(Employee), "Salary", caller);
```
- `PolicyResolver.Explain` is the only explain API in the core package.
- `new PolicyResolver(...)` does not add `AttributePolicyProvider`; only `DwPolicy.Configure` does. Include it yourself, or attributes are ignored.
- `fieldPath` must be the canonical property path. It is trimmed, empty segments are dropped, and matching is case-insensitive.
- An alias is not translated, and the path is not checked for existence.
- Either mistake explains as Allow for every feature, because only wildcard fragments match.
- Errors:
- null type or context → `ArgumentNullException`;
- blank path → `ArgumentException`;
- a provider returning null or a null fragment → `InvalidOperationException` naming the provider.
- Resolution uses the context: with a store configured, an unprepared context throws `PolicyContextNotPrepared`.
- Explaining records no audit events.
```
PolicyExplanation sealed class, DynamicWhere.ex.Policies.DTOs
EntityType : string Type.FullName
FieldPath : string canonical
Policy : FieldPolicy the decision Resolve returns
Features : IReadOnlyList<FeatureExplanation> Where, Select, Order, Group, Aggregate, Segment, in that order
FeatureExplanation sealed class
Feature : PolicyFeature
Effect : PolicyEffect Allow when nothing spoke
DecidedBy : PolicySource? null when nothing spoke
Level : PolicyLevel? null when nothing spoke
TiedWith : IReadOnlyList<PolicySource> equal to the winner on level, specificity, priority and effect
Overrode : IReadOnlyList<PolicySource> outranked and discarded, not merged
IsAttributionAmbiguous : bool TiedWith.Count > 0; DecidedBy is one of several equals
PolicySource sealed class
Origin : string attribute type name, or "Rule <ruleId>"
RuleId : string? null for an attribute
Subject : string? null for an attribute
IsSealed : bool
static FromAttribute(string attributeName, bool isSealed) -> PolicySource blank name → ArgumentException
static FromRule(string ruleId, string subject) -> PolicySource blank argument → ArgumentException
ToString() "<Origin>", "<Origin> (sealed)", or "<Origin> [<Subject>]" for a rule
PolicyEffect Allow=0 Mask=1 Deny=2
PolicyLevel SealedAttribute=1 DynamicUser=2 DynamicRole=3 DynamicTenant=4 DynamicGlobal=5 OverridableAttribute=6
```
- `Effect` includes the resolver's own override. A transformed field without `AllowAggregate` reports `Aggregate` as `Deny`, even when `DecidedBy` is null or an Allow fragment.
- Fragments that decide no feature (an alias or facts only) appear in neither `TiedWith` nor `Overrode`.
```
FieldPolicy sealed class
FieldPath : string Sources : IReadOnlyList<PolicySource> IsSealed : bool
EffectFor(PolicyFeature feature) -> PolicyEffect Allow when no fragment spoke
Allows(PolicyFeature feature) -> bool anything but Deny
IsMasked(PolicyFeature feature) -> bool effect is Mask
AllowedOperators : IReadOnlyList<Operator>? null = unrestricted, empty = none allowed
AllowsOperator(Operator op) -> bool
Alias : string? ForcedPredicates : IReadOnlyList<ForcedPredicate>
RequiredOperators : IReadOnlyList<Operator>? IsRequiredInWhere : bool SatisfiesRequirement(Operator op) -> bool
Transform : ValueTransform? IsTransformed : bool
Facts : FieldFacts? Label, Description, Group : string? Order : int? AllowedValues : IReadOnlyList<string>?
CostWeight : int? AuditedFeatures : PolicyFeature? IsAudited() -> bool IsAudited(PolicyFeature feature) -> bool
TypePolicy sealed class
Aliases : IReadOnlyDictionary<string, IReadOnlyList<string>> public name → canonical paths
Forced : IReadOnlyList<ForcedPredicate>
Required : IReadOnlyDictionary<string, IReadOnlyList<Operator>>
Transforms : IReadOnlyDictionary<string, ValueTransform>
IsEmpty : bool
```
### Custom policy providers
```
IDwPolicyProvider interface, DynamicWhere.ex.Policies.Resolution
IReadOnlyList<PolicyFragment> GetFragments(Type entityType, DwPolicyContext context);
AttributePolicyProvider : IDwPolicyProvider sealed class; public parameterless constructor; const int MaxDepth = 4
```
Pass providers in:
```
DwPolicy.Configure(options, providerA, providerB)
builder.Services.AddDwPolicies(section, configure, providerA, providerB)
new PolicyResolver(new IDwPolicyProvider[] { new AttributePolicyProvider(), providerA })
// for the explicit ApplyPolicy, PolicySimulator and PolicySchemaBuilder
```
- `GetFragments` runs synchronously on the query path, per field per query. Do no I/O; serve from memory.
- Return every fragment for the type; the resolver does the path matching.
- Return an empty list, never null and never a null element (either → `InvalidOperationException`).
- The resolver trusts the `Level` a fragment claims, `SealedAttribute` included.
- `DwPolicy.PrepareAsync` prepares only `StorePolicyProvider` instances. A custom provider gets no per-request async hook.
```
PolicyFragment sealed class, DynamicWhere.ex.Policies.DTOs
PolicyFragment(string fieldPath, PolicyFeature features, PolicyEffect effect, PolicyLevel level,
PolicySource source, int priority = 0, object? payload = null,
IReadOnlyList<Operator>? allowedOperators = null, string? alias = null,
ForcedPredicate? forced = null, IReadOnlyList<Operator>? requiredOperators = null,
TransformStage? transform = null, FieldFacts? facts = null)
const string Wildcard = "*"
FieldPath Features Effect Level Source Priority Payload AllowedOperators Alias Forced
RequiredOperators Transform Facts get-only; FieldPath normalized, Alias trimmed
IsWildcard : bool
Matches(string fieldPath) -> bool wildcard, or equal ignoring case
Covers(PolicyFeature feature) -> bool every bit of feature is in Features
static NormalizePath(string fieldPath) -> string
ForcedPredicate sealed class
static FromConstant(string fieldPath, Operator op, DataType dataType, string value) -> ForcedPredicate
static FromConstant(string fieldPath, Operator op, DataType dataType, string value,
bool allowNull) -> ForcedPredicate 3.1.0
static FromContext(string fieldPath, Operator op, DataType dataType, string contextValue) -> ForcedPredicate
static FromContext(string fieldPath, Operator op, DataType dataType, string contextValue,
bool allowNull) -> ForcedPredicate 3.1.0
static FromNullCheck(string fieldPath, Operator op, DataType dataType) -> ForcedPredicate op: IsNull | IsNotNull
FieldPath Operator DataType Value ContextValue ReadsContext IsNullCheck AllowNull (3.1.0)
FieldFacts sealed class
FieldFacts(string? label = null, string? description = null, string? group = null, int? order = null,
IReadOnlyList<string>? allowedValues = null, int? costWeight = null, PolicyFeature? auditedFeatures = null)
static ForCost(int weight) static ForAudit(PolicyFeature features) static ForLabel(string label)
Label Description Group Order AllowedValues CostWeight AuditedFeatures Describes
```
- `PolicyFragment` → `ArgumentException` for:
- a blank path;
- an alias that is blank, dotted or `"*"`;
- on the wildcard: an alias, descriptive facts (label, description, group, order, allowed values), `requiredOperators` or a `transform`;
- a field fragment whose `forced` names another field.
- A null `source` → `ArgumentNullException`.
- `FieldFacts` → `ArgumentException` when nothing is supplied, for blank text, for empty or blank `allowedValues`,
or for `auditedFeatures` None or an unknown bit; `ArgumentOutOfRangeException` for `costWeight < 0` (0 is allowed).
- `ForcedPredicate` factories → `ArgumentException` for a blank `fieldPath`, `value` or `contextValue`;
`FromNullCheck` takes only `IsNull` / `IsNotNull`.
- `FromConstant` and `FromContext` with `allowNull: true` and `IsNull` or `IsNotNull` → `ArgumentException`
("AllowNull widens a comparison, and a null check compares against nothing.") (3.1.0). A null check ignores
a constant, and a widened `IsNotNull` would render `(field IS NOT NULL OR field IS NULL)`: a scope that scopes nothing.
Without `allowNull`, `FromConstant` accepts a null check and ignores its value, as before.
- `FromContext` with `IsNull` or `IsNotNull` → `ArgumentException`, whatever `allowNull` says (3.1.0). Without
`allowNull` the message is "'IsNull' compares against nothing, so it reads no context value; build it with
FromNullCheck."; with it, the refusal above answers first. A null check has nowhere to put a context value, so
the key is refused where the predicate is built.
- Fixed in 3.1.0. The factory used to accept it, and the key was still required: a caller without it was refused
with `MissingContextValue`, and a caller with it had the value added to a null check that validation refuses
(`ConditionWithOperator[IsNull-IsNotNull]MustHasNoValues`). Every guarded query on the type failed.
- `AllowNull` (3.1.0): true injects `(field op value OR field IS NULL)` in a group of its own (section 14). The
four-argument `FromConstant` and `FromContext` mean `allowNull: false`, and `FromNullCheck` never sets it.
The factories know no member type, so nothing refuses `allowNull` on a member that can never be null. On a
non-nullable value-type member of the queried type such a predicate injects the comparison alone, which is
the same predicate, and its trace reason has no ", or null"; a path through a navigation is always widened,
because the navigation can be absent.
- Transform stages derive from `TransformStage` and each has a public constructor: `MutateStage`, `GeneralizeStage`,
`FormatStage`, `MaskStage`, `TruncateStage`, `DefaultStage` (signatures in section 18).
---
## 22. Audit
```
IDwAuditSink interface, DynamicWhere.ex.Policies.Audit
ValueTask WriteAsync(DwAuditEvent auditEvent, CancellationToken ct = default);
DwAuditEvent sealed class
DwAuditEvent(DateTimeOffset occurredAt, string entityType, string fieldPath, PolicyFeature feature,
PolicyEffect effect, IReadOnlyList<DwSubject> subjects, string? purpose, DwTier tier, bool dryRun)
DwAuditEvent(DateTimeOffset occurredAt, string entityType, string fieldPath, PolicyFeature feature,
PolicyEffect effect, IReadOnlyList<DwSubject> subjects, string? purpose, DwTier tier, bool dryRun,
PolicyErrorCode? errorCode) 3.1.0
both: blank entityType or fieldPath → ArgumentException; null subjects → ArgumentNullException
OccurredAt : DateTimeOffset UTC, when recorded
EntityType : string Type.FullName
FieldPath : string canonical path, never the alias typed; for a refusal, see below
Feature : PolicyFeature the single use: Where, Select, Order, Group, Aggregate or Segment;
for a refusal, the refusal's Feature
Effect : PolicyEffect the policy's effect for that feature, whether or not the query then ran;
Deny for a refusal
Subjects : IReadOnlyList<DwSubject> the context's subjects
Purpose : string? DwPolicyContext.Purpose
Tier : DwTier
DryRun : bool false for a refusal
ErrorCode : PolicyErrorCode? 3.1.0. The refusal an event records; null for a use of an audited field
ToString() "<OccurredAt:O> <EntityType>.<FieldPath> <Feature> <Effect> <ErrorCode> [<subjects>]"
the code only when not null, the subjects only when there are any
```
- An event is recorded each time a guarded query resolves a field for a feature in the field's audited set.
- `[DwAudit]` defaults to `PolicyFeature.All`; rules can add features.
- One event per reference: a field in a condition and in an order gives two events.
- A field the type's `DefaultOrder` adds is a use too (3.1.0): audited for `Order`, it is recorded as an `Order`
use, `Effect` `Allow`, each time a guarded query orders by it, as a caller's own order is. A dry run keeps a
default field that enforcement would leave out, so it records that one too, with its `Order` effect (`Deny` for a
field the caller may not order by) and `DryRun` true.
- An audited member the rows hand back where no path of the policy names it is recorded once the rows show it
(3.3.0). The gate records by path before the query runs, and a member only a subtype of the row's type
declares, or one past the four segments the attribute walk reads, has no path it could ask about, so it came
back inside a row or a navigation kept whole with nothing written down. The outbound walk's second pass
(section 18) reports each one and the terminal records it.
- One `DwAuditEvent` per path per query, not per row. `Feature` `Select`; `Effect` `Mask` where the member is
transformed as well, `Allow` otherwise; `EntityType` the queried type's `FullName`; `FieldPath` the path
through the rows, such as `B.C.D.E.Five`, or `Hidden` for a subtype's member at the root.
- Only a member its own `[DwAudit]` audits for Select, and only where the projection carries it. A member the
declared types hold within four segments is the gate's, and so is a path the projection spells out however
long it is, so neither is recorded twice.
- Recorded in a dry run too, as every audited use is, and read only by a resolver that reads attributes.
- At `Caps.MaxAuditEvents` it fails closed as the gate does, and the rows are withheld: under `Strict` outside
a dry run the clause's own refusal, `FieldDeniedForSelect` or `FieldDeniedForSegment` for a segment, with
`FieldPath` `"*"`; otherwise `CapExceeded`, whose `SourceOrigin` reads
`MaxAuditEvents cap (<n>) reached with the buffer undrained`. The trace records a `Denied` decision on the
path first, and an `AuditRefusals` event records the path.
- Events are recorded whether the use was allowed, masked, dropped or denied, and in dry run too.
- Nothing is recorded for:
- a `DefaultOrder` field left out for this caller, outside a dry run: the query does not order by it, and the
caller never named it;
- a projection the library synthesized, as an act of its own: since 3.3.0 every audited member it hands back
is recorded for `Select` (section 14);
- `PolicySimulator`;
- `Explain`;
- schema building.
- Events buffer on the context (`PendingAuditEvents`); the query path never calls a sink.
- Drain once per request with `DwPolicy.DrainAuditAsync(context, sink)`. Undrained events are lost with the context.
- `Caps.MaxAuditEvents` bounds the undrained buffer.
- The event that does not fit refuses the query, even in dry run: `CapExceeded` (9) with `SourceOrigin`
containing `"MaxAuditEvents"`, and under `Strict` outside a dry run the clause's own field refusal with no
origin at all (3.3.0).
- Draining makes room again.
- A sink must throw when it fails to write. It is passed per call and never stored in options, so a scoped sink works.
- ASP.NET Core package: `app.UseDwPolicyAudit()` drains the context stored on `HttpContext.Features` to the `IDwAuditSink` registered in DI, after each response. The drain does not use the request's abort token: it has a budget of its own, thirty seconds (3.3.0, section 29).
### Refused queries — `DwPolicyOptions.AuditRefusals` (3.1.0)
`[DwAudit]` records uses. A caller probing for columns they may not read is refused at every guess, so a log of uses never shows the probe; this records the refusals.
- Off by default. When true, every `PolicyException` raised by a `PolicyQueryable<T>` method, terminal or composable, and the `PolicyContextNotPrepared` refusal of `ApplyPolicy(context)`, is written to the caller's buffer (`PendingAuditEvents`) and drains to `IDwAuditSink` through `DwPolicy.DrainAuditAsync` or the ASP.NET Core audit middleware, like `[DwAudit]` events.
- It covers a refusal raised anywhere beneath the handle method: the gate, resolution, a store provider, a transform.
- The event:
- `EntityType` = the queried type's `FullName`; `Effect` = `Deny`; `DryRun` = false; `ErrorCode` = the refusal's code;
- `Feature` and `Tier` = the refusal's own, so a store provider's refusal reports `All` and `Strict`;
- `Subjects` and `Purpose` = the context's;
- `FieldPath` = the field the refusal was about, as its canonical path, in both tiers. A `FieldDeniedFor*`, `OperatorNotAllowed` or `RequiredFilterMissing` refusal, a `CapExceeded` refusal from `MaxNavigationDepth`, the audit buffer's refusal whichever code it carries, and `MissingContextValue`, which records the scoped field, are all recorded under the path although the refusal itself named the caller's alias or, under `Strict`, said `"*"`. A name that matches nothing (`Strict`) is recorded as the caller sent it, trimmed and with empty segments dropped. Every other refusal records its own `FieldPath` (section 30): `"*"` for a whole-request refusal such as `QueryStringDenied`, `QueryCostExceeded`, a cap on the request's size or `ApplyPolicy`'s `PolicyContextNotPrepared`; the entity's short type name for a store provider's refusal; the name as written for `AmbiguousFieldName`.
- The recorded path is cut to its first 256 characters followed by `…` (3.1.0). Then every character in Unicode category Control (Cc), Format (Cf), Line Separator (Zl) or Paragraph Separator (Zp) is written as `\u` and four lowercase hex digits: a line feed as `\u000a`, U+2028 as `\u2028`, U+202E as `\u202e`. A character outside the Basic Multilingual Plane is judged whole, and both halves of its surrogate pair are escaped. A name that matches nothing is text the caller wrote: a line break in it would forge a second entry in a log written one event per line, a format character such as U+202E would reverse the text after it without showing itself, and a name a megabyte long would be kept whole.
- Written at most once per refusal, from an exception filter: the refusal is never changed, caught or swallowed.
- A full buffer (`Caps.MaxAuditEvents`) records nothing, and the original refusal is still thrown.
- Not written:
- what dry run only records: it throws nothing there, so there is no refusal. A refusal dry run still throws (section 17) is written, with `DryRun` false;
- a refusal with no guarded context, such as `PolicyRequired` on an unguarded read of a `RequirePolicy` type;
- a `PolicySimulator` refusal, which runs on a copy of the context.
- Off by default because it changes what reaches a sink: a sink registered for `[DwAudit]` starts receiving events with an `ErrorCode`, and a deployment with no sink is warned by the ASP.NET Core audit middleware on every refused request.
---
## 23. Discovery: catalogue, schema, simulation
### DwEntityCatalog
```
DwEntityCatalog sealed class, DynamicWhere.ex.Policies.Discovery; DwPolicyOptions.Entities
DwEntityCatalog()
Expose<T>(string? name = null) -> DwEntityCatalog chainable
Expose(Type type, string? name = null) -> DwEntityCatalog
Resolve(string? name) -> Type? public name (case-insensitive), then exact Type.FullName; else null
NameOf(Type type) -> string? public name; null when never exposed
ToArray() -> Type[]
Entities : IReadOnlyDictionary<Type, string> read-only view
IsEmpty : bool
Freeze() -> void
```
- A null or blank `name` means `type.Name`; names are trimmed.
- Errors:
- a name already held by another type → `ArgumentException`;
- null type → `ArgumentNullException`;
- exposing after `Configure` → `InvalidOperationException`.
- Exposing a type again under another name keeps the old name resolvable, and `NameOf` returns the newest.
- Only exposed types can be described. `Resolve` gives the same null for a type that does not exist and one never exposed.
### PolicySchemaBuilder
```
PolicySchemaBuilder static class
Describe(Type entityType, DwEntityCatalog catalogue, DwPolicyContext context, DwPolicyOptions options,
PolicyResolver resolver, PolicySchemaRequest? request = null) -> PolicySchema
ResolveNavigation(Type entityType, string path, DwPolicyContext context,
DwPolicyOptions options, PolicyResolver resolver) -> string? canonical navigation path or null
PolicySchemaRequest sealed class; every member optional
Paths : IReadOnlyList<string>? { get; init; } navigation roots, canonical or alias per segment; null/empty = the entity
Depth : int? { get; init; } levels from each root; null = Caps.SchemaDepth; 1 = the root's own fields
```
```csharp
PolicySchema schema = PolicySchemaBuilder.Describe(
typeof(Employee), DwPolicy.Options.Entities, caller, DwPolicy.Options, DwPolicy.Resolver,
new PolicySchemaRequest { Paths = new[] { "Manager" }, Depth = 2 });
```
```
PolicySchema sealed class
Entity : string catalogue name
EntityType : string Type.FullName, what rules match on
Roots : IReadOnlyList<string> canonical roots; empty when rooted at the entity
Depth : int levels actually covered after clamping (largest over roots)
MaxDepth : int Caps.MaxNavigationDepth
Truncated : bool MaxSchemaFields cut Fields short
Fields : IReadOnlyList<PolicySchemaField> by Group (ungrouped last), Order (null last), Name; case-insensitive
Nodes : IReadOnlyList<PolicySchemaNode> by Path
PolicySchemaField sealed class
Path : string canonical path; what a rule names
Name : string alias, else Path; what a caller-facing UI shows
Parent : string? navigation path it hangs under; null for the entity's own fields
DataType : DataType
CanWhere CanSelect CanOrder CanGroup CanAggregate CanSegment : bool
Can(PolicyFeature feature) -> bool
IsMasked : bool Select effect is Mask: the value is transformed on output
AllowedOperators : IReadOnlyList<Operator>? null = unrestricted
AllowedValues : IReadOnlyList<string>?
IsRequiredInWhere : bool
CostWeight : int elected weight, else Caps.DefaultFieldCost
Label Description Group : string? Order : int?
PolicySchemaNode sealed class
Path : string
Name : string alias, else Path
Parent : string? null at the root of the view
Entity : string? catalogue name of the navigation's type; null when not exposed
Depth : int level of this node's own fields; the entity's fields are level 1
Expanded : bool the walk listed its fields
RemainingDepth : int navigation levels a request rooted here could still return; 0 = nothing to open
```
- A field is listed only when:
- its type maps to a `DataType`, and
- the caller may use it for at least one feature.
- A fully denied field (`[DwDenied]`) never appears. Audited fields are not marked.
- The entity is level 1; a root path of n segments is level n + 1. A root at level L walks to `min(L + Depth - 1, MaxNavigationDepth)`.
- A larger `Depth` is clamped, never refused; read `Depth` and `MaxDepth` back.
- `Depth < 1` → `ArgumentException`.
- A navigation is expanded while it is below that level and its type has appeared fewer than `SchemaCycleLimit` times on the path.
- The count restarts at each requested root, so `Manager.Manager` is described by requesting it as a root.
- Expanded nodes with nothing usable beneath them are removed. Unexpanded nodes are listed without checking beneath.
- Overlapping roots list each field and node once.
- Reaching `MaxSchemaFields` stops the walk and sets `Truncated`.
- `Describe` → `ArgumentException` when:
- the type is not exposed;
- a root names nothing;
- a root names a simple field the caller can see;
- a root has `>= MaxNavigationDepth` segments.
- A root naming a field the caller cannot use is answered as naming nothing.
- `ResolveNavigation` returns null for nothing, and throws `ArgumentException` for a blank path, a visible simple field, or a path too deep.
- A schema is resolved for the caller on every call and never cached. With a store configured, the context must be prepared.
### PolicySimulator
```
PolicySimulator static class
Simulate<T>(Filter filter, DwPolicyContext context, DwPolicyOptions options, PolicyResolver resolver) -> PolicySimulation<Filter>
Simulate<T>(Summary summary, DwPolicyContext context, DwPolicyOptions options, PolicyResolver resolver) -> PolicySimulation<Summary>
Simulate<T>(Segment segment, DwPolicyContext context, DwPolicyOptions options, PolicyResolver resolver) -> PolicySimulation<Segment>
Simulate<TClause>(Type entityType, TClause clause, DwPolicyContext context,
DwPolicyOptions options, PolicyResolver resolver) -> PolicySimulation<TClause>
where T : class; where TClause : class
PolicySimulation<TClause> where TClause : class sealed class
PolicySimulation(TClause? clause, PolicyTrace trace, PolicyException? refusal)
Clause : TClause? sanitized copy: drops applied, forced predicates injected, names canonical; null when refused
Trace : PolicyTrace
Refusal : PolicyException? null when it would run
WouldRun : bool Refusal is null
```
```csharp
PolicySimulation<Filter> sim = PolicySimulator.Simulate<Employee>(filter, caller, DwPolicy.Options, DwPolicy.Resolver);
if (!sim.WouldRun) logger.LogInformation("{Code}", sim.Refusal!.ErrorCode);
```
- It sanitizes only: no database, no transforms, and no audit events on `context`, because it runs on a copy. The clause passed in is not modified.
- A `PolicyException` becomes `Refusal`. Any other exception propagates, unwrapped even from the runtime overload.
- Under `Strict`, outside dry run, a path that matches nothing is therefore a `Refusal` with the clause's `FieldDeniedFor*` code and `FieldPath` `"*"`, not a `LogicException` (3.1.0). The `Trace` names it.
- A simulated `Filter` or `Segment` that sends no orders gets the type's `DefaultOrder` in `Clause.Orders`, less the fields the caller may not order by (3.1.0).
- A simulation has no source, so it reads T as a source it cannot see into (3.2.0): every denial beneath a member
counts, and with no `Selects` a synthesized `Clause.Selects` keeps only members holding a value. The guarded query
over a projected row keeps its assigned objects, one over an entity keeps its columns, owned and complex members
and asks only about denials whose value it loads, and one in memory keeps values only.
- Runtime overload errors:
- a value-type `entityType`, or a `TClause` other than `Filter`, `Summary` or `Segment` → `ArgumentException`;
- null entity, context or options → `ArgumentNullException`.
- Handle-level refusals are not simulated: `QueryStringDenied`, `TransformRequiresMaterialization`, `PolicyRequired`,
and the `PolicyContextNotPrepared` that `ApplyPolicy(context)` raises. The simulator never reads `IsPrepared`, so
without a store an unprepared context simulates as `WouldRun`; a store provider still refuses a context it never
prepared.
- A simulated Summary's `Clause` includes the floor's `__dwGroupSize` aggregate and its HAVING.
---
## 24. Token vaults
```
IDwTokenVault interface, DynamicWhere.ex.Policies.Tokens
string GetOrCreate(string scope, string value); stable token for scope + value
DwToken static class; helpers for vault implementations
New() -> string 32 lowercase hex characters from 16 RandomNumberGenerator bytes
KeyFor(string scope, string value) -> string "<scope>:<lowercase hex SHA-256 of the UTF-8 value>"
blank scope → ArgumentException; null value → ArgumentNullException
KeyFor(string scope, string value, byte[] key) "hmac:<scope>:<64 lowercase hex>" (3.3.0); HMAC-SHA256 under the
-> string key over UTF8(scope), one zero byte, UTF8(value), so one value in
two scopes shares no digest. Blank scope → ArgumentException;
null value or key → ArgumentNullException; key under
MinimumKeyLength → ArgumentException
RequireKey(byte[] key) -> byte[] (3.3.0) returns a copy, so a caller clearing or reusing its array
cannot re-key a running vault. null → ArgumentNullException;
under MinimumKeyLength → ArgumentException
MinimumKeyLength : const int = 16 (3.3.0) the fewest bytes a vault key may have
KeyedPrefix : const string = "hmac:" (3.3.0) what a keyed mapping's key starts with
InMemoryTokenVault : IDwTokenVault sealed class; parameterless constructor
GetOrCreate(string scope, string value) -> string
Count : int mappings held
Clear() -> void forgets every mapping; tests only (reissues every token)
```
- `InMemoryTokenVault` is process-local, thread-safe and unbounded (no eviction).
- It draws a random 32-byte key of its own per instance and stores under `KeyFor(scope, value, key)` (3.3.0), so a
memory dump holds keyed digests rather than plain ones. Nothing to configure, and no API change: its mappings die
with the process, so the key can too.
- Raw values are never keys.
- Tokens are lost on restart, and two instances or processes issue different tokens for one value.
- `RedisTokenVault` (Redis package) and `EfTokenVault` (EF Core package) keep tokens across restarts and instances.
Each takes a key in a constructor of its own (3.3.0, sections 27 and 28); the constructors that do not take one are
unchanged and store under the unkeyed `KeyFor`.
- **Why a durable vault wants a key (3.3.0).** The unkeyed key is a plain digest, and a tokenized column is nearly
always drawn from a space small enough to hash whole — phone numbers, national identifiers, card numbers. So a copy
of the store, a backup or a replica or a dump, gives back every value in it, and with them the value behind every
token ever issued. Under a key held outside the store, in configuration or a secret manager, the store and the key
have to be taken together. Guard the store as you would guard the column it protects either way.
- **Adoption keeps every token already issued.** A keyed vault meeting a value with no keyed mapping looks up the
unkeyed mapping too, and the token found there is the one written under the keyed key, so yesterday's export
still lines up with today's. The unkeyed mapping stays until `retireUnkeyed` is true; a retiring vault deletes it
the first time it meets the value, whether it wrote the keyed mapping or found it.
- **Roll out in two steps:** give every instance the key, then turn `retireUnkeyed` on. An instance still running
without the key mints a *new* token for a value whose unkeyed mapping is gone, and a value first met while keyed
and unkeyed instances run side by side can end up with two tokens.
- Unkeyed mappings of values never met again stay until an operator deletes them — Redis: `HSCAN` the token hash
and delete the fields that do not match `hmac:*`; SQL: delete the rows of `DwPolicyTokens` whose `Key` does not
start with `hmac:` — knowing such a value gets a new token the next time it is met.
- Changing the key re-issues every token, unless unkeyed mappings remain to adopt from. Another key is another
vault.
- A vault is read on the query path, once per value per row, from many threads. A remote vault must cache in process.
- Implementation contract:
- thread-safe;
- return a stable, non-blank token, and store it before returning;
- never return the input value;
- throw when the store is unreachable.
- The interface has no reverse lookup.
- `InMemoryTokenVault`: a null `value` → `ArgumentNullException`; an empty string gets its own token.
- A tokenizing query while `DwPolicyOptions.TokenVault` is null → `MissingTokenVault` (22).
---
## 25. Dynamic rules
Rules a store supplies at runtime, merged with attributes by precedence level. Types live in `DynamicWhere.ex.Policies.Storage` (rule, stores, serializers), `DynamicWhere.ex.Policies.Resolution` (`StorePolicyProvider`) and `DynamicWhere.ex.Policies.DTOs` (`ForcedPredicate`, `FieldFacts`, transform stages).
### PolicyRule — immutable; every check runs in the constructor
```
new PolicyRule(
DwSubjectKind subjectKind, must be a defined member
string? subjectKey, required unless Global; trimmed; Global stores null
string entityType, Type.FullName; trimmed; must contain '.'
string fieldPath, a path or "*"; trimmed, segments trimmed, empty segments dropped
PolicyFeature features, no bit outside All
PolicyEffect effect, must be a defined member
int priority = 0, tiebreak within a level; higher wins
bool enabled = true, false: kept in the store, never applied
DateTimeOffset? validFrom = null, inclusive; null = already valid
DateTimeOffset? validTo = null, exclusive; null = never ends
string? purpose = null, trimmed; blank -> null
TransformStage? transform = null, one stage of the field's chain
IReadOnlyList<Operator>? allowedOperators = null, copied; null = says nothing
string? alias = null,
ForcedPredicate? forced = null,
IReadOnlyList<Operator>? requiredOperators = null, copied; null = no requirement, empty = nothing satisfies it
FieldFacts? facts = null,
Guid? id = null, null -> Guid.NewGuid()
string? createdBy = null, DateTimeOffset? createdAt = null,
string? updatedBy = null, DateTimeOffset? updatedAt = null)
```
Get-only properties: `Id SubjectKind SubjectKey Level EntityType FieldPath Features Effect Priority Enabled ValidFrom ValidTo Purpose Transform AllowedOperators Alias Forced RequiredOperators Facts CreatedBy CreatedAt UpdatedBy UpdatedAt IsBroad`
```
static string? NormalizeSubjectKey(string? key) Trim().ToLowerInvariant(); blank -> null
bool AppliesAt(DateTimeOffset now) (ValidFrom null or <= now) and (ValidTo null or now < ValidTo)
bool MatchesSubject(DwPolicyContext context) Global, or context holds SubjectKind:SubjectKey (OrdinalIgnoreCase)
bool MatchesPurpose(DwPolicyContext context) Purpose null, or equal to context.Purpose (OrdinalIgnoreCase)
PolicyFragment ToFragment() fragment at Level; source renders "Rule {Id} [{Describe()}]"
string Describe() "Global" or "Kind:Key"
string ToString() "{Describe()} {EntityType}.{FieldPath} {Features} => {Effect}"
bool IsBroad SubjectKind != User
```
The constructor refuses:
- `ArgumentOutOfRangeException` when `subjectKind` or `effect` is not a defined member.
- `ArgumentException` for any of these:
- `features` with an unknown bit.
- `features` None when the rule carries nothing (no alias, operator list, forced predicate, transform or facts).
- `features` None with any effect other than Allow.
- A blank `entityType`, or one with no '.'.
- A blank `fieldPath`, or one with no segment.
- A non-Global rule with no `subjectKey`.
- `validTo <= validFrom`.
- On `"*"` it also refuses an alias, `requiredOperators`, a `transform`, and `facts` with a label, description, group, order or allowedValues. A forced predicate, a cost weight and an audit are accepted on `"*"`.
Not checked by the constructor, by any store or by POST /rules. These are stored, then throw `ArgumentException` on the query path for every caller the rule applies to:
- An alias that is blank, contains '.', or is `"*"`.
- A rule with a real field path whose `forced` predicate names a different field.
The level comes from the subject kind and is never stored:
```
Global -> DynamicGlobal=5 Tenant -> DynamicTenant=4 Custom -> DynamicTenant=4
Role -> DynamicRole=3 User -> DynamicUser=2 no stored rule reaches SealedAttribute=1
```
How a rule applies, evaluated on every query:
- **Entity:** `EntityType` equals `Type.FullName`, case-insensitive.
- **Field:** an exact path, case-insensitive. `"*"` means every field of the type.
- There is no partial wildcard: `"Orders.*"` matches nothing.
- A rule on a navigation does not cover the fields beneath it.
- **Subject:** a Custom identity carries no dimension name, so every Custom subject with the same value matches.
- **Purpose:** a rule with a purpose never applies to a caller whose `Purpose` is null. Write a denial that must always hold without a purpose.
- **Validity window:** tested against UtcNow on each query, never at load.
- **Disabled rules** never reach a snapshot.
- **Transform stages** elect one winner per stage. A rule can add a stage on top of a sealed chain but cannot replace a sealed stage.
### Building rule parts in code
`ForcedPredicate` and `FieldFacts` are listed in section 21 and the transform stage classes in section 18.
`MutateStage` can be built in code, but the Redis and EF Core stores refuse to save it.
```csharp
var deny = new PolicyRule(DwSubjectKind.Role, "Support", typeof(Employee).FullName!, "Position",
PolicyFeature.Select | PolicyFeature.Order, PolicyEffect.Deny, priority: 10,
validTo: DateTimeOffset.UtcNow.AddDays(30));
// Carries a predicate and decides nothing: features None, effect Allow.
var scope = new PolicyRule(DwSubjectKind.Global, null, typeof(Employee).FullName!, "*",
PolicyFeature.None, PolicyEffect.Allow,
forced: ForcedPredicate.FromContext("TenantId", Operator.Equal, DataType.Number, "TenantId"));
// The caller's institution or none (3.1.0): injects (InstitutionId = x OR InstitutionId IS NULL).
var shared = new PolicyRule(DwSubjectKind.Global, null, typeof(Role).FullName!, "*",
PolicyFeature.None, PolicyEffect.Allow,
forced: ForcedPredicate.FromContext("InstitutionId", Operator.Equal, DataType.Number, "TenantId", allowNull: true));
```
### Rule document — `PolicyRuleDocument`
`PolicyRuleDocument` is the only JSON format for a whole rule. Redis stores the whole document; EF stores columns plus the `detail` object. POST /rules takes a different body, `RuleRequest`.
```
static string PolicyRuleDocument.ToJson(PolicyRule rule)
static PolicyRule PolicyRuleDocument.ToRule(string json) goes through the PolicyRule constructor
static string? PolicyRuleDocument.DetailToJson(PolicyRule rule) null when the rule has no detail
static RuleDetail PolicyRuleDocument.ReadDetail(string? json) blank -> RuleDetail.None
static TEnum PolicyRuleDocument.ToEnum<TEnum>(string? name, string what)
static PolicyFeature PolicyRuleDocument.ToFeatures(string? name) e.g. "Where, Select"
new RuleDetail(TransformStage? transform, IReadOnlyList<Operator>? allowedOperators, string? alias,
ForcedPredicate? forced, IReadOnlyList<Operator>? requiredOperators, FieldFacts? facts = null) RuleDetail.None
static TransformStage PolicyPayload.ToStage(string json) static string PolicyPayload.ToJson(TransformStage stage)
```
```jsonc
{
"id": "0b4c2f1e-7d0a-4c1e-9a55-2f6f5d0e9b10", // GUID; absent -> new id
"subjectKind": "Role", // required
"subjectKey": "Support", // absent for Global
"entityType": "MyApp.Models.Employee", // required
"fieldPath": "Email", // required
"features": "Select", // required; comma-separated flag names, "None" or "All"
"effect": "Mask", // required
"priority": 10, "enabled": true, // defaults 0 and true
"validFrom": "2026-09-01T00:00:00+00:00", // ISO 8601; validTo the same; absent = null
"purpose": "support",
"createdBy": "ops", "createdAt": "2026-09-01T00:00:00+00:00", "updatedBy": "ops", "updatedAt": "...",
"detail": { // absent when the rule has none of these
"transform": { "kind": "Mask", "allowAggregate": false, "minGroupSize": 0,
"strategy": "Email", "keepStart": 0, "keepEnd": 0, "maskChar": "*", "preserveLength": true },
"allowedOperators": ["Equal", "In"],
"requiredOperators": ["Equal"],
"alias": "ContactEmail",
"forced": { "fieldPath": "Email", "operator": "IsNotNull", "dataType": "Text" },
// a null check takes neither value nor contextValue; a comparison
// takes one of them, plus "allowNull": true to admit null (3.1.0)
"facts": { "label": "Email", "description": "...", "group": "Contact", "order": 1,
"allowedValues": ["a", "b"], "cost": 5, "audit": "Where, Select" }
}
}
```
Transform payload keys, read by `PolicyPayload.ToStage`:
```
every stage kind (required: Mask | Generalize | Format | Truncate | Default) allowAggregate false minGroupSize 0
Mask strategy (required) keepStart 0 keepEnd 0 maskChar "*" (exactly one char) preserveLength true
pattern replacement text tokenScope
Generalize mode (required) step 0 part "Year" decimals 0
Format format (required)
Truncate length (required) ellipsis
Default value; the key being present, even as null, means a value was supplied
```
Document rules:
- Property names must be exact camelCase. Unknown properties are ignored.
- Enums are names, matched case-insensitively. The following throw `ArgumentException`:
- A JSON number.
- A blank value, or a name that is not defined.
- A string of digits, for the rule-level enums: `subjectKind`, `effect`, `features`, operators, `dataType` and `audit`.
- `"kind": "Mutate"` is refused both when reading and when writing.
- `forced` holds either `value` or `contextValue`, or neither for IsNull/IsNotNull. Both together throw `ArgumentException`.
- A `contextValue` on IsNull/IsNotNull throws `ArgumentException` when the rule is read (3.1.0): the reader builds it
through `ForcedPredicate.FromContext`, which refuses a null check (section 21). Such a rule never worked: the key was
still required, and its value landed on a null check that validation refuses, so every guarded query on the type
failed.
- A `value` on IsNull/IsNotNull is still accepted and ignored.
- `forced.allowNull` (3.1.0): `true` injects `(field op value OR field IS NULL)`, as `[DwForceWhere(AllowNull = true)]` does.
- It is written only when true, so a document for any other predicate is the one earlier releases wrote and read. Absent or JSON null reads as false.
- Anything but a JSON boolean throws `ArgumentException` (the string `"true"` included).
- `true` on a null check (`IsNull` / `IsNotNull`) throws `ArgumentException`, whether or not the object also carries a `value` or `contextValue` (3.1.0). A null check ignores a constant, and an `IsNotNull` rule widened this way would render `(field IS NOT NULL OR field IS NULL)`, a scope that scopes nothing. Without `allowNull`, a stray `value` on a null check is still ignored; a `contextValue` there is refused (above).
- The document knows no member type, so `true` is not refused on a member that can never be null; there the rule injects the comparison alone (section 21).
- An absent operator list and an empty one stay distinct.
- A document or row that cannot be read is never skipped: it fails the load that reads it.
- A broad rule fails the load: fatal at startup, and a refresh failure afterwards, where `StoreFailure` applies.
- A `User` rule is read only when a context is prepared, so it fails `DwPolicy.PrepareAsync` for the callers it
names, in every `StoreFailure` mode, and does not degrade the provider.
---
## 26. Rule stores and StorePolicyProvider
### Contracts
```
interface IDwPolicyStore
ValueTask<StoreSnapshot> LoadAsync(CancellationToken ct) broad zone
ValueTask<NarrowZone> LoadNarrowAsync(IReadOnlyList<string> userIdentities, CancellationToken ct)
ValueTask<long> GetVersionAsync(CancellationToken ct)
IAsyncEnumerable<long>? WatchAsync(CancellationToken ct) null = cannot notify
interface IDwPolicyWritableStore : IDwPolicyStore
ValueTask<PolicyRule> UpsertAsync(PolicyRule rule, CancellationToken ct) insert or replace by Id; bumps the version
ValueTask DeleteAsync(Guid id, CancellationToken ct) bumps the version even when the id is absent
interface IDwPolicyRefresher
ValueTask<long> RefreshAsync(CancellationToken ct) implemented by StorePolicyProvider
```
- No store method runs on the query path. Stores are read at startup, by `PrepareAsync` and by the refresh loop.
- A read-only deployment registers only `IDwPolicyStore`.
- If `LoadAsync` or `LoadNarrowAsync` returns null, the provider throws `InvalidOperationException`.
- The shipped stores do four things a custom store should copy:
- Parse enums with `PolicyRuleDocument.ToEnum` and `ToFeatures`.
- Build user keys with `PolicyRule.NormalizeSubjectKey`.
- Call `SealedFields.Refuse` in `UpsertAsync`.
- Throw on a row that cannot be read.
### Zones
```
new StoreSnapshot(long version, DateTimeOffset loadedAt, IEnumerable<PolicyRule> rules)
long Version DateTimeOffset LoadedAt int Count static StoreSnapshot Empty IReadOnlyList<PolicyRule> For(Type entityType)
new NarrowZone(long version, IEnumerable<PolicyRule> rules)
long Version int Count static NarrowZone Empty IReadOnlyList<PolicyRule> For(Type entityType)
```
- **Broad zone (`StoreSnapshot`):** Global, Tenant, Role and Custom rules. It is loaded whole and swapped atomically.
- **Narrow zone (`NarrowZone`):** User rules. It is loaded when a context is prepared, for that caller's User identities only.
- **Wrong-zone rules:** a User rule in a snapshot, a non-User rule in a narrow zone, or a null throws `ArgumentException`, and the load fails.
- **Disabled rules** are dropped at construction, and `Count` counts only enabled rules. Validity windows are not applied here.
- **`For(Type)`** looks up `Type.FullName`, case-insensitive.
- **`StoreSnapshot.Empty`** (version 0) means no load has succeeded. An empty store returns a real snapshot instead.
- **`StoreSnapshot.LoadedAt`** is the store's own clock and decides nothing.
### InMemoryPolicyStore — core package
```
new InMemoryPolicyStore(Func<string, Type?>? resolveType = null) : IDwPolicyWritableStore, IDisposable
long Version starts at 0; +1 per write
InMemoryPolicyStore Seed(params PolicyRule[] rules) add or replace; one bump; null rule or sealed field -> ArgumentException
LoadAsync LoadNarrowAsync GetVersionAsync WatchAsync (yields every new version) UpsertAsync DeleteAsync
void Dispose() completes every watch
```
- It is thread-safe and not persisted. User identities match OrdinalIgnoreCase.
### Sealed-field refusal on write — `SealedFields`
```
static void SealedFields.Refuse(PolicyRule rule, Func<string, Type?>? resolveType, string parameterName)
```
- **Refusal:** throws `ArgumentException` when a SealedAttribute-level attribute fragment matches the rule's `FieldPath` and shares any of its features. A rule that names only features no sealed attribute covers is accepted.
- **Where it runs:**
- In `InMemoryPolicyStore.Seed` and `UpsertAsync`, `RedisPolicyStore.UpsertAsync` and `EfPolicyStore.UpsertAsync`, using the store's own `resolveType`.
- In POST /rules, using `DwPolicy.Options.Entities.Resolve`.
- **No resolver:** if `resolveType` is null or returns null, the rule is accepted. It still grants nothing, because a sealed attribute outranks every dynamic level when resolving.
- **Wildcards:** paths are compared exactly, so a `"*"` rule is not refused just because one field is sealed.
- **`DwEntityCatalog.Resolve(string? name)`** answers only for exposed types, by public name (case-insensitive) or exact `Type.FullName`.
- **Which catalogue:** take the delegate from the options you will configure, `options.Entities.Resolve`. Until `DwPolicy.Configure` runs, `DwPolicy.Options` is a separate default instance with an empty catalogue.
### StorePolicyProvider
```
: IDwPolicyProvider, IDwPolicyRefresher, IDisposable
static ValueTask<StorePolicyProvider> CreateAsync(IDwPolicyStore store, DwPolicyOptions options,
bool autoRefresh = true, CancellationToken ct = default)
ValueTask<DwPolicyContext> PrepareAsync(DwPolicyContext context, CancellationToken ct = default)
IReadOnlyList<PolicyFragment> GetFragments(Type entityType, DwPolicyContext context)
ValueTask<long> RefreshAsync(CancellationToken ct = default)
long Version bool IsDegraded DateTimeOffset LoadedAt TimeSpan Age Exception? LastError void Dispose()
```
- **`CreateAsync`** runs the first `LoadAsync` without catching it. Any failure throws, whatever `StoreFailure` is, and no provider is created.
- **Settings source:** `StoreFailure`, `MaxSnapshotAge` and `RefreshInterval` come from the `options` passed to `CreateAsync`, not from `DwPolicy.Options`. Pass the same instance to `DwPolicy.Configure`.
- **`DwPolicy.Configure(options, provider)`** always adds `AttributePolicyProvider` as well. `DwPolicy.StoreProviders` lists store providers in the order they were passed.
- **`DwPolicy.PrepareAsync(context, ct)`** calls each store provider's `PrepareAsync` in order, then sets `IsPrepared`. With no store configured it reads nothing, but still sets `IsPrepared`.
- **`PrepareAsync`:**
- Loads the narrow zone for the context's User identities, skipped when there are none.
- Pins onto the context the current snapshot, the provider's `LoadedAt`, the narrow zone, and the identities it read. Preparing again replaces the pin.
- A narrow-load failure propagates, in every mode, and so does a `User` rule the store cannot read.
- **`RefreshAsync`:**
- Always reloads and swaps atomically.
- Stamps `LoadedAt` from the provider's own UTC clock and clears `IsDegraded` and `LastError`.
- On failure it sets `IsDegraded` and `LastError`, keeps the last snapshot, and rethrows. `OperationCanceledException` is rethrown without degrading.
- **`Dispose()`** stops the background loop and waits up to 5 seconds.
`GetFragments` runs these checks in order on every guarded query:
```
1 this provider never prepared the context -> PolicyException PolicyContextNotPrepared
2 the context gained a User identity after it was prepared -> PolicyException PolicyContextNotPrepared
3 IsDegraded and StoreFailure = FailClosed -> PolicyException StoreUnavailable
IsDegraded and StoreFailure = StaticOnly -> returns no fragments from this store
4 now - pinned LoadedAt > MaxSnapshotAge (any mode, healthy too) -> PolicyException StoreUnavailable
5 one fragment for each rule in the pinned snapshot, then the pinned narrow zone, that applies now to this caller
```
- These refusals carry `FieldPath` = the entity's short type name and `Feature` = All.
- They report tier Strict, even under Convenience, and `SourceOrigin` = "{store type name}: {reason}".
- `IsDegraded` is read live, not pinned.
### Failure, staleness and refresh
```
StoreFailureMode LastKnownGood=0 FailClosed=1 StaticOnly=2
DwPolicyOptions StoreFailure = LastKnownGood MaxSnapshotAge = 00:15:00 RefreshInterval = 00:00:30
both TimeSpans must be > 0 (ArgumentOutOfRangeException); no value turns the ceiling off
```
```
Mode While IsDegraded When not degraded
LastKnownGood serves the pinned snapshot until the ceiling ceiling applies
FailClosed StoreUnavailable on every guarded query ceiling applies
StaticOnly no store fragments; ceiling not checked ceiling applies
```
- **Startup:** a failed first load throws in all three modes.
- **Background loop (`autoRefresh: true`):**
- It subscribes to `WatchAsync` when that returns non-null. If opening the watch throws, it polls only.
- It always polls `GetVersionAsync` every `RefreshInterval`. The first poll comes one interval after start.
- It calls `RefreshAsync` only when the reported version differs from the version being served. A lower version also counts as different.
- A poll whose version matches renews the snapshot without reloading: it stamps the load time (as of when it asked) and clears the degraded flag — unless a refresh or poll failed while its read was in flight, in which case it reloads instead (3.1.0).
- **Degraded state:**
- A failed poll sets `IsDegraded` and leaves `LastError` alone. A failed reload sets both.
- Cleared by a successful `RefreshAsync`, and by a poll that reads back the version being served.
- So a store that comes back is served again from the next poll, with or without a write to it.
- **The ceiling:**
- `LoadedAt` moves on `CreateAsync`, on a successful `RefreshAsync`, and on a poll that confirms the served version.
- With `autoRefresh: true` an unchanged healthy store keeps renewing through the poll, so the ceiling bites only when the store cannot be read.
- **With `autoRefresh: false` nothing renews it.** Such a host calls `RefreshAsync` itself, more often than `MaxSnapshotAge`, or every guarded query is refused once the ceiling passes.
- The ceiling measures from the provider load that a context pinned. A context kept longer than `MaxSnapshotAge` is refused even after the provider refreshes, so prepare one context per request.
- **Store-only denials:** under StaticOnly, a denial held only in the store is not applied while degraded.
- **Unreachable store:** `PrepareAsync` for a caller with a User identity throws, in every mode.
### Wiring a store
```csharp
var options = new DwPolicyOptions(); // or new DwPolicyOptions().Bind(configuration.GetSection("DynamicWhere:Policies"))
options.Entities.Expose<Employee>("Employee");
var store = new InMemoryPolicyStore(options.Entities.Resolve);
var provider = await StorePolicyProvider.CreateAsync(store, options);
DwPolicy.Configure(options, provider); // freezes options; a second call must ask for this posture
builder.Services.AddSingleton<IDwPolicyStore>(store); // admin API reads
builder.Services.AddSingleton<IDwPolicyWritableStore>(store); // admin API writes
// once per request
DwPolicyContext caller = await DwPolicy.PrepareAsync(new DwPolicyContext().WithSubject(DwSubjectKind.User, userId));
```
- `AddDwPolicies(IConfiguration section, Action<DwPolicyOptions>? configure = null, params IDwPolicyProvider[] providers)` also accepts the provider. It builds its own options instance, and the provider still uses the one passed to `CreateAsync`.
- The companion packages ship no `IServiceCollection` extension. Construct stores, vaults and providers yourself, as above.
---
## 27. Redis package — DynamicWhere.ex.Policies.Redis
```
net6.0 dependencies: DynamicWhere.ex 3.3.0, StackExchange.Redis 2.8.24 namespace DynamicWhere.ex.Policies.Redis
new RedisPolicyStore(IConnectionMultiplexer redis, string? prefix = null, Func<string, Type?>? resolveType = null)
: IDwPolicyWritableStore
new RedisTokenVault(IConnectionMultiplexer redis, string? prefix = null) : IDwTokenVault
new RedisTokenVault(IConnectionMultiplexer redis, byte[] key, string? prefix = null, bool retireUnkeyed = false) (3.3.0)
string GetOrCreate(string scope, string value) int CachedCount void ClearCache()
null redis -> ArgumentNullException; the multiplexer is not owned and never disposed
null key -> ArgumentNullException; a key under DwToken.MinimumKeyLength (16) -> ArgumentException
```
Key layout, from the internal `RedisPolicyKeys` (prefix trimmed; blank -> `dw:policy`):
```
{prefix}:rules hash ruleId -> rule document JSON broad rules
{prefix}:user:{key} hash ruleId -> rule document JSON key = NormalizeSubjectKey(subjectKey)
{prefix}:owner hash ruleId -> key of the hash holding the rule
{prefix}:version string counter; absent reads as 0
{prefix}:version pub/sub channel; message = the new version
{prefix}:tokens hash "{scope}:{sha256(value) lowercase hex}" -> 32 lowercase hex token
"hmac:{scope}:{hmac-sha256 lowercase hex}" for a keyed vault (3.3.0); one hash holds
both kinds while a deployment moves from one to the other
```
RedisPolicyStore:
- **Upsert:** `SealedFields.Refuse` first. Then one MULTI/EXEC transaction:
- deletes the rule from the hash named in `owner`, if that hash differs,
- HSETs the document and the owner entry,
- INCRs the version.
- **Upsert failures:** an EXEC that does not commit throws `InvalidOperationException`. After commit it PUBLISHes the version; a `RedisException` from that publish is swallowed and the poll catches up.
- **Delete:** one transaction removes the rule and its owner entry and always INCRs the version, then PUBLISHes.
- **Concurrent writers (3.3.0):** both transactions are conditional on the rule's `owner` entry — equal to the owner just read, or absent where the rule was new. Where a rule lives is read before the transaction that moves or deletes it, so two writers of one rule could read the same answer: the slower one then cleaned up after a copy the faster had already moved and left that writer's copy behind, under a user nobody any longer wrote it for and with no owner entry pointing at it, which no later write or delete could find. The writer that loses the race now gets the `InvalidOperationException` a failed commit always raised, whose message ends "Another writer moved or removed the same rule in the meantime; write it again."; write it again.
- **LoadAsync:** reads the version before the rules. A document it cannot read throws `ArgumentException`.
- **LoadNarrowAsync:** one HGETALL per identity, skipping blank identities. Keys are lower-case, so casing never splits one user into two.
- **WatchAsync:** subscribes to the channel. The provider still polls `{prefix}:version`.
- **Shared Redis:** two applications on one Redis need different prefixes, for rules and for tokens.
RedisTokenVault:
- **Cache:** every mapping it resolves is cached in the process, unbounded. Tokens never change, so the cache cannot go stale.
- **Minting:** HSETNX, then HGET the winner after a lost race, so instances agree on one token.
- A field that disappears between the two calls throws `InvalidOperationException`.
- A `RedisException` propagates.
- **Query path:** Redis calls are synchronous and happen on a cache miss, during the query.
- **Protection:** the hash stores the scope in clear and a digest of the value, never the value itself. Unkeyed that digest is a plain SHA-256, and a tokenized column is nearly always drawn from a space small enough to hash whole, so a dump of the hash is a dump of the column. Guard it like the column it protects, and give the vault a key: under one the field is an HMAC, and the hash and the key have to be taken together.
- **Keyed (3.3.0):** `new RedisTokenVault(redis, key, prefix, retireUnkeyed)`. The key is held where the hash is not — configuration or a secret manager — and every instance sharing the hash takes the same one.
- **Adoption:** a value with no keyed field is looked up under its unkeyed field too, and the token found there is the one written under the keyed field, so every token already issued is kept. Both fields are read in one `HMGET`, so a value new to the store costs two round trips instead of one; a value already keyed still costs one, and a cached one none.
- **Retiring:** `retireUnkeyed: true` `HDEL`s the unkeyed field the first time this vault meets the value, once the keyed field is there, whether this vault wrote it or found it. Leave it false until every instance has the key (section 24).
- Minting under a key uses the same `HSETNX`-then-`HGET` race handling, with the same `InvalidOperationException` when the field disappears between the two calls.
```csharp
IConnectionMultiplexer redis = await ConnectionMultiplexer.ConnectAsync(connectionString);
var options = new DwPolicyOptions { TokenVault = new RedisTokenVault(redis, "myapp:policy") };
options.Entities.Expose<Employee>("Employee");
var store = new RedisPolicyStore(redis, "myapp:policy", options.Entities.Resolve);
DwPolicy.Configure(options, await StorePolicyProvider.CreateAsync(store, options));
```
---
## 28. Entity Framework Core package — DynamicWhere.ex.Policies.EntityFrameworkCore
```
net6.0 dependencies: DynamicWhere.ex 3.3.0, Microsoft.EntityFrameworkCore.Relational 6.0.22
namespace DynamicWhere.ex.Policies.EntityFrameworkCore no raw SQL; any EF Core relational provider; ships no migrations
new EfPolicyStore(Func<DbContext> contexts, Func<string, Type?>? resolveType = null) : IDwPolicyWritableStore
new EfTokenVault(Func<DbContext> contexts) : IDwTokenVault
new EfTokenVault(Func<DbContext> contexts, byte[] key, bool retireUnkeyed = false) (3.3.0)
string GetOrCreate(string scope, string value) int CachedCount void ClearCache()
null contexts -> ArgumentNullException
null key -> ArgumentNullException; a key under DwToken.MinimumKeyLength (16) -> ArgumentException
class DwPolicyDbContext(DbContextOptions<DwPolicyDbContext> options) : DbContext
DbSet<DwPolicyRuleRecord> PolicyRules DbSet<DwPolicyVersionRecord> PolicyVersion DbSet<DwPolicyTokenRecord> PolicyTokens
OnModelCreating applies all three configurations below
sealed DwPolicyRuleConfiguration : IEntityTypeConfiguration<DwPolicyRuleRecord> const string Table = "DwPolicyRules"
sealed DwPolicyVersionConfiguration : IEntityTypeConfiguration<DwPolicyVersionRecord> const string Table = "DwPolicyVersion" const int SingleRowId = 1
sealed DwPolicyTokenConfiguration : IEntityTypeConfiguration<DwPolicyTokenRecord> const string Table = "DwPolicyTokens"
```
All of these are in `DwPolicyConfigurations.cs`; no type is named `DwPolicyConfigurations`.
```
DwPolicyRules Id Guid PK, never generated
SubjectKind string(32) required SubjectKey string(256) SubjectKeyNormalized string(256)
EntityType string(512) required FieldPath string(512) required
Features string(128) required Effect string(32) required
Priority int Enabled bool ValidFrom, ValidTo DateTimeOffset? (written as UTC) Purpose string(128)
Detail string, unbounded: JSON of transform, operator lists, alias, forced, facts; null when none
CreatedBy string(256) CreatedAt DateTimeOffset? (UTC) UpdatedBy string(256) UpdatedAt DateTimeOffset? (UTC)
index (SubjectKind, SubjectKeyNormalized, EntityType) index (EntityType, FieldPath)
DwPolicyVersion Id int PK, never generated (one row, Id = 1) Version long, concurrency token UpdatedAt DateTimeOffset
DwPolicyTokens Key string(512) PK, never generated = "{scope}:{sha256 hex}", or "hmac:{scope}:{hmac-sha256 hex}"
under a key (3.3.0; no schema change — a keyed key is at most 326 characters)
Scope string(256) required, indexed
Token string(64) required CreatedAt DateTimeOffset (written as UTC)
DwPolicyRuleRecord Guid Id string SubjectKind string? SubjectKey string? SubjectKeyNormalized string EntityType
string FieldPath string Features string Effect int Priority bool Enabled
DateTimeOffset? ValidFrom DateTimeOffset? ValidTo string? Purpose string? Detail
string? CreatedBy DateTimeOffset? CreatedAt string? UpdatedBy DateTimeOffset? UpdatedAt
static string? Normalize(string? key) static DwPolicyRuleRecord FromRule(PolicyRule rule) PolicyRule ToRule()
DwPolicyVersionRecord int Id long Version DateTimeOffset UpdatedAt
DwPolicyTokenRecord string Key string Scope string Token DateTimeOffset CreatedAt
```
- Enums are stored as names.
- `ToRule()` goes through the `PolicyRule` constructor, so a bad row throws `ArgumentException` and the load fails.
- `SubjectKey` keeps the key as typed. The narrow load matches on `SubjectKeyNormalized` (lower-case), so database collation never matters.
Your own context:
```csharp
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
modelBuilder.ApplyConfiguration(new DwPolicyRuleConfiguration());
modelBuilder.ApplyConfiguration(new DwPolicyVersionConfiguration());
modelBuilder.ApplyConfiguration(new DwPolicyTokenConfiguration()); // only needed for EfTokenVault
}
// generate the migration in your own project, for your provider
var store = new EfPolicyStore(() => new AppDbContext(appDbOptions), options.Entities.Resolve);
```
The packaged `DwPolicyDbContext` is declared in the package assembly. Set `MigrationsAssembly` to your project everywhere its options are built: the runtime options and any `IDesignTimeDbContextFactory<DwPolicyDbContext>`.
```csharp
var policyDb = new DbContextOptionsBuilder<DwPolicyDbContext>()
.UseNpgsql(connection, sql => sql.MigrationsAssembly("YourProject"))
.Options;
var store = new EfPolicyStore(() => new DwPolicyDbContext(policyDb), options.Entities.Resolve);
DwPolicy.Configure(options, await StorePolicyProvider.CreateAsync(store, options));
```
EfPolicyStore:
- **Contexts:** `contexts` is called once per operation and the store disposes what it returns. Return a new context every time, never a shared one. The model must include the rule and version configurations.
- **Change detection:** `WatchAsync` returns null, so changes arrive only by polling the version row every `RefreshInterval`.
- **Writes:**
- Upsert and delete apply the row change and the version bump in one `SaveChanges`. The version row is created at 1 if missing.
- A `DbUpdateException` is retried on a fresh context, up to 8 attempts, with a delay of 2–11 ms × attempt number, then rethrown.
- Upsert overwrites every column of the row with the same Id. Delete bumps the version even when no row matched.
- **LoadAsync:** reads the version first. The broad load takes rows where SubjectKind ≠ "User", SubjectKind is null, or SubjectKeyNormalized is null, so a malformed User row fails the load instead of disappearing.
EfTokenVault:
- **Contexts:** `contexts` is called only on a cache miss and the context is disposed. The model must include `DwPolicyTokenConfiguration`.
- **Minting:** reads with no tracking, then inserts. On a `DbUpdateException` from a primary-key race it clears the change tracker and reads the winning row. If that row is still missing it throws `InvalidOperationException`.
- **Behaviour:** EF calls are synchronous. It caches in the process and uses the same keys as `RedisTokenVault`.
- **Protection:** unkeyed, a row's `Key` is a plain digest of the value, and a tokenized column is nearly always drawn from a space small enough to hash whole, so reading the table turns every token in every result back into the value behind it. Guard it like the column it protects, and give the vault a key: under one the `Key` is an HMAC, and the table and the key have to be taken together.
- **Keyed (3.3.0):** `new EfTokenVault(contexts, key, retireUnkeyed)`. A value with no keyed row is looked up under its unkeyed key too and keeps the token that row gave it, so every token already issued is kept. It costs one more read for a value new to the store, and in retire mode one more read per first-met value. `retireUnkeyed: true` deletes the unkeyed row the first time this vault meets the value, once the keyed row is there, whether this vault wrote it or found it; a concurrent delete of the same row is not an error. Leave it false until every instance has the key (section 24). No schema change.
---
## 29. ASP.NET Core package — DynamicWhere.ex.Policies.AspNetCore
```
net6.0 dependencies: DynamicWhere.ex 3.3.0, FrameworkReference Microsoft.AspNetCore.App
namespace DynamicWhere.ex.Policies.AspNetCore
```
### MapDwPolicyAdmin
```
static IEndpointRouteBuilder MapDwPolicyAdmin(this IEndpointRouteBuilder endpoints,
Action<DwPolicyAdminOptions>? configure = null)
DwPolicyAdminOptions
string RoutePrefix = "/dw-policies" trailing '/' trimmed
string? ReadPolicy authorization policy: schema, GET rules, explain, simulate, health
string? WritePolicy authorization policy: POST rules, DELETE rules
bool AllowAnonymousAccess = false true: all seven routes, writes included, have no authorization
DwClaimsOptions Claims { get; } builds the caller for schema, explain and simulate
```
- **Validation** happens at the call, before any route is added:
- A blank `RoutePrefix` throws `InvalidOperationException`.
- A blank `ReadPolicy` or `WritePolicy` throws `InvalidOperationException` unless `AllowAnonymousAccess = true`. So calling it with no `configure` throws.
- A null `endpoints` throws `ArgumentNullException`.
- **Authorization:** the host registers both policy names with `AddAuthorization` and runs authentication and authorization middleware. A caller that fails a policy gets 403.
- **Routes** are mapped one by one with the prefix concatenated; there is no `MapGroup`. It returns the builder, not a convention builder.
- **DI, per request:**
- GET /rules needs `IDwPolicyStore`; POST and DELETE need `IDwPolicyWritableStore`. A missing registration gives 501.
- Register the instance your provider reads under both types.
- **Global state:** everything else comes from the static `DwPolicy` (`Options.Entities`, `Resolver`, `StoreProviders`). Only exposed entities can be asked about.
- **JSON:**
- Handler error bodies are `{ "error": "message" }`, and property names are camelCase.
- The library registers no JSON enum converter. Enum members inside schema fields and a `Filter` follow the host's minimal-API JSON options; with default options the test suite posts `Filter` enums as numbers.
- **The caller:** schema, explain and simulate build the caller from the request principal via `http.GetPolicyContextAsync(options.Claims)`. There is no parameter to impersonate another caller.
- With `Claims.AllowAnonymous = false`, an unauthenticated principal throws `InvalidOperationException` there, unhandled.
- A host that authorizes without authenticating must set `Claims.AllowAnonymous = true`.
### Endpoints
```
Method Route Policy Body Success Handler statuses
POST {prefix}/schema Read SchemaRequest 200 schema 400, 404
GET {prefix}/rules?subject= Read — 200 [rule] 501
POST {prefix}/rules Write RuleRequest 200 rule 400, 501
DELETE {prefix}/rules/{id:guid} Write — 204 501
POST {prefix}/explain Read ExplainRequest 200 [explanation] 400, 404
POST {prefix}/simulate Read SimulateRequest 200 simulation 400, 404
GET {prefix}/health Read — 200 health 503 (same body)
```
- An unmapped verb gives 405, a non-GUID id gives 404 (route constraint), and a body that is not JSON gives 400.
- These propagate unhandled:
- the anonymous-principal `InvalidOperationException`;
- a `PolicyException` such as `StoreUnavailable` from schema or explain (`PolicyException` derives from `LogicException`, not `ArgumentException`);
- exceptions from the store's own `UpsertAsync` or `DeleteAsync`, and store I/O errors.
```
SchemaRequest(string? Entity, IReadOnlyList<string>? Paths = null, int? Depth = null)
ExplainRequest(string? Entity, string? Field, IReadOnlyList<string>? Paths = null, int? Depth = null)
SimulateRequest(string? Entity, Filter? Filter)
RuleRequest(Guid? Id, string? SubjectKind, string? SubjectKey, string? EntityType, string? FieldPath,
string? Features, string? Effect, int Priority = 0, bool Enabled = true,
DateTimeOffset? ValidFrom = null, DateTimeOffset? ValidTo = null,
string? Purpose = null, string? Alias = null)
```
### POST /schema — `{ entity, paths?, depth? }`
- **`entity`:** an exposed type's public name (case-insensitive) or its exact `Type.FullName`. An unknown type and an unexposed type both give 404 `No entity named '{entity}'.`
- **`paths`:** navigation roots, as canonical names or aliases; several are allowed per request. It is a body list rather than a query string because a comma is legal inside an alias.
- **`depth`:** levels to walk from each root. Null means `Caps.SchemaDepth`. A value above the cap is clamped, not refused.
- **400:**
- `depth` below 1,
- a blank path,
- a path with at least `MaxNavigationDepth` segments,
- a path naming a value field the caller can see.
- **404:** a path naming nothing, or a value field the caller cannot see: `'{path}' is not a navigation of '{entity}'.`
```
{ entity, entityType, roots[], depth, maxDepth, truncated,
fields[ { path, name, parent, dataType, canWhere, canSelect, canOrder, canGroup, canAggregate, canSegment,
isMasked, allowedOperators, allowedValues, isRequiredInWhere, costWeight, label, description, group, order } ],
nodes[ { path, name, parent, entity, depth, expanded, remainingDepth } ] }
```
- **Top-level fields:**
- `depth` = the levels actually covered.
- `maxDepth` = `Caps.MaxNavigationDepth`.
- `truncated` = `MaxSchemaFields` cut the list short.
- **Field entries:** `name` = the alias, or the path. `parent` = the owning node's path, or null.
- **Node entries:**
- `expanded` = its fields were described.
- `remainingDepth` = further levels beneath it. 0 means nothing to open, and the value accounts for `SchemaCycleLimit`.
- `entity` = the catalogue name, or null.
- A sealed-denied field, such as one marked `[DwDenied]`, is absent.
### GET /rules?subject=Kind[:Key]
- **Source:** reads the store directly (`LoadAsync`, or `LoadNarrowAsync([Key])` for `User`), not the provider's snapshot, so a write shows up at once.
- **What it lists:** enabled rules only, for exposed entity types only, in no order. Validity windows are not applied.
- **No `subject`:** every enabled broad rule. User rules need `subject=User:{key}`; `subject=User` alone returns `[]`.
- **`Kind` alone:** every rule of that kind. A key is compared after `NormalizeSubjectKey`. A kind that does not parse is treated as Custom.
```
rule = { id, subjectKind, subjectKey, level, entityType, fieldPath, features, effect, priority, enabled,
validFrom, validTo, purpose, alias, allowedOperators, requiredOperators,
facts: { label, description, group, order, allowedValues, cost, audit } | null,
createdBy, createdAt, updatedBy, updatedAt }
```
- Enums appear as names and `subjectKey` as typed. `transform` and `forced` are not included.
### POST /rules — `RuleRequest`
```json
{ "id": null, "subjectKind": "Role", "subjectKey": "Support", "entityType": "MyApp.Models.Employee",
"fieldPath": "Position", "features": "Select, Order", "effect": "Deny",
"priority": 10, "enabled": true, "validFrom": null, "validTo": "2026-12-31T00:00:00Z",
"purpose": null, "alias": null }
```
- **Enums** are names: `subjectKind` is Global|Tenant|Role|User|Custom, `features` is comma-separated flag names, `effect` is Allow|Mask|Deny. Digits, blanks or unknown names give 400.
- **`id`:** null creates a rule; an existing id replaces the whole rule.
- **What the body cannot carry:** a transform, operator lists, a forced predicate or facts. Write those with `IDwPolicyWritableStore.UpsertAsync(new PolicyRule(...))`.
- **Audit stamps:**
- `createdBy` and `updatedBy` = `Identity.Name`, else the `ClaimTypes.NameIdentifier` claim, else `sub`.
- `createdAt` and `updatedAt` = now.
- All four are stamped on every POST, a replace included, and any values in the body are ignored.
- **400 `{error}`:** any `PolicyRule` refusal (`ArgumentException`, `FormatException`, `OverflowException`), or a sealed field. The sealed check runs only when `entityType` resolves through `DwPolicy.Options.Entities`.
- **200:** the stored rule, in the GET shape.
- **Refresh:** it does not refresh the provider. Enforcement sees the rule after the next watch message or version poll, or after a `RefreshAsync` call.
### DELETE /rules/{id}
- Returns 204 whether or not the rule existed, and 501 when no `IDwPolicyWritableStore` is registered.
### POST /explain — `{ entity, field?, paths?, depth? }`
- **`entity`, `paths`, `depth`:** same handling and same 400/404 as /schema.
- **With `field`:** matched against this caller's schema fields by `path` or `name`, case-insensitive, and returns a one-element array.
- A field missing from that schema (misspelt, beyond depth, or denied and omitted) gives 404 `'{field}' is not a field of '{entity}' that this caller can use.`
- **Without `field`:** one entry per schema field.
```
[ { field, entityType, name, isSealed,
features[ { feature, effect, decidedBy, level, attributionAmbiguous, tiedWith[], overrode[] } ] } ]
```
- **`features`:** 6 entries, one per feature. When nothing spoke, `effect` is Allow and `decidedBy` and `level` are null.
- **`isSealed`:** an attribute decided at least one feature absolutely.
- **Sources** render as `Rule {id} [Global|Kind:Key]`, or as the attribute's name with ` (sealed)` appended when it is sealed.
- **`attributionAmbiguous`:** `tiedWith` is non-empty, so `decidedBy` names one of several equal sources.
### POST /simulate — `{ entity, filter }`
- **Errors:**
- 404 for an unknown entity.
- 400 `A filter is required.`
- 400 `{error}` when core validation rejects the filter (a `LogicException`, for example a select naming a property the type lacks).
- Under the Strict tier, outside dry run, a name the type lacks is a policy refusal instead (3.1.0): 200 with `wouldRun: false`, `refusal.field` `"*"`, `refusal.reason` null, and the trace naming it.
- **Strict refusals (3.1.0):** `refusal.field` is `"*"` for codes 1–6 and `CapExceeded`, and `refusal.reason` is null for codes 1–6, as on the `PolicyException` (section 30). The `trace` still names the field and the reason.
- **No side effects:** nothing executes and nothing is audited; it runs on a copy of the caller's context.
```
{ wouldRun, filter: Filter | null,
refusal: { code, field, feature, reason } | null, code = PolicyErrorCode name; reason = SourceOrigin
trace[ { field, feature, action, reason } ] }
```
- **A policy refusal** is 200 with `wouldRun: false` and `filter: null`.
- **Otherwise** `filter` is the sanitized copy: denied parts dropped, predicates injected, names made canonical, and, when it sent no orders, the type's `DefaultOrder` added (3.1.0).
### GET /health
```
{ healthy, configured, tier, dryRun, maxSnapshotAge,
stores[ { version, loadedAt, ageSeconds, degraded, lastError } ] } lastError = exception message, or null
```
- `healthy` = no store provider is configured, or none is degraded. It returns 200 when healthy and 503 with the same body when not.
- An `ageSeconds` above `maxSnapshotAge` does not make it unhealthy.
### ClaimsPrincipal to context
```
static class DwClaimsAdapter
static ValueTask<DwPolicyContext> CreateContextAsync(ClaimsPrincipal principal, DwClaimsOptions options, CancellationToken ct = default)
static DwPolicyContext FromClaims(ClaimsPrincipal principal, DwClaimsOptions options) not prepared
static ValueTask<DwPolicyContext> ToPolicyContextAsync(this ClaimsPrincipal principal, DwClaimsOptions options,
CancellationToken ct = default) ClaimsPrincipalPolicyExtensions
static ValueTask<DwPolicyContext> GetPolicyContextAsync(this HttpContext http, DwClaimsOptions options) DwPolicyHttpContextExtensions
DwClaimsOptions
IList<string> UserClaimTypes = { ClaimTypes.NameIdentifier, "sub" }
IList<string> RoleClaimTypes = { ClaimTypes.Role, "role", "roles" }
IList<string> TenantClaimTypes = { "tenant", "tenant_id", "tid" }
IDictionary<string, DwSubjectKind> SubjectClaimTypes claim type -> kind; empty; keys case-insensitive
IDictionary<string, string> ValueClaimTypes context key -> claim type; empty; keys case-insensitive
bool AllowAnonymous = false
string? Purpose = null copied to DwPolicyContext.Purpose
```
- **Two builders:** `CreateContextAsync` is `FromClaims` followed by `DwPolicy.PrepareAsync`. A `FromClaims` context is unprepared, and since 3.1.0 `ApplyPolicy(context)` refuses it with `PolicyContextNotPrepared` whether or not a store is configured: prepare it before querying, or use `CreateContextAsync`. `FromClaims` is for adding to the context before preparing it.
- **Refusals:**
- A null principal or options throws `ArgumentNullException`.
- If `Identity.IsAuthenticated` is not true and `AllowAnonymous` is false, it throws `InvalidOperationException`.
- With `AllowAnonymous = true`, an unauthenticated principal's claims are still read.
- **Subjects:** every claim of every listed type becomes a subject, so a caller keeps all their roles, not just the first. Blank values are skipped and duplicates (case-insensitive) collapse.
- **`ValueClaimTypes`:** the first claim of the mapped type becomes `WithValue(key, value)`. A missing claim leaves the key absent, and a forced predicate that reads it refuses with `MissingContextValue`.
- **`GetPolicyContextAsync`:**
- Builds the context from `http.User`, prepares it with `http.RequestAborted`, and stores it with `http.Features.Set`.
- Later calls in the same request return that same instance and ignore `options`.
### Audit middleware
```
app.UseDwPolicyAudit(); static IApplicationBuilder UseDwPolicyAudit(this IApplicationBuilder app)
new DwPolicyAuditMiddleware(RequestDelegate next, ILogger<DwPolicyAuditMiddleware>? log = null) Task InvokeAsync(HttpContext http)
```
- **When it runs:** after the rest of the pipeline, in a `finally`, so also when the request threw. It then drains the request's pending audit events.
- **How long it has:** the drain does not use the request's abort token. It has a budget of its own, thirty seconds, which the caller cannot cancel and a hung sink cannot outlast (3.3.0). A client that closed the connection, as the rows arrived or the moment they had, used to cancel the write that follows the response: the sink threw, the middleware logged it, and the events went with the context — an audited read with nothing written down, for the price of a socket.
- **Which context:** only the `DwPolicyContext` in `http.Features`, which `GetPolicyContextAsync` stores there. For a context you built yourself, call `http.Features.Set(context)`. With no context or no pending events it does nothing.
- **Sink:** `IDwAuditSink`, resolved from `RequestServices` (so a scoped sink works), written through `DwPolicy.DrainAuditAsync`.
- **Failures:**
- No sink registered: logs a warning and discards the events. The warning ends "Register one, or stop recording them: remove [DwAudit] from the fields that produced them, or turn off DwPolicyOptions.AuditRefusals." (3.1.0).
- A throwing sink: logs an error and does not rethrow. The unwritten events stay on the context and are lost with the request.
- A sink that does not finish inside the thirty-second budget is cancelled, and the cancellation is logged and dropped the same way (3.3.0).
- The request's own exception is never replaced.
- **Events:** the middleware records nothing itself. Events come from audited fields (`[DwAudit]`, or a rule's `facts.audit`), one `DwAuditEvent` per use, and, with `DwPolicyOptions.AuditRefusals` (3.1.0), one per refused guarded query, with `OccurredAt EntityType FieldPath Feature Effect Subjects Purpose Tier DryRun ErrorCode`.
- **Placement:** register it before anything that touches audited fields. The source recommends placing it before authentication and routing.
---
## 30. Policy error codes
```
PolicyException : LogicException namespace DynamicWhere.ex.Exceptions
PolicyException(PolicyErrorCode errorCode, string fieldPath, PolicyFeature feature, DwTier tier)
ErrorCode : PolicyErrorCode branch on this
Code : string ErrorCode.ToString(), for logs and JSON
FieldPath : string the name the caller wrote; "*" for the whole request, including the
PolicyContextNotPrepared ApplyPolicy raises; under the Strict tier "*" for
codes 1–6, CapExceeded and MissingContextValue too (3.1.0), and for
AmbiguousGroupKey, TransformRequiresMaterialization, MissingHashSalt and
MissingTokenVault (3.3.0); the entity's
short type name for PolicyRequired, StoreUnavailable and a store provider's
PolicyContextNotPrepared;
the transformed paths joined by ", " for TransformRequiresMaterialization and
AmbiguousGroupKey outside the Strict tier
Feature : PolicyFeature None for ApplyPolicy's PolicyContextNotPrepared; All for a store provider's refusals
Tier : DwTier the tier in force; always Strict for PolicyRequired, StoreUnavailable and a store
provider's PolicyContextNotPrepared (ApplyPolicy's carries the configured tier)
RuleId : string? { init; } the rule that decided, when exactly one source decided; null for codes 1–6
under the Strict tier (3.1.0)
SourceOrigin : string? { init; } the attribute, rule, cap or reason that decided, e.g.
"MaxPageSize cap (1000), request had 1001"; null for codes 1–6 and
MissingContextValue under the Strict tier (3.1.0), and for
AmbiguousGroupKey, MissingHashSalt and MissingTokenVault (3.3.0).
TransformRequiresMaterialization keeps its origin, which names the method
and what to call instead, never a field
Message "<Code>: field '<FieldPath>', feature '<Feature>', tier '<Tier>'."
```
`PolicyErrorCode` (namespace `DynamicWhere.ex.Policies.Enums`) numbers are fixed; new members are only appended.
A refusal throws from the method called on the guarded handle; the exception carries no trace (run the request
through `PolicySimulator` or in dry run to see the decisions).
```
# Name Raised when In dry run
1 FieldDeniedForWhere a Where, Having or Segment condition uses a field denied for Where recorded
(both tiers; under Strict a Segment condition gets code 6 instead)
2 FieldDeniedForSelect Strict: a selected field, or a navigation with a denied field beneath recorded
it, is denied for Select; any tier: "a.b" when its key "a.Id" is denied
3 FieldDeniedForOrder Strict: an order field is denied for Order (Convenience drops it) recorded
4 FieldDeniedForGroup a group field is denied for Group (both tiers) recorded
5 FieldDeniedForAggregate an aggregate field is denied, or is transformed without AllowAggregate recorded
on every stage (both tiers)
6 FieldDeniedForSegment a field denied for Segment is used in a Segment (both tiers); Strict: recorded
a Segment condition on a field the caller may not Select, and (3.1.0)
every other field refusal inside a Segment, codes 1–5 included
7 AllSelectsDenied Convenience dropped every requested select (FieldPath "*") recorded
8 OperatorNotAllowed an operator outside the field's allowed set ([DwOperators], rules) recorded
9 CapExceeded MaxPageSize, MaxConditions, MaxConditionDepth, MaxConditionSets, recorded, except the
MaxConditionValues, MaxAggregates, MaxOrderFields or MaxNavigationDepth audit buffer: throws
exceeded; or the context already holds MaxAuditEvents undrained events,
which under Strict outside a dry run is refused as a field instead. The
buffer is also reached on the way out, by an audited member no path names
(3.3.0, section 22): the rows are withheld, FieldPath is the path under
Convenience and "*" under Strict
10 PolicyRequired a DynamicWhere extension method called on a throws
[DwEntity(RequirePolicy = true)] type outside ApplyPolicy
11 RequiredFilterMissing a [DwRequireWhere] field has no AND-reachable condition using one of recorded
its operators
12 MissingContextValue a [DwForceWhere] ContextValue key is absent or null in the context; recorded
under Strict FieldPath "*" and no SourceOrigin (3.1.0)
13 AmbiguousFieldName a name could mean two fields: an alias equal to a path or another alias. throws
Convenience and dry run only: under Strict the name is refused as an
unknown one, with the clause's own code (3.3.0)
14 QueryStringDenied Strict tier and getQueryString: true (FieldPath "*") recorded
15 AmbiguousGroupKey two summary rows share their key values once transformed; under Strict throws
FieldPath "*" and no SourceOrigin (3.3.0)
16 TransformRequiresMaterialization SelectDynamic, FilterDynamic, Group or Summary on the guarded handle throws
for a type whose values are transformed on the way out for this caller:
one the policy names a transformed path on, or (3.3.0) one a row of which
can hold a member declaring a transform attribute anywhere in what the
type can reach. Under Strict FieldPath is "*" rather than the transformed
columns (3.3.0), and so it is in both tiers where the policy names no
transformed path; the origin, which names the method and what to call
instead, stays
17 StoreUnavailable a FailClosed store provider is degraded, or the context's pinned throws
snapshot is older than MaxSnapshotAge
18 PolicyContextNotPrepared ApplyPolicy(ctx) sees a context that never went through PrepareAsync throws
(3.1.0, store or no store), a store provider sees one it never prepared,
or one that gained a User subject after it was prepared
19 QueryCostExceeded the request's total field cost is above MaxQueryCost (FieldPath "*"); recorded
under Strict checked after the field gates (3.1.0)
20 GroupTooSmall the caller used the reserved name "__dwGroupSize" as an alias, Having throws
field or order field. A small group never raises it: it is suppressed
21 MissingHashSalt MaskStrategy.Hash while DwPolicyOptions.HashSalt is empty; under Strict throws
FieldPath "*" and no SourceOrigin (3.3.0)
22 MissingTokenVault MaskStrategy.Tokenize while DwPolicyOptions.TokenVault is null; under throws
Strict FieldPath "*" and no SourceOrigin (3.3.0)
```
"recorded" means dry run writes a `Denied` decision to the trace and runs the query anyway.
Under the Strict tier (3.1.0) codes 1–6 name no field and no source, so a denied field, an alias and a name that
matches nothing on `T` answer with the same exception and the same message. Outside dry run, a condition, select,
order, group or aggregate path that matches nothing is refused with the code of its clause, as a `[DwDenied]` field
would be, instead of `ConditionMustHasValidFieldName`; inside a `Segment` every one of them is `FieldDeniedForSegment`.
`MissingContextValue` names no field and no source either. The trace keeps the field, and so does the refusal event
`AuditRefusals` writes (section 22). The Convenience tier still names the field, with `RuleId` and `SourceOrigin`.
The library maps nothing to HTTP. `PolicyException` derives from `LogicException`, so catch it first and keep
deployment faults apart from caller refusals:
```csharp
catch (PolicyException ex) when (ex.ErrorCode is PolicyErrorCode.StoreUnavailable
or PolicyErrorCode.PolicyContextNotPrepared
or PolicyErrorCode.MissingHashSalt
or PolicyErrorCode.MissingTokenVault
or PolicyErrorCode.PolicyRequired
or PolicyErrorCode.TransformRequiresMaterialization)
{
return Results.Problem(ex.Code, statusCode: 500); // misconfiguration or a dependency is down
}
catch (PolicyException ex)
{
return Results.Json(new { error = ex.Code, field = ex.FieldPath }, statusCode: 403); // the policy refuses
}
catch (LogicException ex)
{
return Results.BadRequest(new { error = ex.Message }); // malformed request: an error string of section 8
}
```
---
## 31. Policy recipes and inference channels
### Recipes
- **Tenant boundary:**
- `[DwForceWhere(Operator.Equal, ContextValue = "TenantId")]` on the column.
- `.WithValue("TenantId", tenantId)` on the context.
- `[DwEntity(RequirePolicy = true)]` on the class.
- Optionally `[DwNoWhere]` on the column.
Every query becomes `(caller's group) AND TenantId = x`; a missing value throws `MissingContextValue`.
- **One tenant or none** (3.1.0), such as a system role no institution owns:
`[DwForceWhere(Operator.Equal, ContextValue = "TenantId", AllowNull = true)]` on `int? InstitutionId`.
Every query becomes `(caller's group) AND (InstitutionId = x OR InstitutionId IS NULL)`; a missing value
still throws `MissingContextValue`.
- **Soft delete:** `[DwForceWhere(Operator.Equal, Value = "false")]` on `IsDeleted`, or
`[DwForceWhere(Operator.IsNull)]` on `DeletedAt`.
- **Confirm an identifier, never search or read it** (support agent):
`[DwOperators(Allow = new[] { Operator.Equal, Operator.In })]` +
`[DwMask(MaskStrategy.Partial, KeepEnd = 4)]` + `[DwNoOrder]`.
- The output is `************4242`, and an exact-value filter still matches.
- Without `[DwOperators]`, `StartsWith` plus `TotalCount` sweeps the values.
- Without `[DwMask]`, the value is returned.
- Without `[DwNoOrder]`, sorting ranks the real values.
- **Bands, never values** (analyst):
`[DwGeneralize(GeneralizeMode.Round, Step = 5000, AllowAggregate = true, MinGroupSize = 5)]` +
`[DwNoOrder]`. Values come back rounded, and aggregates only for groups of 5 or more.
- **Join separately run exports on a hidden, high-entropy ID:** `[DwMask(MaskStrategy.Hash)]` +
`[DwNoOrder]`, with the same `HashSalt` (16+ characters) in every pipeline. No shared store is needed.
Whoever holds the salt can recompute every digest.
- **Low-entropy regulated ID with a right to erasure:** `[DwMask(MaskStrategy.Tokenize)]` + `[DwNoOrder]`
+ a durable vault. To erase, delete the vault entry at `DwToken.KeyFor(scope, value)`, where the scope
is `TokenScope` or the field path — or, where the vault holds a key, at
`DwToken.KeyFor(scope, value, key)` (3.3.0), which only a holder of the key can compute. A vault that has
not retired its unkeyed mappings may still hold one for the value; delete both.
- **One subject across entities:** give every tokenized member that must match the same explicit
`TokenScope`, e.g. `"patient-id"`.
- **Different visibility per role:**
- `[DwDenied]` (sealed) for what no rule may ever grant.
- `[DwMask(..., Overridable = true)]` lets a role rule decide Select and replace the mask stage with
another stage, but never remove it.
- To show one role the raw value, use `[DwMutate(typeof(T))]` and check
`context.Policy.Identities(DwSubjectKind.Role)`.
- **Caller's own names and a filter UI:** `[DwAlias("customer_name")]` +
`[DwDescribe(Label = "Customer", Group = "Identity", Order = 1)]` +
`[DwAllowedValues("Active", "Suspended", "Closed")]`.
- **Refuse unscoped scans, allow scoped ones:** `[DwRequireWhere]` on `Department`.
- **Expensive, sensitive field:** `[DwCost(10)]` + `[DwAudit(PolicyFeature.Select | PolicyFeature.Where)]`.
```
This needs or else
any transform [DwNoOrder] sorting ranks real values; paging reads them
AllowAggregate = true a floor above 1 a group of one returns the exact value
masked but filterable [DwOperators] TotalCount counts matches without selecting
MaskStrategy.Hash HashSalt of 16+ characters MissingHashSalt
MaskStrategy.Tokenize TokenVault MissingTokenVault
tokens that must match one explicit TokenScope the scope follows each field path
any policy [DwEntity(RequirePolicy = true)] a DynamicWhere call without ApplyPolicy reads all
```
### Inference channels
There are eleven. The first eight let a caller learn a value, or what the policy hides, without reading it;
**Forgotten guard**, **Empty store** and **An unrecorded read** are bypasses.
- **Set operations:** `EXCEPT` or `INTERSECT` rebuild a field that is denied for Select but allowed for
Where, from set membership.
- Strict refuses any Segment condition on a select-denied field with `FieldDeniedForSegment`.
- `[DwDeny(PolicyFeature.Segment)]` refuses the field in any Segment, in both tiers.
- **Small-group aggregates:** `MAX`, `MIN` and `SUM` run on real values, so a group of one returns the
exact value. Transformed fields are not aggregatable without `AllowAggregate`, and the group floor
(default 5) removes small groups.
- **TotalCount:** a filter on a protected field counts matches without selecting it. This is inherent to
allowing Where; restrict the field with `[DwOperators(Allow = new[] { Operator.Equal, Operator.In })]`.
- **Sort plus paging:** ordering by a masked field ranks real values, and range filters converge on them.
Use `[DwNoOrder]`; `ValidateModel` warns about every transformed field that can still be sorted.
- **getQueryString:** the SQL names denied columns and the injected predicates. Strict throws
`QueryStringDenied`; Convenience returns the SQL.
- The trace on a result names the same things: `result.Policy` lists the fields a policy dropped, the
attribute or rule that sealed each one, and every injected predicate, and an API that serializes the
result sends it on. Strict leaves it off unless `IncludeTraceInResult = true`; Convenience returns it (3.1.0).
- **The audit cap:** a buffer at `MaxAuditEvents` refuses the next audited use. Until 3.3.0 that refusal carried
`CapExceeded` and an origin naming the cap, while an unknown name — never audited, so never reaching the cap —
carried the ordinary field refusal. Under `Strict` outside a dry run the cap now refuses with the clause's own
field refusal, so the two cannot be told apart; `Convenience` and a dry run still answer `CapExceeded`.
- **The navigation cap on an aliased name:** `MaxNavigationDepth` counts the canonical path, so an alias the caller
wrote as one token was refused with the cap's own code and an origin stating that path's depth, where a name
matching nothing got the clause's refusal. Under `Strict` outside a dry run such a name is refused as an unknown
name is (3.3.0); a caller who wrote the path themselves still meets the cap.
- **A refusal that names a field:** four did under `Strict` until 3.3.0 — an ambiguous name, an ambiguous grouping
key, a clause that cannot be transformed, and a deployment with no hash salt or token vault. An ambiguous name is
now refused as an unknown name is; the grouping key, the hash salt and the token vault name the clause and carry
no origin, and the clause that cannot be transformed names the clause and keeps an origin naming the method and
what to call instead, never a field. `Convenience` and a dry run, either switch, are unchanged, and
`AuditRefusals` still records the real field for all four.
- **An unrecorded read:** until 3.3.0 `[DwAudit]` recorded only a field the request named, so a caller sending no
`Selects` received every audited member of the row with nothing written down. Every audited member a projection
the caller did not name hands back is recorded for `Select` now, one event per query.
- **Which fields exist:** Convenience answers a name that matches nothing with `ConditionMustHasValidFieldName`
and a denied field with a refusal that names it and its source, so a caller can map the hidden columns one
guess at a time. Strict answers both alike, with the clause's `FieldDeniedFor*` code, `FieldPath` `"*"` and
no source (3.1.0), and closes the side doors too: inside a `Segment` every field refusal is
`FieldDeniedForSegment`, a name padded with dots is normalized as a real path is, `MaxQueryCost` is checked
after the gates so a `[DwCost]` weight cannot set a hidden field apart, and `MissingContextValue` names
neither the scope's column nor its context key (section 17).
- **Forgotten guard:** a code path that never calls `ApplyPolicy` reads everything. Use
`[DwEntity(RequirePolicy = true)]`, which covers only DynamicWhere.ex extension methods.
- **Empty store:** a store with no rules still leaves every attribute enforcing.
- **Not closed by anything:** under `Hash` or `Tokenize`, a caller who can write a chosen value and read it
back learns its stand-in. Use `Fixed`, `Null` or a denial when the column need not group or join.
Posture:
- Use `Strict` unless callers need `getQueryString`.
- Keep the default floor.
- Prefer `Tokenize` with a durable vault over `Hash`.
- Run `DwPolicy.ValidateModel` at startup and treat its warnings as a checklist.
- Put `[DwEntity(RequirePolicy = true)]` on sensitive types.
- Restrict operators rather than allowing free filtering on protected fields.
---
## 32. Traps — policies
Read before generating policy attributes.
1. **A transform without `[DwNoOrder]` leaks through sorting.** ORDER BY runs on the stored value, so
paging ranks the real values. `ValidateModel` only warns.
2. **`AllowAggregate = true` needs a floor above 1.** Aggregation runs in SQL before any transform. The
default `Caps.MinGroupSize` of 5 covers it, but `Caps.MinGroupSize = 1` with no per-field `MinGroupSize`
lets `MAX` over one row return that row's value.
3. **What protects a masked but filterable column is `[DwOperators]`, not the mask.** Allow only `Equal`
and `In`, so a caller can confirm a known value but cannot sweep for one.
4. **A forced predicate wraps the caller's group; it never merges into it.** The result is
`(A OR B) AND TenantId = 5`. Do not hand-build the equivalent.
5. **`[DwDenied]` on a navigation denies only that path; `Contact.Email` stays open.** Either decorate the
members of the navigated type, which applies on every path reaching them (up to 4 segments), or deny
each child path. A `"*"` rule denies every field of the entity, and `"Contact.*"` is not a wildcard.
The contrast is a member whose type the framework declares: `Salary.Value`, `Secret.Length`, `Born.Year`,
`Bag.Count`. Nothing can be decorated there, so since 3.3.0 such a path takes the policy of the member it
reads — its denials, operators, cost and audit, never its alias, required filter, forced scope or
description (section 14). A navigation into an application's own type is still a separate field, because its
members can carry attributes of their own.
6. **The default token scope is the field path relative to the queried entity.** Equal member paths on
two entities share tokens, while the same member reached through a navigation gets different tokens.
Set `TokenScope` wherever tokens must or must not match.
7. **Neither `Hash` nor `Tokenize` hides equality.** A caller who writes a value and reads it back learns
its stand-in. `Hash` emits 64 hex characters and `Tokenize` 32, so switching strategies changes the
column width.
8. **Prepare the context once per request.** Since 3.1.0 `ApplyPolicy(context)` raises
`PolicyContextNotPrepared` for a context that skipped `PrepareAsync`, with or without a store; with
attributes alone `PrepareAsync` reads nothing but records that it ran. With a store, adding a `User`
subject after preparation also raises it.
9. **Transforms rewrite the returned instances**, which were loaded `AsNoTracking`. Never attach and save
them. `AsUnguardedQueryable()` output is never transformed.
10. **The group floor removes rows; it never refuses.** It applies to every guarded summary by default
(5). `TotalCount` excludes removed groups, and a result with only small groups is empty.
11. **Policy attributes on public fields are ignored.** `AttributeUsage` allows `Field`, but only
properties are read.
12. **`RequirePolicy` guards only DynamicWhere.ex extension methods.** Plain LINQ or EF on the `DbSet` is
not intercepted.
13. **No runtime rule can unmask a field.** An Allow rule on Select leaves the mask running. A sealed
transform also adds a sealed Mask on Select, so no rule can deny Select either; mark the transform
`Overridable` if a rule must be able to.
14. **`Overridable` does nothing on `[DwOperators]` or `[DwForceWhere]`.** A rule cannot widen a sealed
`[DwAudit]` either.
15. **Aliases rename output columns only in dynamic filter results and summaries.** Typed results keep
member names.
16. **`[DwAudit]` records what the request reads, not only what it names (3.3.0).** A field the request names
and a `DefaultOrder` field the query orders by (3.1.0), plus every audited member a projection the caller
did not name hands back, and every audited member the rows hand back where no path of the policy names it
(sections 14 and 22) — one event per path per query. A default field left out for the caller still records
nothing. Until 3.3.0 a request without `Selects` returned audited columns with no `Select` event at all, so a
deployment upgrading sees more events and can reach `Caps.MaxAuditEvents` where it did not.
17. **`AmbiguousGroupKey` compares only the transformed keys.** A summary mixing untransformed and
transformed keys can be refused even though its rows differ.
18. **`[DwFormat]` ignores its format for a string value.** `[DwGeneralize(GeneralizeMode.Round, ...)]`
plus `[DwFormat("C0")]` on a string member returns the plain rounded number.
19. **The composable `Group` and `Summary` return a query you materialize yourself.** The floor and the forced
predicates are applied, but nothing transforms or renames those rows, and a type whose values are
transformed on the way out for this caller refuses both with `TransformRequiresMaterialization` — since
3.3.0 including a type whose only transform sits where the policy names no path, on a member a subtype
declares or five segments down (section 13).
20. **`ValidateModel` does not catch a malformed `[DwAlias]` name.** A blank, dotted or `"*"` alias throws
`ArgumentException` on every guarded query of the type, and of any type that navigates to it. Since 3.1.0
it does report a malformed `[DwForceWhere]`.
21. **Raising `Caps.MaxNavigationDepth` above 4 reaches past the attribute walk.**
`AttributePolicyProvider.MaxDepth` is a constant 4, so no fragment of the walk names a path of 5 or more
segments. Since 3.3.0 the attributes of the member at the end of such a path are read directly — the deny
family, `[DwOperators]`, the transform stages, `[DwCost]`, `[DwAudit]`, `[DwDescribe]` and allowed values —
by any resolver that reads attributes, and a transform declared there is applied wherever a row carries the
member (section 18), so `Selects` naming it returns it transformed, though it cannot be a grouping key or an
aggregated field. What is
declared about the queried entity itself is not read there, as it is not around a cycle: `[DwAlias]`,
`[DwRequireWhere]`, `[DwForceWhere]`. Until 3.3.0 such a member was allowed and untransformed.
22. **A guarded query over an in-memory source masks your objects.** `list.ApplyPolicy(ctx).ToList(filter)` without
`Selects` returns the list's own instances and transforms them in place, so the list stays masked afterwards.
EF Core queries run `AsNoTracking` and are unaffected. Query a copy, or send `Selects`.
23. **Dry run returns what the policy would withhold.** Denied fields come back, forced predicates are not applied
(rows outside a tenant scope appear), caps and cost do not refuse. Only transforms still run. Never enable it
for callers who must not see everything.
24. **A store provider renews its snapshot from the poll, not only from a reload.** A poll that reads back the
version being served stamps the load time and clears the degraded flag. With `autoRefresh: false` nothing
renews it: call `RefreshAsync` more often than `MaxSnapshotAge`, or every guarded query starts throwing
`StoreUnavailable`. A context pinned before a renewal is refused either way — prepare one per request.
25. **`new PolicyResolver(...)` does not include `AttributePolicyProvider`.** With the four-argument `ApplyPolicy`,
`PolicySimulator` or `PolicySchemaBuilder`, attributes are ignored unless the list contains
`new AttributePolicyProvider()`. `DwPolicy.Resolver` always contains it.
26. **`ApplyPolicy` works without `DwPolicy.Configure`** — on a frozen default: Convenience tier, floor 5, no salt,
no vault, no store. A startup path that forgets `Configure` silently runs the weaker tier.
27. **`DwPolicy.ValidateModel` throws when any error exists** (`InvalidOperationException` listing all of them), so
code that checks `report.Errors` afterwards never runs. Use `PolicyModelValidator.Inspect` to get the report
without throwing; log its `Warnings`.
28. **An existing `catch (LogicException)` also catches policy refusals and deployment faults** (`StoreUnavailable`,
`MissingHashSalt`, `MissingTokenVault`, `PolicyContextNotPrepared`). Catch `PolicyException` first (section 30).
29. **Guarded dynamic and summary rows become `ExpandoObject`** when an alias renames a column or the group floor
applied (every guarded `ToList(Summary)` with the default floor). System.Text.Json writes their keys as they
are (`"Name"`, `"dept"`), not in the camelCase of the envelope.
30. **`AddDwPolicies` builds its own options instance.** A `StorePolicyProvider` reads `StoreFailure`,
`MaxSnapshotAge` and `RefreshInterval` from the options given to `CreateAsync`; with a store, bind and configure
by hand (section 12).
31. **POST /rules cannot write transforms, operator lists, forced predicates or facts.** Its body has no field for
them. Write such rules with `IDwPolicyWritableStore.UpsertAsync(new PolicyRule(...))`.
32. **A stored rule can throw on every query.** No store and not POST /rules validates a rule's alias or its forced
predicate's field; a bad one is saved and then throws `ArgumentException` on the query path for every caller it
applies to. Build rules in a test with `new PolicyRule(...)` and `ToFragment()` first.
33. **Under the Strict tier `result.Policy` is null.** Since 3.1.0 a guarded result carries the trace only when
`DwPolicyOptions.IncludeTraceInResult` allows it, and null follows the tier: off under Strict. Read
`PolicyQueryable<T>.LastTrace` in-process. Setting the option to true sends the dropped fields, the attributes
that sealed them and every injected predicate to whoever reads the result.
34. **Under the Strict tier a misspelt field is a policy refusal, not a validation error.** Since 3.1.0 a path that
matches nothing throws `PolicyException` with its clause's `FieldDeniedFor*` code, exactly as a denied field
does, instead of `LogicException("ConditionMustHasValidFieldName")`; under the section 30 mapping that is a
403, not a 400. Its `FieldPath` is `"*"` and `RuleId` and `SourceOrigin` are null, as on every such refusal,
so never build a message or a log line from them; a `PolicySimulator` trace and the `AuditRefusals` event
name the field.
35. **`AllowNull = true` widens the rows, not the caller, and never satisfies `[DwRequireWhere]`.** A context
without the value is still refused with `MissingContextValue`. The injected
`(field op value OR field IS NULL)` is an `Or`, not a narrowing condition, so a `[DwRequireWhere]` on the
same member still demands the caller's own filter.
36. **With `AuditRefusals` on, a sink receives refusals beside uses.** An event whose `ErrorCode` is not null
records a refused query: its `Effect` is `Deny`, its `FieldPath` can be `"*"` or a name that matches nothing
on the type, and it is written whether or not any field carries `[DwAudit]`. Branch on `ErrorCode` before
counting an event as an access.
37. **`[DwEntity(DefaultOrder = ...)]` orders only guarded queries, and is never a tiebreak.** Unguarded calls
ignore it. A caller who sends any `Orders` gets exactly those, so rows tied on them can still move between
pages, and an `IQueryable` ordered before `ApplyPolicy`, or by a composed `Order` before `Page`, keeps its own
order; a list sorted in memory before `ApplyPolicy` is not seen as ordered and gets the default, so send the
order with the filter. A projected query takes the default only when its outermost `Select` builds T in an
object initializer assigning every default field a column (3.2.0); a computed value, even one EF Core could
translate, leaves it unordered, and the guarded `Select` composed afterwards never takes it.
A default field the caller may not order by is left out without an error. End every
order meant for paging with a unique field, such as the key.
38. **A `[DwEntity]` on a derived type replaces its base type's.** The attribute allows one per type, and .NET
inheritance hands a derived type its own when it declares one, so the base type's `RequirePolicy` and
`DefaultOrder` are gone rather than merged. `[DwEntity(DefaultOrder = "Id")]` on a subclass of a
`RequirePolicy` type lets a DynamicWhere call on the subclass run without `ApplyPolicy`. Repeat every setting on
the derived type.
---
## 33. Reflection cache
The cache is automatic; tuning it is optional. The query engine and the policy layer look up type
members through one static, thread-safe cache: every Filter, Segment and Summary validation and every
expression build. Nothing has to be registered or called. Without `CacheExpose.Configure` the defaults
apply.
- One cache per process, shared by every DbContext, request and thread.
- It starts empty and is never shared between app instances.
- `CacheExpose` is a static class, so there is nothing to inject.
- A lookup takes no lock and a hit allocates nothing (3.3.0). The configuration in force is read with one
volatile read rather than locked and copied on every lookup; `GetCacheConfigOptions()` still returns a copy,
because an instance a caller edited would be the one in force.
```
Store (CacheMemoryType) Key Value Holds
TypeProperties Type Dictionary<string, PropertyInfo> public instance properties, keys OrdinalIgnoreCase
PropertyPath (Type, string) string raw path in, declared-casing path out; successes only
CollectionElementType Type Type? element type; null when not a recognized collection
```
- It holds only these three stores: no compiled expressions, no LINQ strings, no query results.
- The policy layer has its own static caches: attribute fragments per type, `[DwMutate]` transformer
instances, and compiled getters and setters. `CacheOptions` does not size them, and `CacheExpose`
neither reports nor clears them.
- A store is filled on the first lookup that misses. `WarmupCache` fills stores ahead of traffic.
```
DynamicWhere.ex.Optimization.Cache.Source CacheExpose every other class in this namespace is internal
DynamicWhere.ex.Optimization.Cache.Config CacheOptions
DynamicWhere.ex.Optimization.Cache.Enums CacheEvictionStrategy CacheMemoryType
DynamicWhere.ex.Optimization.Cache.DTOs CacheStatistics CacheConfiguration CacheMemoryUsage
CachePerformanceEvaluation CacheMonitoringSession
DynamicWhere.ex.Optimization.Cache.Input HealthAlertsInput CacheFullCheckInput AccessTrackingInput<TKey> MemoryCalculationInput
DynamicWhere.ex.Optimization.Cache.Output CacheCounts TrackingCounts CacheDatabases
```
### Cache enums — verbatim
```
CacheEvictionStrategy FIFO=0 LRU=1 LFU=2 LRU is the default
CacheMemoryType TypeProperties=0 PropertyPath=1 CollectionElementType=2
```
### CacheOptions
```
MaxCacheSize int 1000 > 0 cap per store, not in total
LeastUsedThreshold int 25 1–50 % of a store removed per eviction pass
MostUsedThreshold int 75 50–99 must equal 100 − LeastUsedThreshold; eviction never reads it
EvictionStrategy CacheEvictionStrategy LRU
EnableLruTracking bool true reported only; auto-validation overwrites it
EnableLfuTracking bool false reported only; auto-validation overwrites it
AutoValidateConfiguration bool true correct inconsistencies instead of throwing
Validate() -> void ArgumentOutOfRangeException / ArgumentException; may modify this instance
Clone() -> CacheOptions
```
`Configure` calls `Validate()`, which checks in this order:
1. A value out of range throws `ArgumentOutOfRangeException`, with or without auto-validation.
2. If the thresholds do not sum to 100: with auto-validation, `MostUsedThreshold` becomes
`100 − LeastUsedThreshold`; without it, `ArgumentException`.
3. Tracking flags: with auto-validation they are set from the strategy (FIFO false/false, LRU true/false,
LFU false/true). Without it, a flag that is true for a strategy that does not use it throws
`ArgumentException`.
- Set `LeastUsedThreshold` on its own. The correction only runs in that direction, and an out-of-range
`MostUsedThreshold` throws even though it would have been overwritten.
- With `AutoValidateConfiguration = false`, FIFO and LFU must also set `EnableLruTracking = false`,
because its default of `true` throws.
- There is no `CacheOptions.Default`; the default is `new CacheOptions()`.
### Presets — static factories on CacheOptions, each returning a new mutable instance
```
MaxCacheSize LeastUsed MostUsed Strategy Intended for
new CacheOptions() 1000 25 75 LRU general default
ForHighMemoryEnvironment() 5000 10 90 LRU high memory, conservative eviction
ForLowMemoryEnvironment() 250 40 60 LFU low memory, aggressive eviction
ForDevelopment() 100 50 50 FIFO development and testing
ForHighFrequencyAccess() 2000 20 80 LFU repeated access to the same items
ForTemporalAccess() 1500 25 75 LRU recent-access patterns
```
Every preset sets `AutoValidateConfiguration = true`. The tracking flags keep their defaults until
`Configure` validates the options.
### Configure
```csharp
using DynamicWhere.ex.Optimization.Cache.Config; // CacheOptions
using DynamicWhere.ex.Optimization.Cache.Enums; // CacheEvictionStrategy, CacheMemoryType
using DynamicWhere.ex.Optimization.Cache.Source; // CacheExpose
CacheExpose.Configure(CacheOptions.ForHighMemoryEnvironment());
// or: the action receives new CacheOptions(), not the active options
CacheExpose.Configure(o => { o.MaxCacheSize = 2000; o.EvictionStrategy = CacheEvictionStrategy.LFU; });
// or: adjust a preset
var options = CacheOptions.ForLowMemoryEnvironment();
options.MaxCacheSize = 500;
CacheExpose.Configure(options);
CacheExpose.WarmupCache<Customer>("Contact.Email", "Orders.Items.Sku"); // after Configure
CacheExpose.WarmupCache(typeof(Customer), "Name");
```
- Callable at any time, from any thread, any number of times. Each call replaces the previous options.
- Validation runs before the swap: if it throws, the active options stay. A null argument throws
`ArgumentNullException`.
- A copy is stored, so later edits to your instance do nothing. `GetCacheConfigOptions()` also returns
a copy.
- Each cache call reads the options once when it starts, so calls already running finish on the old ones.
- Existing entries stay.
- A lowered `MaxCacheSize` trims one eviction pass per later miss.
- `ForceEvictionOnAllCaches()` trims immediately.
- There is no `IConfiguration` binding and no DI registration for the cache; configure it in code.
- Configure and Clear act process-wide, parallel tests included. `ClearAllCaches()` keeps the options;
`Configure(new CacheOptions())` restores the defaults.
### Eviction
- Runs only when a lookup misses in that store and the store already holds more than `MaxCacheSize`
entries. The pass runs before the new entry is added, so a store reaches `MaxCacheSize + 1` entries,
or more when misses happen concurrently.
- Removes `max(1, count × LeastUsedThreshold / 100)` entries (integer division), from that store only.
- FIFO removes the first keys in `ConcurrentDictionary` enumeration order. That type keeps no insertion
order, so FIFO does not remove the oldest entries first.
- LRU removes the oldest last-access ticks first. LFU removes the lowest access counts first, breaking
ties by key hash code. Both consider only entries that have a record of their own kind, and delete the
record with the entry.
- An undefined strategy value, or an exception during eviction, falls back to FIFO removing 50% of the store.
### Access tracking
- These calls record an access before they look anything up: `GetTypeProperties`, `FindProperty`,
`GetCollectionElementType` and `IsCollectionType`. `ValidatePropertyPath` records one only once the path has
validated, so a path that fails validation records nothing (3.1.0). The query engine's own calls count too.
- Under LRU a read refreshes the entry's last-access time, `DateTime.UtcNow.Ticks`, once it is a second old
rather than on every read (3.3.0): eviction only asks which entries are oldest, and an entry read a moment ago
is already among the newest. LFU adds 1 on every read, and FIFO records nothing. Only `EvictionStrategy`
decides this; the `Enable*Tracking` flags have no runtime effect.
- A record is written even when the key never enters the store, such as an internal field-type lookup.
Eviction only deletes records of entries it removes, so `TrackingCounts` can exceed `CacheCounts`.
- Fixed in 3.1.0: `ValidatePropertyPath` recorded the access before validating, so under LRU (the default) or
LFU every distinct invalid field name a caller sent stayed recorded for the life of the process, or until
`ClearCache` / `ClearAllCaches`. A caller sending unique invented names grew the process without limit, faster
under the Strict tier, which resolves every unknown name of a request. A failed path now leaves no record.
- Changing strategy leaves the old records in place. An entry with no record for the current strategy is
never evicted. Example: an entry cached under FIFO and not read since the switch to LRU. Warm up after
`Configure` for this reason.
- Statistics, counts, reports and alerts do not record accesses.
### CacheExpose — every public member
```
Configuration
Configure(CacheOptions options) -> void
Configure(Action<CacheOptions> configureOptions) -> void
GetCacheConfigOptions() -> CacheOptions a copy
Reflection — reads through the cache, fills it, records an access
GetTypeProperties(Type type) -> Dictionary<string, PropertyInfo>
FindProperty(Type type, string propertyName) -> PropertyInfo?
IsCollectionType(Type type) -> bool
GetCollectionElementType(Type type) -> Type?
ValidatePropertyPath(Type rootType, string propertyPath) -> string
WarmupCache<T>(params string[] commonPropertyPaths) -> void
WarmupCache(Type type, params string[] commonPropertyPaths) -> void
Statistics and reports
GetCacheStatistics() -> CacheStatistics
GetCacheConfiguration() -> CacheConfiguration
GetMemoryUsage() -> CacheMemoryUsage
EvaluatePerformance() -> CachePerformanceEvaluation
CreateMonitoringSession() -> CacheMonitoringSession
GenerateHealthAlerts(HealthAlertsInput input) -> List<string>
GenerateMonitoringReport() -> Dictionary<string, object>
GeneratePerformanceReport() -> string
GenerateCompactStatusReport() -> string
GenerateCacheAnalysisReport() -> string
GetQuickHealthSummary() -> string
Management
ClearAllCaches() -> void
ClearCache(CacheMemoryType cacheType) -> void
ForceEvictionOnAllCaches() -> void
GetCacheCounts() -> CacheCounts
GetTrackingCounts() -> TrackingCounts
IsCacheFull(CacheMemoryType cacheType) -> bool
IsCacheFull(CacheFullCheckInput input) -> bool
IsEvictionNeeded(CacheMemoryType cacheType) -> bool
CalculateEvictionCount(int currentCacheSize) -> int
GetEvictionStrategyDescription() -> string
Utilities
FormatBytes(long bytes) -> string
GetMemorySizeConstants() -> Dictionary<string, long>
CalculateStringSize(string str) -> long
```
Reflection members:
- `GetTypeProperties` returns the cached dictionary itself, including properties inherited from base
classes. Never modify it. When two names are equal ignoring case, the property reflection lists last wins.
- `FindProperty` looks up one name, case-insensitively. A dotted name returns null.
- `GetCollectionElementType` recognizes arrays, plus generic types whose definition is exactly `List<>`,
`ICollection<>`, `IEnumerable<>`, `IList<>`, `HashSet<>` or `ISet<>`.
- Everything else returns null, including `IReadOnlyCollection<>`, `IReadOnlyList<>`, `Collection<>`,
a class deriving from `List<T>`, and `string`.
- `IsCollectionType(t)` is `GetCollectionElementType(t) != null`.
- `ValidatePropertyPath`:
- throws `LogicException` with message `"FieldPath[<path>]StartsWithReservedName"` when the first segment is one
of the parser's own words (3.1.0, section 5). Checked before anything is looked up, so nothing is cached and
no access is recorded;
- splits on `.`, trims each segment and drops empty ones;
- matches segments case-insensitively, stepping into the element type of a recognized collection;
- returns the declared names joined by `.`, e.g. `" contact . EMAIL"` → `"Contact.Email"`;
- throws `LogicException` with message `"ConditionMustHasValidFieldName"` when a segment is missing.
Failures are not cached, and each raw spelling is stored as its own entry.
- `WarmupCache` caches the type's properties, then validates each path.
- A failing path is skipped silently, and a null array is allowed.
- The first validation of a path also caches every type it walks, and the collection check of each
segment's property type (null for non-collections).
Management members:
- `ClearAllCaches()` empties all three stores and all six tracking dictionaries. `ClearCache` empties one
store and its two tracking dictionaries. Queries then refill the stores as they run.
- `IsCacheFull(CacheMemoryType)` and `IsEvictionNeeded` run the same test: count strictly greater than the
active `MaxCacheSize`, not equal to it. `IsCacheFull(CacheFullCheckInput)` tests against `input.MaxSize`
and returns false when that is ≤ 0.
- `CalculateEvictionCount(n)` is `max(1, n × LeastUsedThreshold / 100)` under the active options.
- `ForceEvictionOnAllCaches()` runs one eviction pass on each store, whatever its size.
- `FormatBytes` returns `"n B"` under 1024, then `"x.x KB"`, `"x.xx MB"`, `"x.xxx GB"`.
- `CalculateStringSize(s)` is `24 + 2 × s.Length`, or 0 for null or empty.
- `GetCacheCounts` and `GetTrackingCounts` only read counts. Every call that returns memory figures walks
every entry of every store: `GetCacheStatistics`, `GetMemoryUsage`, and every report, alert and evaluation.
### Memory figures are estimates
They are computed from fixed 64-bit constants, not measured from the GC. `GetMemorySizeConstants()`
returns these constants:
```
ObjectReference 8 StringOverhead 24 DictionaryOverhead 72 ConcurrentDictionaryOverhead 256 DictionaryEntryOverhead 32
ConcurrentDictionaryEntryOverhead 48 TupleOverhead 24 LongValue 8 PropertyInfoSize 200 NullableByte 1
each store and each tracking dictionary: 0 when empty, otherwise 256 plus
TypeProperties 128 per type + (264 + 2 × name length) per property
PropertyPath 128 + 2 × (input length + output length) per entry
CollectionElementType 65 per entry
tracking dictionary 64 per Type-keyed record; 112 + 2 × path length per path-keyed record
```
### Result types
`*` marks a computed, read-only member. Every type except `CacheMonitoringSession` is a mutable snapshot
that does not update.
```
CacheCounts
int TypePropertiesCount PropertyPathCount CollectionTypeCount TotalCachedEntries*
static FromValues(int typePropertiesCount, int propertyPathCount, int collectionTypeCount) GetSummary() -> string
TrackingCounts
int TypeAccessRecords PathAccessRecords CollectionAccessRecords LRU
int TypeFrequencyRecords PathFrequencyRecords CollectionFrequencyRecords LFU
int TotalLruRecords* TotalLfuRecords* TotalTrackingRecords*
static FromValues(int typeAccessRecords, int pathAccessRecords, int collectionAccessRecords,
int typeFrequencyRecords, int pathFrequencyRecords, int collectionFrequencyRecords) GetSummary() -> string
CacheStatistics
int TypePropertiesCount PropertyPathCount CollectionTypeCount TotalCachedEntries*
int TypeAccessRecords PathAccessRecords CollectionAccessRecords
int TypeFrequencyRecords PathFrequencyRecords CollectionFrequencyRecords TotalTrackingRecords*
long TypePropertiesMemoryBytes PropertyPathMemoryBytes CollectionTypeMemoryBytes
LruTrackingMemoryBytes LfuTrackingMemoryBytes TotalMemoryBytes*
double TypePropertiesMemoryMB* PropertyPathMemoryMB* CollectionTypeMemoryMB*
LruTrackingMemoryMB* LfuTrackingMemoryMB* TotalMemoryMB* 3 decimals
CalculateUtilizationPercentage(int maxCacheSize) -> double mean of the three stores' count ÷ max × 100; 0 when max ≤ 0
CalculateMemoryEfficiency() -> double entries per MB; 0 when TotalMemoryMB is 0
CalculateAverageEntrySize() -> double bytes per entry; 0 when empty
GetMemoryDistribution() -> Dictionary<string, double> percent, 1 decimal; empty when total is 0
GetSummary() -> string
static FromValues(the 14 settable properties above, in that order, as camelCase parameters)
CacheMemoryUsage
long TypePropertiesMemory PropertyPathMemory CollectionTypeMemory LruTrackingMemory LfuTrackingMemory
long TotalMemory* CacheOnlyMemory* TrackingOnlyMemory*
double TotalMemoryMB* CacheOnlyMemoryMB* TrackingOnlyMemoryMB* 3 decimals
GetMemoryDistribution() -> Dictionary<string, double> percent, 2 decimals; empty when total is 0
CalculateTrackingOverheadPercentage() -> double
CalculateCacheEfficiencyRatio() -> double cache ÷ tracking; +∞ when there is no tracking memory
GetLargestMemoryConsumer() -> (string ComponentName, long MemoryBytes)
EvaluateMemoryHealthStatus(double warningThresholdMB = 50.0, double criticalThresholdMB = 100.0) -> string
GetOptimizationRecommendations() -> List<string> never empty
GetDetailedSummary() -> string GetCompactSummary() -> string
static FromValues(long typePropertiesMemory, long propertyPathMemory, long collectionTypeMemory,
long lruTrackingMemory, long lfuTrackingMemory) static Empty()
CacheConfiguration
int MaxCacheSize LeastUsedThreshold MostUsedThreshold
string EvictionStrategy "FIFO" | "LRU" | "LFU"
bool EnableLruTracking EnableLfuTracking AutoValidateConfiguration IsTrackingEnabled*
string EvictionStrategyDescription* MemoryOverhead* "Minimal" | "Low (timestamp tracking)" | "Low (frequency tracking)"
ValidateConfiguration() -> List<string> issues; empty when consistent; never throws
GetSummary() -> string
static FromValues(int maxCacheSize, int leastUsedThreshold, int mostUsedThreshold, string evictionStrategy,
bool enableLruTracking = false, bool enableLfuTracking = false, bool autoValidateConfiguration = true)
CachePerformanceEvaluation
CacheOptions Configuration CacheStatistics Statistics CacheMemoryUsage MemoryUsage
double PerformanceScore List<string> Recommendations List<string> HealthAlerts DateTime Timestamp (UTC)
GetSummary() -> string
CacheMonitoringSession not thread-safe
new CacheMonitoringSession() starts the clock
RecordSnapshot() -> void appends CacheExpose.EvaluatePerformance()
GetPerformanceTrend() -> string first snapshot against last; needs two or more
GetHistory() -> List<CachePerformanceEvaluation> a copy
```
- Distribution dictionaries use the keys `TypeProperties`, `PropertyPaths`, `CollectionTypes`,
`LruTracking` and `LfuTracking`.
- `EvaluateMemoryHealthStatus` returns an emoji icon followed by one of:
- `CRITICAL: {MB:F2} MB (>{critical} MB)` at or above the critical threshold;
- `WARNING: …` at or above the warning threshold;
- `HEALTHY: {MB:F2} MB (<{warning} MB)` otherwise.
- `PerformanceScore` ranges 0–100 and is the mean of four scores:
- `min(100, utilization%)`
- `min(100, entries per MB ÷ 10)`
- `100 − tracking overhead%`
- health at 50/100 MB: 100 healthy, 70 warning, 30 critical
- Each of these adds a recommendation:
- tracking overhead above 30%
- TypeProperties above 60% of memory
- PropertyPaths above 40% of memory
- total above 100 MB, or above 50 MB
- cache ÷ tracking below 2
- when none applies, a single "optimal" line
- `GetPerformanceTrend` reports the first condition that holds:
- score change > +5: improving
- score change < −5: declining
- memory growth > 10 MB: memory increasing
- entries growth > 1000: cache growing
- otherwise: stable
### Health alerts and monitoring data
```
HealthAlertsInput
CacheOptions Config (required) double WarningThresholdMB = 50.0 double CriticalThresholdMB = 100.0
static WithDefaults(CacheOptions config) -> HealthAlertsInput
static Create(CacheOptions config, double warningThresholdMB, double criticalThresholdMB) -> HealthAlertsInput
IsValid() -> bool Config not null, both thresholds > 0, critical > warning
GetSummary() -> string
CacheFullCheckInput
CacheMemoryType CacheType int MaxSize
static Create(CacheMemoryType cacheType, int maxSize) -> CacheFullCheckInput
static FromConfig(CacheMemoryType cacheType, CacheOptions config) -> CacheFullCheckInput MaxSize = config.MaxCacheSize
IsValid() -> bool MaxSize > 0
```
`GenerateHealthAlerts` returns one string per rule that fires. An empty list means no rule fired.
```
total memory ≥ CriticalThresholdMB, else ≥ WarningThresholdMB CRITICAL / WARNING
mean store utilization ≥ 90% of Config.MaxCacheSize WARNING
tracking overhead ≥ 40% WARNING
fewer than 50 entries per MB WARNING always fires on an empty cache
a store's count ≥ 95% of Config.MaxCacheSize WARNING "<store> cache is near capacity", per store
input fails IsValid() one "ERROR: Invalid health alerts input parameters" item, no throw
```
- Build the input from `CacheExpose.GetCacheConfigOptions()`. A bare `new HealthAlertsInput()` has a null
`Config` and returns only the error item.
- Icons in the output are broken:
- alert strings and `GetQuickHealthSummary()` start with a literal `?` or `??`;
- the performance and analysis reports use U+FFFD as bullets and `?` as chart bars.
Match on the words `CRITICAL`, `WARNING` and `ERROR`, never on the icons.
- `GetQuickHealthSummary()` returns `"<icon> <entries> entries, <FormatBytes(total bytes)>"`.
- `GenerateCompactStatusReport()` returns
`"Cache Status: n entries | Utilization: x% | Memory: yMB | Strategy: S | Health: <EvaluateMemoryHealthStatus()>"`.
- `GeneratePerformanceReport()` contains:
- configuration
- entries, utilization, memory and efficiency
- the detailed memory summary
- recommendations
- `GenerateCacheAnalysisReport()` contains:
- each store's count against `MaxCacheSize`
- tracking record counts
- eviction size, and whether each store needs eviction
- memory distribution
`GenerateMonitoringReport()` keys:
```
timestamp DateTime (UTC) cache_strategy string health_status string
total_entries int total_memory_bytes long
total_memory_mb memory_efficiency utilization_percentage tracking_overhead_percentage cache_efficiency_ratio double
type_properties_count property_path_count collection_type_count int
type_access_records path_access_records collection_access_records int
type_frequency_records path_frequency_records collection_frequency_records int
```
### Public types that reach nothing
No `CacheExpose` member accepts or returns these types. Building one does not touch the live cache.
```
AccessTrackingInput<TKey> where TKey : notnull
TKey Key CacheOptions Config ConcurrentDictionary<TKey, long> AccessTimes ConcurrentDictionary<TKey, long> AccessCounts
static Create(TKey key, CacheOptions config, ConcurrentDictionary<TKey, long> accessTimes,
ConcurrentDictionary<TKey, long> accessCounts) IsValid() -> bool
CacheDatabases
ConcurrentDictionary<Type, Dictionary<string, PropertyInfo>> TypePropertiesCache
ConcurrentDictionary<(Type, string), string> PropertyPathCache
ConcurrentDictionary<Type, Type?> CollectionElementTypeCache
ConcurrentDictionary<Type, long> TypePropertiesAccessTime
ConcurrentDictionary<(Type, string), long> PropertyPathAccessTime
ConcurrentDictionary<Type, long> CollectionElementTypeAccessTime
ConcurrentDictionary<Type, long> TypePropertiesAccessCount
ConcurrentDictionary<(Type, string), long> PropertyPathAccessCount
ConcurrentDictionary<Type, long> CollectionElementTypeAccessCount
static FromDictionaries(the nine above, in that order, camelCase) GetCacheCounts() GetTrackingCounts()
AreAllDatabasesInitialized() -> bool
MemoryCalculationInput
the same nine properties
static Create(the nine, in that order, camelCase) static FromDatabases(CacheDatabases databases) IsValid() -> bool
GetCacheCounts() GetTrackingCounts() GetMeasurementSummary() -> string
```
---
## 34. Version history, breaking changes and limits
### History
```
3.3.0 A second host may configure the same posture, a path the query cannot compute is refused rather than run,
a path or a member the attribute walk cannot name is still policed, a malformed request is refused as one
rather than reaching the caller as a server error, a token vault can hold a key, and Clone is public on
Filter, Segment and Summary.
- New. DwPolicy.Configure, and AddDwPolicies with it, take a second call asking for the posture already in
force and do nothing. A different posture still throws. The check is inside the lock that configures, so a
caller needs no IsConfigured check of its own — that check is a check-then-act two hosts can both pass.
Compared: tier, DryRun, IncludeTraceInResult, AuditRefusals, HashSalt, StoreFailure, MaxSnapshotAge,
RefreshInterval, every cap value, the exposed catalogue with every name it answers to and the name each
type is reported under, and the provider types in order. The group floor and the tier's own answer for
IncludeTraceInResult are compared by the value that applies, not by whether somebody wrote them down;
every other cap is compared as written.
Not compared and not replaced: TokenVault, Services and the provider instances, so a second host runs with
the first host's vault, container and rule stores. AddDwPolicies registers the posture in force rather than
the instance it built.
- Behaviour change. Under Strict outside a dry run, a path that exists on the type and names no value the
query can compute — LocalizedText.IsEmpty over two columns, an unmapped getter on the entity — is refused
as an unknown name is, in every clause the database has to compute, where the provider used to throw and
the caller saw a five-hundred. Selects is not one of them: EF Core evaluates the last projection on the
client. Refused only where the producible members are known: the queried type's own model, and the
initializers of a projection EF Core ran before ApplyPolicy, including a member it copies from the entity.
Rows in memory, a framework member such as Length or Year, anything beneath a column, a query a provider
in front of EF Core translates — an expression expander, a decompiler — Convenience and a dry run are
unchanged. The refusal raises no [DwAudit] event, as an unknown name raises none; AuditRefusals and the
trace record it. A simulation has no source, so PolicySimulator and /simulate cannot refuse such a path.
- Behaviour change. LastTrace is assigned before a request is sanitized, so a refused request leaves its own
trace readable instead of the previous request's.
- Security fix and behaviour change. Under Strict outside a dry run, four refusals that named a field name
the clause instead. AmbiguousFieldName said the caller's name matched more than one field, and so at
least one; it is refused as an unknown name is now, with the trace keeping the ambiguity.
AmbiguousGroupKey reported the column behind the caller's alias and an origin saying its values are
transformed; TransformRequiresMaterialization listed every transformed column on the type;
MissingHashSalt and MissingTokenVault named the masked field a deployment had not configured for. The
three report "*" and no origin, and TransformRequiresMaterialization keeps an origin that names the
method rather than a field. Convenience and a dry run, either switch, are unchanged.
- Behaviour change and security fix. Under Strict outside a dry run, a query exhausting the audit buffer is
refused with the clause's own field refusal rather than with CapExceeded and an origin naming the cap. An
unknown name is never audited, so the two answers told a caller which names are real and audited.
Convenience and a dry run are unchanged.
- New. Clone() is public on Filter, Segment and Summary. A deep copy: the condition tree, the projection list,
each order, the page, and a summary's GroupBy and Having. Every node is new; a condition's values are the
caller's own objects, in a new list.
- Security fix and behaviour change. A path the attribute walk cannot name takes the policy of the
member it reads. The pipeline validates, and a provider translates, a path continuing beneath a member
whose type the framework declares: Salary.Value and Salary.HasValue on a decimal?, Secret.Length on a
string, Born.Year and Born.Date.Year on a DateTime, Bag.Count on a dictionary, Lines.Count on an
application's own collection class (the collection's own member, not an element's). No attribute can
be placed there and no fragment named such a path, so it resolved as allowed: a [DwDenied] decimal?
was filtered on, sorted by, grouped by with its values as the group keys, aggregated
(MAX(Salary.Value)) and handed back by a dynamic projection, under Strict; a transformed member gave
its stored value the same way; a [DwAudit] member was read with nothing recorded; a [DwCost] member
cost the default; and a [DwOperators] restriction did not hold. Such a path now takes every fragment
of the member it reads, whichever provider supplied it, an attribute and a store rule on Salary alike:
the deny effects per feature, the operator restriction (intersected), the cost weight and the audited
features. It does not take what is said to the caller about the member: the alias, the required filter
([DwRequireWhere] on TenantId is not satisfied by a filter on TenantId.Value), the forced scope, and
the descriptive facts (label, description, group, order, allowed values). A rule naming the sub-path
itself still applies alongside. One feature is one feature: [DwNoWhere] Born refuses WHERE Born.Year
and still allows GROUP BY Born.Year, and a member nothing denies is read beneath exactly as before, so
Name.Length still runs. A member only a subtype of the navigated type declares is not such a path:
Zone.Parent, where a subclass of Zone's type declares Parent, is decided by the fragments naming it,
so a grant of Zone under a "*" deny does not grant it. Both tiers; present in 3.2.0 and earlier.
- Security fix and behaviour change. Past the attribute walk's depth, the member at the end of the path
is read directly. Caps.MaxNavigationDepth defaults to 4, the depth the walk reads to, and can be
raised; a request naming five or more segments then reached what no attribute fragment covered, and a
[DwDenied] member at segment five was filtered on, grouped by and returned, under Strict. Its
attributes are read now: the deny family, [DwOperators], the transform stages, [DwCost], [DwAudit],
[DwDescribe] and allowed values. What is declared about the queried entity itself is not read there,
as it is not around a cycle: [DwAlias], [DwRequireWhere], [DwForceWhere]. Only a resolver that reads
attributes does this, which every resolver DwPolicy.Configure builds does. Default configuration was
never exposed to this one.
- Behaviour change. Beneath a transformed member there is no member to apply the chain to, Bonus.Value
being a decimal where Bonus is what is rounded, so Select, Group and Aggregate on such a path are
refused: FieldDeniedForSelect, FieldDeniedForGroup and FieldDeniedForAggregate, with a Selects entry
dropped under Convenience as any field denied for Select is. A transformed member past the walk is
itself a member, so it comes back transformed where a row carries it, a typed projection and a
generated row alike: the chains of the members a projection names past the walk are handed to the
outbound walk beside the type's own list. Only Group and Aggregate are refused there, because a
summary's own transform finds a generated row's columns by the type's list, which stops at four
segments. Refusing Select there instead made the projection gate read a masked column five segments
down an Include chain as a denial and leave the whole included navigation out rather than mask the
column. Where and Order follow the member's own decision and run on stored values, as they do along a
named path.
- Security fix. A transform the outbound walk reached no path to is applied to the rows themselves. That
walk transformed along the paths the policy names, the declared types four segments deep, and a value
sitting elsewhere in the materialized rows came back exactly as stored: a [DwMask] member five
segments down an included or in-memory graph (B.C.D.E.Card, while B.C.D.Pin four segments down was
masked), a masked member only a subtype of the row's type declares (Dog.Chip on rows typed Animal, in
memory or in a TPH hierarchy), a masked member of an object a dictionary holds, and the far side of a
cycle, with no Selects, with a navigation named whole in Selects, and in a dynamic projection holding
a real object, at the default caps, under Strict. The rows are walked by run-time type as well now,
and a member that declares a transform attribute and was not transformed along a named path is
transformed by its own attributes, exactly once: an object reached both ways is not transformed twice.
Only members that declare a transform or can lead to one are read, so a navigation whose type can
reach neither a transform nor an audit for Select is never touched and a lazy loader behind it is not
woken, and a model that declares neither anywhere pays for no second pass. A transformed member with
no setter fails the query with InvalidOperationException, as one along a named path does; the trace
records the path with its stages and the note
"(declared on the member; no path of the policy names it)"; no rule can speak to such a
member, since no path names it; a resolver built over no AttributePolicyProvider reads no attribute
here either; and it runs in a dry run, as transforms always have. Default configuration, both tiers.
Still a limit: a member typed object, or a collection that is not generic, says nothing about what it
holds and is not read into.
- Security fix. The attribute walk returned at its depth limit with the type still marked as being
inside it, so a type first met at the fourth segment read as a cycle wherever it was met again in the
same walk, and what a cycle leaves out, [DwForceWhere], [DwRequireWhere] and [DwAlias], was left out
of a shorter path reaching that type directly. Which of two members was declared first decided whether
a forced tenant scope on a navigated type applied. The scope, the requirement and the alias now apply
on every path within four segments that is not around a cycle, as documented. A query that ran
unscoped is scoped, and a required filter may now be demanded.
- Fix. The page offset, (PageNumber - 1) * PageSize, was worked out in 32 bits and wrapped for a large
enough page number: a negative offset is an error on SQL Server and PostgreSQL, so the request became
a five-hundred, and the first page again on SQLite and in memory, so a page far past the last row
returned rows. It is worked out in 64 bits and held to int.MaxValue now, in Page and in the three
summary methods, guarded or not. A page past the last row is an empty page however far past it is,
as it always was for a page number that did not wrap. The policy layer caps PageSize (MaxPageSize) and
never PageNumber, so a guarded query took the same path.
- Security fix. The ASP.NET Core audit middleware drained a request's events with the request's own
abort token, so a client that closed the connection, as the rows arrived or the moment they had,
cancelled the write that follows the response: the sink threw, the middleware logged it, and the
events went with the context, an audited read with nothing written down. The drain has a budget of its
own now, thirty seconds, which the caller cannot cancel and a hung sink cannot outlast.
- Fix. RedisPolicyStore.UpsertAsync and DeleteAsync commit conditionally on the rule's owner entry.
Where a rule lives is read before the transaction that moves or deletes it, so two writers of one rule
could read the same answer: the slower one cleaned up after a copy the faster had already moved and
left that writer's copy behind, a rule applying to a caller nobody any longer wrote it for, with no
owner entry to find it by. The writer that loses the race gets the InvalidOperationException a failed
commit always raised, whose message now says another writer moved or removed the same rule and to
write it again, and should retry.
- Fix. DynamicWhere.ex names Microsoft.Extensions.Caching.Memory 6.0.2, the version patched for
CVE-2024-43483 (GHSA-qj66-m88j-hmgj). EF Core 6.0.22 asks for 6.0.1 or later and 6.0.1 is the last
version open to it, so a host on the EF Core 6 floor resolved a vulnerable version through all four
packages. A host on EF Core 8 or later already resolves a newer one and sees no change.
- Fix. A guarded Summary whose Having carried "conditions": null or "subConditionGroups": null failed
with a NullReferenceException wherever the group floor is on, Caps.MinGroupSize of 2 or more, which
the default of 5 is, although the same summary ran unguarded. A request body sending either overwrites
the list's initializer, and the floor's own walk over Having, which looks for its reserved alias, was
the one reader in the gate that did not check. It runs now, and the floor still applies.
- Performance. A property lookup takes no lock and allocates nothing on a hit. Every lookup of every
query, several per field, locked and copied the cache configuration, allocated an input object for the
tracking call and built a closure whether or not the entry was cached, and under LRU, the default,
wrote the entry's last-access time on every read. One million lookups of one cached member went from
152 ms to 35 ms on one thread and from 2697 ms to 108 ms on eight, and from 167 MB allocated to 22 MB.
A last-access time is refreshed once it is a second old instead (section 33); GetCacheConfigOptions()
still returns a copy.
- Security fix and behaviour change. SelectDynamic, Group, FilterDynamic and Summary on the guarded
handle hand back a query for the caller to run, which the library never sees materialized, so they are
refused with TransformRequiresMaterialization on a type whose values are transformed on the way out.
Whether a type is one was read from the paths the policy names, so a type whose only transforms sit
off them, on a member only a subtype declares, five segments down, or on an object a dictionary holds,
was handed the query and its rows came back exactly as stored: the same gap the outbound walk's second
pass closed for the terminals, one method call away from them. The refusal asks what a row of the type
can hold as well, any transform attribute anywhere in what the type can reach, read only by a resolver
that reads attributes. With no named column to list it names the clause, FieldPath "*", in both tiers,
where under Convenience it otherwise lists the transformed columns. A type nothing transforms anywhere
still gets its query.
- Security fix and behaviour change. [DwAudit] records a member no path of the policy names, once the
rows show it. The gate records a use by path, before the query runs, and a member only a subtype of
the row's type declares, or one past the four segments the attribute walk reads, has no path it could
ask about: handed back inside a row returned whole or a navigation kept whole, it was read with
nothing written down. The outbound walk's second pass reports each audited member it meets where no
path names it, and the terminal records it: one DwAuditEvent per path per query, not per row, Feature
Select, Effect Mask where the member is transformed as well and Allow otherwise, EntityType the
queried type's full name, and FieldPath the path through the rows. Only a member its own [DwAudit]
audits for Select, and only where the projection carries it, since a member the projection left out is
not a read. A member the declared types hold within four segments is the gate's and is left to it, and
so is a path the projection spells out however long it is, so neither is recorded twice. At
Caps.MaxAuditEvents it fails closed as the gate does and the rows are withheld: under Strict outside a
dry run the clause's own refusal with FieldPath "*", FieldDeniedForSegment inside a segment, and
CapExceeded otherwise. Recorded in a dry run too, read only by a resolver that reads attributes, and a
model that declares neither an audit for Select nor a transform anywhere pays for no second pass.
Default configuration, both tiers; a deployment already running the control sees more events for such
models.
- Docs. DwEntity(DefaultOrder)'s remarks said a projected query takes no default; it has since 3.2.0. The
aggregation sections now lead with Caps.MinGroupSize shipping on at 5.
- Fix and behaviour change. A DataType.Number value is read as the expression parser reads it, not as the
host's culture does. The builder writes a number into the expression unquoted, exactly as sent, while
validation checked it with byte/short/int/long/float/double/decimal TryParse in the host's culture, and
the two disagreed: "1,000", "5-", "+5", ".5", "5.", "-.5", "1.e5", "NaN", "Infinity", "-Infinity" and an
integer past UInt64, or below Int64 when negative, all passed validation and then threw the parser's own
ParseException, which a host maps to a server error; "1,5" passed on a de-DE host and was refused on an
en-US one; and "NaN" and "Infinity" were written into the expression as identifiers, so on a type with a
member of that name the condition compared two columns. A value is read in two steps now. First the
parser's own grammar, in the invariant culture and ASCII digits only: optional white space, an optional
minus, digits, an optional fraction with a digit on both sides of the point, an optional exponent; an
integer must fit UInt64, or Int64 when negative, and a real has no bound. Then, in a WHERE condition and
for the operators that write the value into a comparison, whether the literal compares with the member the
condition names — the parser itself is asked, against the member's declared type, a collection at the end
of the path standing for itself. Refused there, where the parser used to throw: a literal written with a
point and no exponent on a nullable integral member, an exponent form or a real longer than a decimal
holds on a decimal, an integer above Int64.MaxValue on a signed integral member, a negative number on
ulong, any number on a string, bool, Guid, DateTime or char member or on a collection of simple values,
and a nullable enum under an ordering operator. A HAVING condition reads the grammar and stops, since an
alias has no member type to ask about. Every refusal is a LogicException with InvalidFormat, the same in
both tiers under a policy, where the gate still answers first. Nothing that ran before is refused now:
every value refused is one the parser refused. Section 4.
- Fix and behaviour change. A null entry in a request's list is a malformed request, refused as one. A body
can say "conditions": [null], "subConditionGroups": [null], "conditionSets": [null], "orders": [null],
"aggregateBy": [null] or "selects": [null]. Nothing read a list expecting that, so the null surfaced
wherever it was first touched: a NullReferenceException from the sort-order check, from the ordering, or,
under a policy, from the copy the sanitizer takes before it reads anything, and an ArgumentNullException
from the name lookup — a server error for a request that was simply malformed. Every method that takes a
shape walks its lists before anything else reads them now, with or without a policy, in both tiers, sync
and async: the composables Where(ConditionGroup), Order(List<OrderBy>), Select, SelectDynamic, Group and
Summary, and every terminal for a Filter, a Segment and a Summary. Under a policy the walk runs at the top
of the sanitizer, before the caps and before the gate, because it is about the request's shape and not a
policy decision. A null condition, sub-group, condition set, order or aggregate is a LogicException naming
the list, ListOf[Conditions]MustNotHasNullEntry. A Selects entry that is null or blank is
ConditionMustHasValidFieldName, the refusal a null or blank GroupBy field has always had. A list that is
itself null still means what it meant; a ConditionSet whose ConditionGroup is null is still an
ArgumentNullException, as is a null Summary.GroupBy; and a null element inside Condition.Values still
reads as "". Clone copies a null entry as a null entry rather than failing on it, so the refusal is the
running method's and reads the same for a copy. Sections 2 and 8.
- New. A token vault can hold a key, so a copy of the store gives no value back. The key a vault stores a
mapping under is a plain SHA-256 of the value, and a tokenized column is nearly always drawn from a space
small enough to hash whole, so a backup, a replica or a dump of the store gives back every value in it,
and with them the value behind every token ever issued. DwToken.KeyFor(scope, value, key) is an HMAC-SHA256
under a key of at least DwToken.MinimumKeyLength, sixteen bytes, over the scope, one zero byte and the
value, written DwToken.KeyedPrefix + scope + ":" + 64 lowercase hex characters, so one value in two scopes
shares no digest; DwToken.RequireKey refuses a key that is null or short and returns a copy of it.
RedisTokenVault and EfTokenVault take the key in a constructor of their own, with a retireUnkeyed switch;
the existing constructors are unchanged and unkeyed, and the unkeyed KeyFor is unchanged.
InMemoryTokenVault draws a random 32-byte key of its own per instance, with nothing to configure and no
API change. A keyed vault adopts the token an unkeyed mapping already gave a value, so every token already
issued is kept, and the unkeyed mapping stays until retireUnkeyed is on: give every instance the key
first, because one still running without it mints a new token for a value whose unkeyed mapping is gone.
No schema change — a keyed key is at most 326 characters against the 512 the Key column holds — and one
more round trip or read for a value the store has not seen. Sections 24, 27 and 28.
3.2.0 A projected row keeps its members, denials the gate could not see are enforced, a default order that
reaches projected rows, and a CancellationToken on every async terminal. The security fixes refuse or
withhold what 3.1.0 returned; the bullets marked "Behaviour change" also change what a correct query
returns; and one call form stops compiling (the token bullet).
- Security fix and behaviour change. With no Selects, a field denied for Select only beneath a member, none at
the top of T, synthesized no projection, so the whole row came back with the denied value in it: in a list or
nested object of a row projected before ApplyPolicy, in a row in memory, and in an entity's included,
automatically included, lazily loaded or owned member. Such a denial now synthesizes the projection when its
value can reach the result, in both tiers, typed and dynamic, for a Filter and a Segment. On an entity that
means beneath a column, an owned or complex member, or a navigation the query loads; a denial beneath a
navigation nothing loads never leaves the database, and the entity is read as in 3.1.0.
- Security fix. Where the query hides what loads, every navigation now counts as loaded: an include named from
the root and re-rooted by Select(o => o.Customer), SelectMany or Join, which EF Core still applies, and a
projection behind another Select (an identity Select, a member of an anonymous row, a conditional). So does
a lazy loader the constructor takes, delegate or ILazyLoader, kept in a field or a property of any name, and
an initializer after a constructor with arguments counts every member as assigned. So do an injected
DbContext and EF Core 7's asynchronous loader delegate. So does a reshaped chain whose lambda hands its rows
an object an application's method returns from the row, or one it captured: another query with its own
include or projection, or an object in memory. Each returned the denied value. A reshaped chain with none of
these is still read from the model, so it is not projected for a denial beneath a navigation it does not
load. A specification, a repository's query, FromSql and a context's Set through an interface are evaluated
as EF Core evaluates them, a context's query function is a query root, and an anonymous object carrying
range variables or a composite key, or a value that only feeds a predicate or a key, builds nothing.
- Security fix. A guarded query through a provider that wraps EF Core's, LinqKit's AsExpandable or
DelegateDecompiler's Decompile, ran tracking: EF Core's AsNoTracking hands such a query back unchanged. The
rows' navigations were then filled from entities the context already tracked, the denied ones included,
and a masked value became a pending change the next SaveChanges would write. AsNoTracking now goes into the
query itself.
- Security fix and behaviour change. A member declared as a base type or an interface holds its subtypes,
whose denied fields the declared type never names. They are read now: the types the EF Core model derives,
for an entity, and every loaded subtype for a projected or in-memory row, an open generic one and an
application's subclass of a framework class included. A query over the root of a hierarchy whose derived
type declares a denied field, an included or named base-typed navigation, a base-typed member of a row, and
a projection constructing a subtype of T, all returned it. Such rows are projected to T and such members
narrowed to the declared type, which drops the subtype's fields, allowed ones too. Over an abstract T the
typed terminals then fail with SelectTypeMustHaveParameterlessConstructor, as for any T they cannot build;
the dynamic terminals return the root's allowed members. A rule on a subtype's field through a base-typed
member was dropped as naming nothing; it is enforced.
- Security fix. A deny-family attribute on an override, on a public member a subtype hides with new, on the
implementation of an interface member (through a variant instantiation too), or on the interface member a
class implements, was read only from its own declaration, so the base type's or the interface's path
filtered, sorted, grouped and returned the value. It applies to the path now.
- Security fix. Under a "*" deny with exact allows, a path the walk never asked about (past four segments,
around a cycle, with no setter, on a subtype) resolved as allowed, so a member holding one was returned
whole, named or not. Each such path is asked of the policy now; one it does not name is denied, and a
member holding one is refused, narrowed or projected as one with a denial beneath it is.
- Security fix. A field denied at the top of T whose type is not a simple value (a blob, a list, an owned
object or a JSON column, say) synthesized no projection either, so with nothing else denied it came back.
- Security fix. The attribute walker read any namespace starting with "System" as the framework's, so an
application's SystemsCorp.Payroll got no fragment beneath its types, and a [DwDenied] field there was
returned, filterable and sortable. Only System and the namespaces beneath it are the framework's now.
- Security fix. Under Convenience, Selects naming a navigation whose element key (Id) is denied narrowed the
key away, and the core's typed projection added it back. Such a narrowing is refused with
FieldDeniedForSelect in both tiers, as naming a sibling of the key already was. A navigation named through
another, Main.Lead, now gates Main's key, which the builder adds; it did not.
- Security fix. Selects naming a member typed as a collection the core does not unwrap (IReadOnlyList<T>,
IReadOnlyCollection<T>, Collection<T> or an application's own) returned every field beneath it, denied ones
included, in both tiers: the projection gate read collections through a narrower list than the attribute
walker. It reads them as the walker does, and a narrowing the core cannot project is refused.
- Security fix. Selects naming a member that carries a field denied for Select no path names returned it:
deeper than four segments, inside a framework generic such as Dictionary<string, T>, or, on a named entity
navigation, in its owned chain or a converted column. What a member carries is read from the source, from
the EF Core model for an entity. Strict refuses it; Convenience narrows it where the core can, and refuses it
where it cannot. A denied property with no setter, and a rule on a path reached through a cycle, are found
beneath a named member too.
- Behaviour change. The synthesized projection keeps what the source carries. A row a projection builds keeps
its assigned nested objects and lists; an entity keeps its columns, converted and JSON ones included, and its
owned and complex members, and every member holding a collection of simple values (byte[], List<string>). In
3.1.0 all of these came back null or empty whenever a field was denied. A member is kept whole when nothing
it can hold is denied, narrowed around a denial where the core's narrowing translates, and otherwise left out
whole with a "left out whole" Dropped decision. An entity's navigations and the objects of a row in memory
are left out, as before, and each one the unguarded call would have returned is recorded as Dropped; a
value EF Core does not map is left out too. A member that can hold an object of any type, a geometry or a
JSON bag say, asks for no projection on its own, and an entity keeps it whole, unless a value converter
hands back its value, directly or inside a complex property: a converter is the application's code, so
such a column is left out. BitArray and the framework's string collections hold values. An application's
own collection class, generic or not, has its own members read; a collection of values stays a value unless
one of them is denied. Two members sharing a name, one hidden with new under another type or spelled in
another case, are left out when either holds a denial, since the core reads one and a row carries both.
Rows in memory are projected when a member a base type declares, and the row type hides with new, is
denied. A projected member is read as the type its initializer constructs, and an
initializer after a constructor with arguments narrows its own bindings. The trace records a member left
out only when a projection is built, or, in a dry run, would be.
- Behaviour change. [DwEntity(DefaultOrder)] applies to a projected source whose outermost Select builds T
in an object initializer assigning every field the default names a column: a mapped member, read directly,
through reference navigations or through EF.Property. A computed value or any other projection still leaves
the query in its own order. A Select, or a Filter with Selects, composed on the guarded handle keeps the rest
of the chain unordered. A composed Filter that sent orders gets no default later in the chain, as a composed
Order already did not.
- Every async terminal has overloads taking a CancellationToken, guarded and unguarded: ToListAsync and
ToListAsyncDynamic with a Filter, ToListAsync with a Summary, and ToListAsync with a Segment. The 3.1
signatures are unchanged, so code compiled against 3.1 still binds. The token reaches the count and the
read. ToListAsync(filter, default) no longer compiles, since default fits both bool and CancellationToken,
and a reflection lookup of one of these methods by name alone finds more overloads than it did.
- Behaviour change. The async Summary counts through EF Core's CountAsync, where it counted synchronously, and
it and ToListAsyncDynamic read through EF Core's ToListAsync instead of Dynamic LINQ's ToDynamicListAsync,
so on an EF Core query a canceled token reaches the database. A provider that is not EF Core's keeps Dynamic
LINQ's read, on the calling thread.
3.1.0 Dates rebuilt, segments combined in the database, five new caps, two new error codes, preparation enforced,
a strict tier that discloses less, declared default orders, forced predicates that admit null, audited
refusals, and long In lists that no longer end the process. The eleven bullets marked "Behaviour change"
change what code written for 3.0.0 does; the others fix what threw or add what was missing.
- Behaviour change. DataType.Date / DataType.DateTime resolve the member's type before building the
predicate. Every comparison on a DateTimeOffset member used to throw, and DataType.Date on any nullable
date member used to throw; both work. The null guard is emitted only where the value can be null:
IsNull / IsNotNull on a non-nullable date member of the entity answer false / true (on a DateTimeOffset
member they threw), and through a navigation they test the navigation.
- Behaviour change. A date value must be ISO 8601, year-first, or a format declared through
DwDates.Configure. A numeric day/month date such as "01/09/2026" throws the new AmbiguousDateFormat;
lenient forms such as "12:00" are InvalidFormat. The server's culture no longer decides anything.
DateTimeOffset values normalise to UTC, and a value with no zone is read as UTC. C# date objects in
Values are written year-first; a DateTime of Kind Local compared under DataType.DateTime with a
DateTimeOffset member is written with its UTC offset, so it filters on the moment it holds. Configure
refuses a declared format whose own text ISO 8601 or a year-first date already reads, such as
yyyy-MM-dd'T'HH:mm:ss'Z': declaring one could only change what such a value means, and off UTC it did.
- DateOnly members can be compared; no comparison on one worked under either date data type (IsNull and
IsNotNull did).
- HAVING on a date alias takes its type from the aggregate, so it works on DateTimeOffset.
- Security fix and behaviour change. A member named Root, It or Parent, and an alias named root, it or
parent, was read by System.Linq.Dynamic.Core as a context keyword. Root.Name and It.Name filtered,
sorted, grouped, aggregated and projected the row's own Name; Parent threw. Under ApplyPolicy that
projected [DwDenied] values, let a filter test a denied column, and applied a [DwForceWhere] scope
reached through such a navigation to the row's own column. Expressions are parsed with a library-owned
ParsingConfig with the context keywords off, and ParsingConfig.Default is no longer read. The words the
parser does keep are refused by name: a path whose first segment is new, iif, np, isnull, is, as, cast,
true, false or null, whatever the letter case, throws LogicException
FieldPath[<path>]StartsWithReservedName in every clause, guarded or not. Seven of them used to raise the
parser's ParseException, True and False an InvalidOperationException, and Null was read as the null
literal, so the query returned no rows and no error. Only a path's first segment is affected, so Owner.New
names the member. Predefined type names such as String, Math and Guid name members too, and always did.
- Behaviour change. ToListAsync(Segment) combines its condition sets into one query the database answers.
Union and Intersect combine the sets' conditions; Except removes its set's rows with NOT EXISTS on the
primary key; a type with no primary key uses SQL UNION / INTERSECT / EXCEPT. The sets used to be loaded
into lists and combined by object reference, so with AsNoTracking(), with Selects, and under ApplyPolicy
(always untracked) Intersect returned nothing, Except removed nothing and Union counted a row once per
set. Ordering, paging, projection and TotalCount now run in SQL exactly as for a Filter: text sorts by
the database's collation, Orders apply before Selects, and only the requested page is read.
- Behaviour change. PageCount on an unpaged result is 1 (0 with no rows), rather than TotalCount, or 0
for a Segment with condition sets.
- DwCaps.DefaultPageSize (default 0 = off) bounds a guarded query that sends no Page.
- Behaviour change for guarded requests. DwCaps.MaxConditionDepth (default 10) bounds how deeply
condition groups nest, and DwCaps.MaxConditionSets (default 10) how many condition sets a Segment
carries. A guarded request nested 11 levels deep, or a segment with 11 or more sets, which 3.0.0 ran,
is refused with CapExceeded unless the deployment raises the cap. Unguarded calls are not capped.
- Behaviour change for guarded requests. DwCaps.MaxConditionValues (default 1000) bounds the values one
condition carries, comparing the largest condition of the where clause, Having and every Segment set: an
In was one comparison per value for the price of one condition and one field. DwCaps.MaxAggregates
(default 50) bounds a summary's AggregateBy entries, on the Summary terminals and the composable Group and
Summary; the group floor's own count is not counted. Both refuse with CapExceeded and FieldPath "*" in
both tiers ("MaxConditionValues cap (1000), request had 1001"), refuse a value below 1, freeze with the
posture and bind from Caps:MaxConditionValues and Caps:MaxAggregates. An aggregate with no field, such as
a Count, is charged DefaultFieldCost toward MaxQueryCost; it was free. Every count cap is now checked
before any name is resolved, so an oversized request that also names an unknown field is refused with
CapExceeded, where 3.0.0 answered ConditionMustHasValidFieldName first. Unguarded calls are not
capped.
- Security fix. In, NotIn, IIn and INotIn on Text, and In and NotIn on Guid, Number and Enum, joined their
values into one flat chain, one level of expression nesting per value. EF Core and the expression compiler
walk that tree recursively, so a condition with about seven hundred values overflowed the request
thread's stack and ended the process, guarded or not; no catch can stop a stack overflow. A list longer
than 32 values is now a balanced tree of flat chains of at most 32 terms. A list of 32 or fewer is written
exactly as before, and the rows returned are the same.
- Behaviour change. ErrorCode.SelectTypeMustHaveParameterlessConstructor replaces the English sentence a
Select on an unconstructible type used to throw; LogicException gained Subject, which carries the type
name.
- Behaviour change. ApplyPolicy(ctx) refuses a context that never went through PrepareAsync, with or
without a store configured. DwPolicyContext.IsPrepared is public.
- Behaviour change. Under the Strict tier a guarded result carries no trace: result.Policy is null unless
DwPolicyOptions.IncludeTraceInResult (bool?, default null = follow the tier: off under Strict, on under
Convenience) is true. PolicyQueryable<T>.LastTrace still holds it.
- Behaviour change. Under the Strict tier an unknown field and a denied field answer alike. Outside dry
run, a path that matches nothing is refused with its clause's FieldDeniedFor* code instead of
LogicException ConditionMustHasValidFieldName; every FieldDeniedFor* refusal carries FieldPath "*" and
no RuleId or SourceOrigin, and every CapExceeded refusal carries FieldPath "*". Inside a Segment every
field refusal is FieldDeniedForSegment, and a name padded with dots is normalized as a real path is.
MaxQueryCost is checked after every field gate, so a [DwCost] weight cannot tell a hidden field from a
missing one. MissingContextValue carries FieldPath "*" and no SourceOrigin.
The trace keeps the field. The Convenience tier and dry run are unchanged.
- [DwEntity(DefaultOrder = "CreatedAt desc, Id")] orders a guarded query whose caller sends no orders,
less the fields that caller may not order by, and in a Segment less the fields denied for segments.
Unguarded calls ignore it, and so does a projected query or one that composed an Order. A field the
default keeps that is audited for Order is recorded as a use, as a caller's own order is; a field left out
is not. PolicyModelValidator reports unreadable entries, fields no query can order by and fields sealed
attributes deny for ordering as errors; unknown fields, fields only overridable attributes deny for
ordering, and fields denied for segments as warnings.
- [DwForceWhere(AllowNull = true)], ForcedPredicate.AllowNull with FromConstant / FromContext overloads
taking bool allowNull, and "allowNull" in a stored rule's forced object inject
(field op value OR field IS NULL). AllowNull on IsNull or IsNotNull is refused by the attribute, by both
factories and by a stored rule, value or no value. PolicyModelValidator now reports every malformed
[DwForceWhere] at startup; it used to surface on the first guarded query of the type.
- DwPolicyOptions.AuditRefusals (default false) writes every refused guarded query to the context's audit
buffer, under the canonical path, cut to 256 characters with control, format, line separator and
paragraph separator characters escaped. DwAuditEvent gains ErrorCode and a constructor taking it.
- StorePolicyProvider renews MaxSnapshotAge on a poll that reads back the version it serves, and that poll
clears IsDegraded. A healthy store nobody wrote to refused every guarded query one MaxSnapshotAge after
the provider last loaded it. A poll whose read was overtaken by a failed refresh or poll reloads instead.
- The composable PolicyQueryable<T>.Group applies the group floor. It went past the summary pipeline and
returned the small groups ToList(Summary) suppresses; it and the composable Summary also handed back the
floor's own __dwGroupSize column, which they no longer do.
- A forced null check built from a context key failed every guarded query on its type: the key was still
required, and its value landed on a null check that validation refuses. ForcedPredicate.FromContext now
refuses IsNull and IsNotNull and points to FromNullCheck, and a stored rule of that shape is refused when
it is read, as [DwForceWhere] already refused a ContextValue on a null check.
- The reflection cache no longer keeps an access record for a field path that fails validation. Under LRU,
the default, or LFU every invented name a caller sent stayed recorded for the life of the process.
3.0.0 Field-level policies (sections 12–32) and three companion packages. Additive for 2.x callers:
- every 2.x signature and the query engine are unchanged;
- FilterResult<T> and SummaryResult gained Policy : PolicyTrace? (null unless guarded), so JSON output
gains "policy": null;
- the core package gained Microsoft.Extensions.Configuration.Abstractions, .Configuration.Binder and
.DependencyInjection.Abstractions 6.0.0;
- PolicyException derives from LogicException, so an existing catch (LogicException) also receives
policy refusals;
- every extension method first checks [DwEntity(RequirePolicy = true)]; types without it are unaffected;
- the group floor (default 5) applies to guarded summaries only.
2.1.5 XML documentation fixes only.
2.1.4 Security fix. Condition values are escaped before they are embedded: a value ending in \ used to throw
ParseException, and a crafted value could close its literal and append predicate logic.
AggregateBy.Alias must be an identifier (AggregationMustHasValidAlias): an alias holding a comma used to
append projection terms.
2.1.3 MIT license; no API change.
2.1.2 Ordering by a path through a collection ("Tags.Value") works — Min ascending, Max descending — where it
threw "No property or field 'Value' exists in type 'List`1'". Added
OrderField[<field>]CannotEndOnCollectionOfComplexElements.
2.1.0 Condition.Values became List<object> (was List<string>): JSON callers are unaffected, C# code assigning a
List<string> no longer compiles. Values are coerced per DataType. Cache presets added.
```
### Limits by design
- `Segment` is async only. On a type with a primary key, `Except` needs a provider that translates a correlated
`EXISTS`. A type with no primary key is combined with SQL `UNION` / `INTERSECT` / `EXCEPT`: it needs the operators
the request uses, and every column to be comparable (section 6).
- `Select<T>` needs a public parameterless constructor on T. Records with only positional constructors, and
classes whose constructors all take arguments, cannot be targets; use `SelectDynamic`.
- The I-variants call `ToLower()` on both sides. On a case-sensitive collation (PostgreSQL with the `C` locale) the
database may not use an index for `LOWER(column)`; add a functional index or use the plain operators.
- Values are inlined as escaped literals, not SQL parameters. EF Core escapes them for SQL, so this is not an
injection path, but every distinct value is a distinct statement and a separate plan-cache entry.
- A path through a collection means "any element matches". There is no `All()` and no negated `Any()`.
- A member whose name is one of the parser's own words — `new`, `iif`, `np`, `isnull`, `is`, `as`, `cast`, `true`,
`false`, `null` — cannot be queried at all. No clause can name it as a path's first segment: the path is refused
with `FieldPath[<path>]StartsWithReservedName` (section 5). Rename the CLR property and map the column with
`[Column]`. Reached through a navigation (`Owner.New`) it is an ordinary member.
- A declared date format may not write text ISO 8601 or a year-first date already reads; `DwDates.Configure`
refuses one (section 4). ISO 8601 and year-first dates are read on every deployment and cannot be turned off.
- A shadow property cannot be named by any clause, guarded or not: a field path names CLR members, and a shadow
property has none, so the name matches nothing and is refused as an unknown name. Map it to a property, or reach
it through a projection that assigns it with `EF.Property`.
- An alias spelled like another member of the same type is reported by `ValidateModel` (3.3.0). A generated row
cannot carry one name twice, so the rename is not applied there and both columns keep their own names.
- A member no database can compute — a getter over columns, or an unmapped getter on an entity — cannot be
filtered, ordered, grouped or aggregated under `Strict` on a source the library can read (3.3.0, section 17). Map
it, or name the columns beneath it. A projection may still name it, because EF Core evaluates the last projection
on the client. Under `Convenience` every clause still reaches the provider and still throws there.
- A column the model maps only on a subtype cannot be filtered, ordered, grouped or aggregated through the base type
(3.3.0, section 17). EF Core translates a member against the type the query is over, so it fails there too: query
the derived type, `Set<Merchant>()`.
- A second host configuring the same posture runs with the first host's `TokenVault`, `Services` and rule store
instances (3.3.0, section 12). They are not compared and not replaced.
- A simulation has no source, so `PolicySimulator` and `/simulate` cannot refuse a path no database can compute;
they show such a request running where the strict query refuses it (3.3.0, section 17).
- A condition's values are read once to validate their format and again to build the predicate. A value whose
`ToString()` answers differently each time is validated as one value and queried as another; pass values that do
not change. No policy decision reads a value's content — only how many there are.
- Summary rows flatten dotted group fields (`CategoryName`); `SelectDynamic` rows nest them (`Category.Name`).
- `Filter` applies `Orders` and `Page` on T before the projection, so order fields need not be selected.
- `getQueryString` needs an EF Core provider.
- Enum filtering works whether the column stores names or numbers: the parser converts the name to the enum value
before EF Core translates it. `Contains` / `StartsWith` / `EndsWith` work only on string members.
- Cache configuration changes are eventually consistent: calls already running finish with the options they read.
- With no `Selects`, a guarded query over an entity leaves out every navigation EF Core does not own once a denial
needs a projection (section 17), an included one too. Under Convenience, name the navigation in `Selects` to get
it narrowed; under Strict, name its allowed fields.
- A forced scope declared on a list's element type filters the rows that hold the list, never its elements. Selects
naming the list returns every element, those the scope excludes included, as in every release; a synthesized
projection leaves such a list out. Scope the elements where the row is built.
- A member typed `object`, a framework interface or a collection that is not generic (`IEnumerable`, `ArrayList`,
`Array`, an application's own) is opaque to the policy. It never asks for a projection; when one is needed anyway,
a projected row, a row in memory, and an entity's converted column leave it out, and an entity's other columns
keep it (what EF Core materializes itself holds no application object); and naming it
returns whatever it holds. A converted column counts as able to hold anything when its type can: `object`, a
`Dictionary<string, object>`, or a type with such a member. `BitArray`, `StringCollection`, `StringDictionary` and
`NameValueCollection` hold values. A value converter that returns an application type through a column typed
`object`, and an unmapped getter typed `object` over a private navigation, are opaque the same way: with nothing
else denied the row comes back as loaded. Type the member as what it holds.
A framework generic holding a policed type (`Dictionary<string, LineDto>`) has no paths beneath it: naming it is
refused in both tiers where the core cannot narrow it (at the top of T, or on a row in memory), narrowed away
under Convenience beneath a navigation, and a synthesized projection leaves it out.
- A projection builds the declared type. A query over the root of a hierarchy whose derived type declares a denied
field comes back as root-type rows, the derived types' allowed fields dropped too; over an abstract root the typed
terminals fail with `SelectTypeMustHaveParameterlessConstructor` and the dynamic ones return the root's members.
Query the derived type (`OfType<T>()`) to keep its fields. Subtypes are read from the assemblies loaded
when the query runs, which hold every type a row can have.
- Under a `"*"` deny, a member is kept whole only when every path beneath it the walk skips, one with no setter,
on a subtype or past four segments, is one the policy names; around a cycle it never is. Otherwise it is
narrowed, left out or refused.
- A member EF Core does not map counts as loaded and is read as its type, since its getter can hand out what EF
Core loaded: a denied field in its type asks for a projection, which leaves the member out, since the projection
cannot assign it. A getter that copies a denied column into a type with no denial is the application's to
withhold.
- Rows in memory can be any loaded subtype, and the policy does not look at the rows. When a subtype of T declares
a denied field, or a subtype or implementation of a member's type does, the rows are projected and their objects
left out, even if no row is that subtype.
- A deny-family attribute on an override, or on a member a subtype hides with `new`, denies the base path for every
subtype loaded, not only those the EF Core model maps: a view model deriving from an entity and overriding one of
its members decides the entity's own path. Declare such a class apart from the entity to keep the path open.
- Types from an unloadable `AssemblyLoadContext` stay referenced by the policy's caches, so the context is not
collected while the process runs.
- A method or property in a reshaping lambda that builds a query from captured values (a repository's query, a
specification) runs once more per guarded read, when the guard reads what it returns; one that returns a different
query on each call is enforced as it answered the guard.
- A provider that wraps EF Core's gets `AsNoTracking` only when its query's expression shows the EF Core root, as
LinqKit's and DelegateDecompiler's do; one that hides it behind its own expression runs tracking.
- A member declared as a framework collection (`IEnumerable<string>`, `List<string>`) that holds an application's own
collection class at run time is read as the framework type, so that class's own members are not read. Serializers
write only the elements.
- On EF Core 6, a reshaped query whose projection EF Core 6 cannot translate (a count over `GroupBy`/`First`, a
`SelectMany` over a captured query that builds its rows) fails guarded, where it ran unguarded; EF Core 7 and later
translate it.
- A default order reaches a projected row only through columns of the entity its `Select` reads: a projection over
an anonymous or other intermediate row takes no default.
- A path beneath a member whose type the framework declares (`Salary.Value`, `Born.Year`) can carry no
attribute of its own. It takes that member's deny effects, operator restriction, cost and audit, and never its
alias, required filter, forced scope or description (3.3.0, section 14); a rule naming the path itself still
applies alongside. Where the member is transformed, such a path cannot be selected, grouped or aggregated at
all: there is no member beneath it to apply the chain to.
- Past the attribute walk's four segments, which only a raised `Caps.MaxNavigationDepth` lets a request name, the
member at a path's end is read for its own attributes (3.3.0, section 14), and what is declared about the
queried entity itself is not read there, as it is not around a cycle: `[DwAlias]`, `[DwRequireWhere]`,
`[DwForceWhere]`. A transformed member there cannot be a grouping key or an aggregated field, because a
summary's own transform finds a generated row's columns by the type's list, which stops at four segments.
- A transform the policy names no path to is applied to the rows by run-time type (3.3.0, section 18). A member
typed `object`, or a collection that is not generic, says nothing about what it holds and is not read into.
Type the member as what it holds.
- A simulation reads T as a source it cannot see into (section 23): every denial beneath a member counts, and a
synthesized `Clause.Selects` keeps only members holding a value.
---
## 35. Worked examples (C#)
### Usings
```csharp
using DynamicWhere.ex.Source; // extension methods
using DynamicWhere.ex.Classes.Core; // Condition ConditionGroup ConditionSet OrderBy PageBy GroupBy AggregateBy
using DynamicWhere.ex.Classes.Complex; // Filter Summary Segment
using DynamicWhere.ex.Classes.Result; // FilterResult<T> SummaryResult SegmentResult<T>
using DynamicWhere.ex.Enums; // DataType Operator Connector Direction Intersection Aggregator
using DynamicWhere.ex.Exceptions; // LogicException PolicyException
```
### Filter
```csharp
var filter = new Filter
{
ConditionGroup = new ConditionGroup
{
Connector = Connector.And,
Conditions =
{
new Condition { Sort = 1, Field = "Department", DataType = DataType.Text,
Operator = Operator.Equal, Values = { "Engineering" } },
new Condition { Sort = 2, Field = "Salary", DataType = DataType.Number,
Operator = Operator.Between, Values = { 50000, 120000 } },
},
},
Orders = new List<OrderBy> { new() { Sort = 1, Field = "HireDate", Direction = Direction.Descending } },
Page = new PageBy { PageNumber = 1, PageSize = 25 },
Selects = new List<string> { "Id", "FirstName", "Department" },
};
FilterResult<Employee> page = await db.Employees.ToListAsync(filter); // whole Employee rows, unselected members default
FilterResult<dynamic> slim = await db.Employees.ToListAsyncDynamic(filter); // rows with Id, FirstName, Department only
```
### Summary
```csharp
var summary = new Summary
{
GroupBy = new GroupBy
{
Fields = { "Department" },
AggregateBy =
{
new AggregateBy { Field = "Salary", Alias = "Total", Aggregator = Aggregator.Sumation },
new AggregateBy { Alias = "Headcount", Aggregator = Aggregator.Count },
},
},
Having = new ConditionGroup
{
Conditions = { new Condition { Sort = 1, Field = "Headcount", DataType = DataType.Number,
Operator = Operator.GreaterThanOrEqual, Values = { 3 } } },
},
Orders = new List<OrderBy> { new() { Sort = 1, Field = "Total", Direction = Direction.Descending } },
};
SummaryResult result = await db.Employees.ToListAsync(summary);
foreach (dynamic row in result.Data) Console.WriteLine($"{row.Department}: {row.Total} ({row.Headcount})");
```
### Segment
```csharp
var active = new ConditionGroup { Conditions = { new Condition { Sort = 1, Field = "IsActive", DataType = DataType.Boolean,
Operator = Operator.Equal, Values = { true } } } };
var onLeave = new ConditionGroup { Conditions = { new Condition { Sort = 1, Field = "Status", DataType = DataType.Enum,
Operator = Operator.Equal, Values = { "OnLeave" } } } };
var segment = new Segment
{
ConditionSets =
{
new ConditionSet { Sort = 1, ConditionGroup = active },
new ConditionSet { Sort = 2, Intersection = Intersection.Except, ConditionGroup = onLeave },
},
};
SegmentResult<Employee> result = await db.Employees.AsNoTracking().ToListAsync(segment); // one query
```
### An ASP.NET Core endpoint
```csharp
builder.Services.Configure<Microsoft.AspNetCore.Http.Json.JsonOptions>(o =>
o.SerializerOptions.Converters.Add(new JsonStringEnumConverter())); // accept "IContains", not only 5
app.MapPost("/employees/search", async (Filter filter, AppDbContext db) =>
{
try
{
return Results.Ok(await db.Employees.ToListAsync(filter));
}
catch (LogicException ex)
{
return Results.BadRequest(new { error = ex.Message });
}
catch (System.Linq.Dynamic.Core.Exceptions.ParseException ex)
{
return Results.BadRequest(new { error = ex.Message });
}
});
```
### A policy-protected entity
The examples above query `Employee` unguarded. Declared as below, with `RequirePolicy = true`, each of those calls
throws `PolicyRequired`: query it through `ApplyPolicy`, as the next example does.
```csharp
using DynamicWhere.ex.Policies.Attributes;
using DynamicWhere.ex.Policies.Enums;
[DwEntity(RequirePolicy = true)]
public class Employee
{
public Guid Id { get; set; }
public string FirstName { get; set; } = string.Empty;
// Confirmable, not searchable, and unreadable: the operators stop a sweep,
// the token stops the read, and [DwNoOrder] stops the sort from ranking it.
[DwAlias("Code")]
[DwOperators(Allow = new[] { Operator.Equal, Operator.In })]
[DwMask(MaskStrategy.Tokenize)]
[DwNoOrder]
public string EmployeeCode { get; set; } = string.Empty;
[DwMask(MaskStrategy.Email)]
[DwNoOrder]
public string Email { get; set; } = string.Empty;
// Rounded on the way out, aggregatable only over groups of five or more.
[DwGeneralize(GeneralizeMode.Round, Step = 5000, AllowAggregate = true, MinGroupSize = 5)]
[DwNoOrder, DwAudit, DwCost(10)]
public decimal Salary { get; set; }
// Every guarded query is scoped to this, asked for or not.
[DwForceWhere(Operator.Equal, Value = "true")]
public bool IsActive { get; set; }
// The caller's tenant, from the context.
[DwForceWhere(Operator.Equal, ContextValue = "TenantId")]
[DwNoWhere, DwNoSelect]
public int TenantId { get; set; }
[DwDenied]
public JsonDocument? WorkSchedule { get; set; }
}
```
### Wiring the policy layer
```csharp
using DynamicWhere.ex.Policies.Config;
using DynamicWhere.ex.Policies.Context;
using DynamicWhere.ex.Policies.Source;
using DynamicWhere.ex.Policies.Tokens;
// Program.cs — once
builder.Services.AddDwPolicies(builder.Configuration.GetSection("DynamicWhere:Policies"), options =>
{
options.Entities.Expose<Employee>("Employee");
options.TokenVault = new InMemoryTokenVault(); // RedisTokenVault or EfTokenVault in production
});
DwPolicy.ValidateModel(DwPolicy.Options, typeof(Employee)); // throws InvalidOperationException on any model error
// per request
app.MapPost("/employees/search", async (Filter filter, HttpContext http, AppDbContext db) =>
{
DwPolicyContext caller = await DwPolicy.PrepareAsync(new DwPolicyContext()
.WithSubject(DwSubjectKind.User, http.User.FindFirstValue(ClaimTypes.NameIdentifier)!)
.WithSubject(DwSubjectKind.Role, "Support")
.WithValue("TenantId", int.Parse(http.User.FindFirstValue("tenant_id")!)));
// with the ASP.NET Core package: DwPolicyContext caller = await http.GetPolicyContextAsync(claimsOptions);
try
{
FilterResult<Employee> result = await db.Employees.ApplyPolicy(caller).ToListAsync(filter);
return Results.Ok(result); // result.Policy: the trace; null under Strict by default
}
catch (PolicyException ex) { return Results.Json(new { error = ex.Code, field = ex.FieldPath }, statusCode: 403); }
catch (LogicException ex) { return Results.BadRequest(new { error = ex.Message }); }
});
```
It says what the library does, not what it should do
The reference is written against version 3.3.0 from the source, by hand, and checked by running the library, so it states behaviour — including the parts that are deliberately blunt, such as neither hashing nor tokenization hiding equality. Where a page in these docs disagrees with it, the file is the one to trust. For the reasoning behind a rule, the human pages carry it: start at Use cases or Security.