An experimental native Rust library for OneNote revision stores (.one and
.onetoc2). It reads committed object graphs, creates a small notebook without
a template, appends property, text, paragraph, outline and formatting edits with a
recoverable commit protocol, and interprets MS-ONE document structure, formatting, media, and historical pages.
The storage gates are recorded in PROGRESS.md; document-model
verification is recorded in M6-ACCEPTANCE.md. Concurrent-editing
verification is recorded in MILESTONE7.md. Crash recovery and the
read/write HTML diagnostic editor are recorded in MILESTONE8.md.
Embedded SMB coordination, durable offline editing and document-growth acceptance
are recorded in MILESTONE9.md.
Use disposable copies for notebook editing. Header version notification now follows durable transaction publication, fixing a native cached-reader race. The lost-reply acceptance records the failure, reduced regression model, twelve-client repeat and cold OneNote verification of all 3200 editing intents.
crates/onestore contains the library, examples and integration tests. New Rust
prototypes belong in sibling directories under crates/ and depend on
onestore = { path = "../onestore" }. The root manifest discovers these crates.
The consumer boundary and API tradeoffs are recorded in API-AUDIT.md.
onestore-diagnostic in the notebook crate backs the HTML diagnostic editor.
notebook provides notebook discovery, optional embedded
network access (feature smb) and local SQLite persistence and reconnect reconciliation for text, insertion and formatting.
Shared native fixtures, specifications, evidence and Python/VM tools stay at the
repository root; fuzz/ remains an independent cargo-fuzz workspace.
Run Cargo commands from the root. Select -p onestore when working only on the
library, or --workspace for checks across all crates. Example binary paths
remain target/debug/examples/… for the native verification tools. The collaboration
harness also accepts --client-profile release.
| API | Contract |
|---|---|
Store, RevisionIndex, ResolvedRevision | Parse storage, resolve revisions and reference graphs, expose roots and objects |
PropertySets, Object::references | Decode properties and ID streams while retaining raw values |
Object::file_reference, Store::file_data | Identify internal/external payloads and read internal payload bytes |
document::Document, Revision::text_runs | Interpret document objects and inherited text formatting while retaining unknown properties and revision identities |
Document::active, Document::pages_in, Revision::parents, RevisionIndex::active | Resolve the active revision, the pages of a space and parent links without repeating the lookups |
Page::copy | The page's content under fresh identities for copying into another section, definitions and payloads included; indent levels only an outline group carries refuse |
page::link::internal_link, page::link::parse_internal_link | Build and read the onenote:#… URLs OneNote stores for links to sections, pages and paragraphs, by identity |
page::Recording, page::MediaIndex, PageOp::Media | An audio or video file's recording identity, kind and length, and a paragraph's link to a moment in recordings, as OneNote stores them; the page lists its recordings as they come and go |
page::Page, page::Paragraph, page::Ink, page::Math | Build an editable page model (title, outlines, paragraphs with coalesced text spans, tables, images, attachments in paragraphs or on the page, ink drawings and handwriting decoded to stroke polylines in page points with each point's pen pressure, a moved drawing offset by its position, a highlighter's raster operation, a drawn shape's kind and anchors) with stored identities; equations parse from their linear text and run data into a tree that renders the MathML OneNote exports; content outside the model is retained as Unsupported |
protected::Key, Section::unlock | Open a password-protected section's key with its password (Key::open), or make one for a new password with OneNote 2010's encryption data (Key::new); keep the section open under it, decoding every object and payload, and seal edits under it as OneNote does (a fresh IV per object) |
protected::rekey | Set, change or remove a section's password as OneNote 2010 does: the section written anew under fresh file, space and payload identities, each space keeping its labelled revisions as checkpoints |
protected::UnlockedSection | A protected section's labelled revisions decoded into a Document for inspection and export, by password or Key; cleared on drop, while strings and exports made from it are the caller's |
create_section | Create one page containing one plain-text paragraph and an author, including Unicode |
PageCreation, SectionOp::Create | Add an empty top-level page and its section entry atomically, under identities the intent retains across retries |
PageEdit, SectionOp::Pages | Publish explicitly selected page moves and indentation changes together, preserving page content and historical revisions |
SectionOp::Delete | Remove explicit pages and their section references atomically while retaining stored revisions |
create_table_of_contents | Create ordered section entries from filenames and file identities |
TocEdit, edit_table_of_contents | Add, rename, order and remove a table of contents' section and group entries, and colour the notebook, as one revision's Transaction; a section's colour is SectionOp::Color in its own metadata |
place, place_file | Name a file for its notebook as OneNote does on adoption (parent TOC identity and name CRC in the header), so OneNote keeps its identity, on any CommitIo or under the filesystem adapter |
place_image, reidentify | The same on an image held whole, or outside any notebook; and a new file identity and version, as OneNote's Save As copies and Unpack Notebook give (corpus/notebook-package) |
TextAttribute, PageOp::Format | Change character formatting over a UTF-16 range while sharing immutable styles; preserve unselected runs |
PageOp::Style, PageOp::Restyle, op::restyle | Give a paragraph a paragraph style (with its NextStyle, as OneNote 2010 writes its headings), or move every paragraph of a style to a new style object of the same name; style objects are read-only, and run or paragraph values equal to what the old style gave are cleared so they follow the new one. op::restyle(page, sheet) brings a page's named styles to a sheet of definitions, joining styles that share a name |
OutlineEdit, PageOp::Outline | Change ordinary outline position/width, the position of a file or ink drawing on the page, or a paragraph's saved expansion default, preserving identities and content |
Transaction, Stamp | The bytes a commit writes (appended data, in-place list-tail and log patches, header) and the header and length it requires unchanged; commit under caller-held exclusion, commit_file under the conservative filesystem adapter, apply to the base image in memory; serializable for queues |
Arena, Section | Keep a section parsed across edits: open validates an image once; seal appends one revision per changed space as a Transaction on stamp, checking only what it appends; replay applies a queued transaction; image and page read the result |
op::{Edit, Op, PageOp, TableEdit, SectionOp}, Section::apply | Object-level edits (text, formatting, links, equations, paragraph insertion/split/join/move/deletion/levels, outlines, lists, tags, styles, paragraph formatting, tables, pictures, attachments, ink, page creation/import/moves/removal, conflict pages) applied whole or not to a kept-open section with emitter-chosen identities, UTF-16 ranges and the edit's time; refusals name the target, identity or structure at fault |
SectionOp::Conflict, Section::conflicts, ConflictPage | Keep the version of a page a merge could not take as OneNote 2010 does: a read-only conflict page (jcidConflictPageMetaData, IsConflictPage, conflict objects marked) under the page's manifest, which says it has conflict pages; list each page's, newest first, and delete one (SectionOp::Delete) as OneNote's Delete Conflict Page does |
Section::versions, Section::version, PageVersion, SectionOp::RestoreVersion, SectionOp::DeleteVersions | A page's versions as OneNote 2010 keeps them: earlier revisions of the page's own space, each current under its own context and listed by the page's version history, newest first with its author and time; read one as a page (O(section)); restore one as Restore Version does (the page's next revision builds on it and the page as it stood becomes the newest version) or unlist some as Delete Version does, their revisions staying stored (corpus) |
Section::page_at, Section::open_at, Section::seal_as | Read a page as any revision the file stores holds it, or the section as chosen revisions of its spaces leave it (a merge's common ancestor); seal revisions under chosen identities so a later merge recognises them |
op::lower, op::lower_page | The ops turning a range of paragraphs, or a whole page model, into another, computed from the models alone, for editors and imports |
op::predict | The page an op leaves, as the section stores it and reading it back shows it |
read_file | Read a snapshot under whole-file exclusion |
read_snapshot | Read a validated snapshot through fresh positioned I/O while the caller excludes maintenance |
CommitIo, confirm, confirm_file, Stamp::check | Supply another storage backend with equivalent exclusion and ordered durability; check that a stamp still holds |
supersede_file | Put a file written anew (protected::rekey) in place of a section under the exclusion commit_file takes, while its stamp holds; on Windows the old file goes aside first, as OneNote's maintenance does |
Edits enter a kept-open Section as ops and leave as one appended revision per
changed space; only section and page creation, imports and opening files handle
whole images. Text edits maintain run boundaries, inherit the insertion run's formatting,
and promote legacy text to Unicode when needed. Explicit and body-derived
navigation titles update in the same transaction; unsupported fields,
protected objects and split surrogate pairs are
rejected before writing. Appended snapshots cap revision dependency depth at 512
while retaining historical revisions. TOC snapshots can remap encoded CompactIDs
without changing their resolved references. A password-protected section opens with
Section::unlock under the Key its password opens, and every revision it seals names
the key (MS-ONESTORE 2.5.19 asks that of each revision; OneNote names it only in those
without a dependency, and reads both). Section::open refuses one. Incorrect
passwords, unsupported protection profiles and work-limit failures remain distinct
(protected::Error).
PageCreation::new appends, or inserts before the first page space of an existing
series. Some("") creates an empty title field; None omits the title node.
dated(date, time) gives the title OneNote 2010's date and time fields showing that
text, as OneNote titles a new page; in_space(guid) creates the page in the space {guid},1 with a series named after it, as a merge that must make the same page every time does; keeping(identity, created) keeps another page's
identity and creation time, as a page moved to the recycle bin keeps them. The page has
no body outlines or applied template; create_empty_section makes a section for such
pages, and PageOp::Color sets or clears a page's colour (0x14001d2a on the page node,
absent for "No color"), and PageOp::RuleLines its rule lines (six properties on the page
node, absent for None; rule-lines).
Retain the intent to preserve its page, title and space identities; existing
identities require reconciliation before retry. Body insertion and title edits
use those identities through page ops. Native page-creation fixtures
cover duplicate Unicode titles, native edits and Rust follow-up edits.
PageEdit::set_level changes one page's indentation in place. PageEdit::move_to
moves before an existing page space, or appends for None, and sets its level.
SectionOp::Pages applies moves in slice order and publishes the final order,
series membership and metadata levels in one transaction. Each page occurs once;
levels are 1–3 and the final first page must have level 1. A following deeper-level
page remains in place unless explicitly selected. To move a group, supply all its
pages in order. Retain the intents across retries so newly formed series keep
their identities; reposition(PagePosition, level) revises their placement while
preserving those identities. Native page-edit fixtures
cover individual tabs, selected and collapsed groups, nesting and promotion.
SectionOp::Delete removes exactly the supplied page spaces,
including subpages only when selected explicitly. The first remaining page becomes
top-level; other page levels and surviving content are retained. The operation
creates no recycle-bin copies and preserves prior revisions, so it is not secure
erasure. Duplicate, missing or non-page identities reject the entire batch;
an empty selection leaves the file unchanged.
PageOp::Insert and PageOp::Add update child references, reference counts,
modification times and automatic titles atomically. Paragraphs can be nested or
inserted into table cells; outline coordinates use points. Inserted text keeps its
spans' formats; PageOp::Format over an empty text's 0..0 sets its insertion style.
New objects carry the emitter's identities, so a queued edit replays onto another
image unchanged; an identity already on the page refuses the edit. Formatting accepts explicit attributes, preserves inherited values,
and gives retired immutable styles zero current references while retaining history.
Outline layout edits use points. Width is at least 36 points; an explicit user width
and an automatic maximum-width hint remain distinct. Native layout generates the
rendered height. Saved paragraph collapse defaults can be overridden by the native
client's cached view. Native layout captures
verify these edits through a fresh OneNote cache, including fields and nested content.
PageOp::Move takes an existing parent and an optional direct sibling to insert
before; None appends. Paragraphs retain their descendants and explicit list styles.
Outlines remain page children, so reordering changes their stacking order while
retaining coordinates. PageOp::Delete removes the selected subtree from the active
graph and preserves historical objects. Empty outlines/groups are removed; surviving
group indentation is normalized without shifting other paragraphs. An edit that
leaves a table cell without a paragraph is refused; insert its replacement in the
same edit.
Title/protected content, ambiguous ancestry, cycles, and incompatible destinations
reject before publication. Modification times, move attribution and automatic titles
publish with the tree change. These explicit destinations differ from keyboard list
indentation, which can also substitute list markers.
Paragraph splits preserve character formatting, retain tags on the left, and clone
mutable list objects without restarting numbering. PageOp::Split names
the new right paragraph/text identities. Its publication includes
the complete child graph and title metadata; repeating an existing identity requires
reconciliation. Title containers, generated fields, recording-linked text and
associated run metadata are rejected before I/O. Native split controls and subsequent
typing checks reside in the paragraph corpus.
Joins retain the left paragraph. Nonempty left text keeps its identity; empty left
text adopts the right text identity. The left tags win: right-side tags are removed
from active text even when the left text is empty. History retains the original
objects. Select the preceding leaf text; where that leaf is deeper than the right
paragraph, right children move to its ancestor at the right paragraph's level.
Ambiguous ancestry, unsupported indentation transitions and unknown implicit
font/language inheritance reject before I/O. This is a logical join, so keyboard
actions that only change list or indentation state remain separate operations.
Generated fields, protected targets and unsupported run-data boundary changes are
rejected before publication. The notebook crate
documents durable local operations and reconciliation. The
document-writer acceptance
includes twelve mixed native/Rust clients, outages, lost replies and native revision retirement.
External .onebin references identify payloads for the caller to obtain. Cloud
FSSHTTP synchronization and a C ABI are outside the implemented surface.
Requires Rust 1.97 or later for the verified build. Examples create new destinations and refuse to overwrite them. The Python tools require Python 3.10 or later and Pillow.
| 1 | cargo run --example create_notebook -- /tmp/one-demo 'Hello from Rust.' 'Example Author' |
| 2 | cargo run --example inventory -- /tmp/one-demo/synthetic.one |
| 3 | cargo run --example inspect -- /tmp/one-demo/synthetic.one |
| 4 | cargo run --example document -- /tmp/one-demo/synthetic.one /tmp/one-model |
| 5 | cargo build -p notebook --bin onestore-diagnostic |
| 6 | python3 tools/notebook_report.py /tmp/one-demo /tmp/one-report --timezone America/Los_Angeles |
A seeded text edit on a disposable copy records its page, UTF-16 range, replacement and outcome as JSON:
| 1 | cargo run --example random_edit -- /tmp/one-demo/synthetic.one /tmp/edited.one 42 |
The report contains readable pages, document JSON, assets, source identities and coordinates. It preserves paragraph nesting, lists, tables, links and tags. Historical contexts, recycle-bin pages and default templates are represented separately. Native ink is decoded to strokes and equations to MathML; both retain their source data. The report is a reading view; its native PDF references supply the original canvas layout.
Read and validate a snapshot before interpreting its graph:
| 1 | use onestore::{read_file, RevisionIndex, Store}; |
| 2 | |
| 3 | fn main() -> Result<(), Box<dyn std::error::Error>> { |
| 4 | let bytes = read_file("notebook/synthetic.one")?; |
| 5 | let store = Store::parse(&bytes)?; |
| 6 | if !store.checksum_mismatches.is_empty() { |
| 7 | return Err("Transaction checksum damage".into()); |
| 8 | } |
| 9 | let index = RevisionIndex::parse(&store)?; |
| 10 | index.validate_current()?; |
| 11 | Ok(()) |
| 12 | } |
Store::parse exposes checksum mismatches for diagnostic readers; the writer
rejects them. The resolved graph borrows the snapshot. Open it as a Section,
apply ops naming objects from that graph, seal, and commit the Transaction. The
edit_property example demonstrates selection by JCID/property/expected bytes,
with optional explicit IDs when conflict copies contain identical text.
| 1 | exclusive lock → header and length check |
| 2 | → append data → flush |
| 3 | → prepare header metadata → flush |
| 4 | → publish transaction counter → flush |
| 5 | → finish counter rollover → flush |
| 6 | → notify cached readers → flush → unlock |
Stale snapshots fail before writing: every committed transaction and placement rewrites
the header (MS-ONESTORE 2.3.1), so the body is never compared. Live readers must use
equivalent exclusion.
Native conflict creation can still expose cross-space references before their
targets are saved; such snapshots must be rejected and reread while synchronization
proceeds. read_file and Transaction::commit_file serialize within the process because
macOS SMB locks can be reentrant. The lock is nonblocking across processes;
contention requires a fresh read and a later retry.
| Error state | Meaning and caller action |
|---|---|
NotCommitted | This edit was not published. Preparation bytes may exist. Reread before retrying. |
Unknown | Publication may have persisted despite the error. Reread and resolve the intended edit before retrying. |
Committed | Publication was durably acknowledged; counter cleanup or lock release failed. Reopen the committed result instead of replaying the edit. |
At counter rollover, empty transactions make intermediate published counts valid. The highest changed byte is flushed before lower bytes are cleaned up. The 255→256 and 65535→65536 boundaries and interrupted cleanup states have independent native acceptance captures. A no-op still flushes; a failed flush has an unknown durability outcome.
The filesystem adapter uses whole-file locking. On macOS it acquires the lock
as part of opening the file (O_EXLOCK to commit, O_SHLOCK to read, with
O_NONBLOCK): separate open and flock calls allowed overlapping exclusive holders
and stranded server locks under multi-process SMB contention.
tools/smb_lock_race.py reproduces that failure without notebook parsing or writing,
and with --shared-reads checks that shared readers never overlap a writer. On the
tested macOS SMB mount, POSIX byte-range locks returned ENOTSUP, and flock of
either kind became an exclusive lock over the whole file, which fails OneNote's reads.
smbfs sends O_SHLOCK and O_EXLOCK as share modes: a read excludes writers,
OneNote's included, but not OneNote's readers; a commit excludes everyone. It does not
reproduce native reader/writer concurrency. The
locking audit records native coordination bytes, write-open share
modes, and a working macOS SMB-specific byte-range lock probe. sync_all falls
back to fsync on macOS only when F_FULLFSYNC is unsupported. Successful SMB FLUSH replies were observed on the
wire. Correctness requires the backend to honor exclusion and ordered flushes.
The evidence covers transport failures, not physical server power loss or every
filesystem's lock implementation.
For shared network notebooks, use the optional
notebook::smb module. It uses native share modes,
shared reader guards, writer exclusion and fresh pathname identity checks without
an OS-mounted share. Its coordination acceptance covers native
maintenance, mixed readers/writers, reconnects and uncertain publication. The
filesystem adapter retains its conservative locking; mounted-path freshness
across native replacement is not established by that exclusion.
| 1 | cargo test --all-targets |
| 2 | cargo clippy --all-targets -- -D warnings |
| 3 | python3 tools/verify-corpus.py |
| 4 | python3 tools/verify-reader.py # requires Pillow and the local private corpus |
| 5 | python3 tools/verify-writer.py |
| 6 | python3 tools/verify-collaboration.py |
| 7 | python3 tools/verify-document.py /path/to/copied/notebook /path/to/native/read |
The frozen private corpus is excluded from version control. Its verifier checks 26 pages,
581 text objects, hyperlink targets, exact image bytes, and native image conversions.
Synthetic corpus manifests bind binary fixtures to independent native XML and
attachment captures. verify-corpus.py also requires that private corpus.
Storage tests exercise malformed references, deep properties, historical revision preservation, short I/O, stale snapshots, counter tears, and every injected I/O failure point at ordinary and rollover commits. The crash model persists arbitrary subsets of unflushed bytes and is shared with the stateful commit fuzzer.
| 1 | cargo +nightly fuzz run revisions -- -max_total_time=120 -max_len=262144 -rss_limit_mb=2048 |
| 2 | cargo +nightly fuzz run commit -- -max_total_time=300 -max_len=4096 -rss_limit_mb=2048 |
| 3 | cargo +nightly fuzz run paragraph -- -max_total_time=120 -max_len=160 -rss_limit_mb=2048 |
Fuzz targets cover storage, properties, revisions, scalar edits, creation, and
multi-edit interrupted commits. Seed the revision target with native .one files
using symlinks under fuzz/corpus/revisions; empty seed directories mostly exercise
header rejection. Bounded run counts and native findings live in PROGRESS.md.
The document fuzzer mutates native property streams, repairs their checksums, and traverses every retained revision and resolved text run. Its public seeds live in the source target; private seeds are supplied only at runtime. Native edit-history tests compare independently generated operations, OneNote XML and the Rust model; failed histories can be replayed and shrunk in fresh disposable clones. The document feature matrix is in FEATURES.md, and the milestone's acceptance contract is in MILESTONE6.md.
The stage-5 gate uses OneNote 2010 build 14.0.7015.1000 on Windows 7, a macOS SMB mount, and Samba on zenith. The collaboration corpus captures native lock contention, different-paragraph merging, same-paragraph conflicts, offline editing/reconnection, and lost successful FLUSH replies at preparation, publication, and counter cleanup. Each final notebook was reopened from a fresh native cache. Competing text survives as native conflict pages; OneNote's COM hierarchy omits those pages, so the offline case also includes a native UI capture. Full-page COM updates produced an extra conflict copy during the disjoint case and two recorded geometry changes; the verifier checks those exact changes as well as retained image, ink, table, and attachment content.
tools/native/profile.ps1 parks/restores the personal native profile and cache;
cold.ps1, read.ps1, and collaborate.ps1 operate on disposable test roots.
tools/zenith-locks observes server locks. tools/smb-proxy.py traces and interrupts
a dedicated loopback test session; its control JSON selects the successful response
and occurrence to withhold. The captured trace and result files are the regression
oracle; replaying the native experiments requires the supplied Windows/share setup.
OneNote 2010 wraps an AES-128 key in Office's Agile password encryption (MS-OFFCRYPTO:
SHA-1 of a 16-byte salt and the UTF-16LE password, then 100,000 rounds of SHA-1 over the
round number and the hash; three block keys decrypt, with the salt as IV, the verifier
input, its SHA-1 zero-padded to 32 bytes, and the key). The XML sits after the words
3, length, 16, length - 16 and the version 4.4 with flags 0x40, inside the
encryption-data container every revision names. Each property object is stored as its
reference streams, a length, a random IV and the CBC encryption of a padding count, the
property bytes and random padding; read-only objects hash the clear bytes zero-padded to
8 bytes. Payloads are their length and bytes, randomly padded, under CBC with the IV
SHA-1(key data salt, block 0). Setting, changing or removing a password writes the section
anew, as OneNote does, under fresh identities so that no cache confuses the old file's
objects or payloads with the new ones (corpus/protected-sections).
| 1 | use onestore::{Arena, Section, protected::{Key, rekey}}; |
| 2 | # fn example(image: Vec<u8>, password: &str) -> Result<(), Box<dyn std::error::Error>> { |
| 3 | let key = Key::open(&image, password)?; |
| 4 | let arena = Arena::default(); |
| 5 | let mut section = Section::unlock(&arena, image.clone(), &key)?; |
| 6 | assert!(!section.pages()?.is_empty()); |
| 7 | let changed = rekey(&image, Some(&key), Some(&Key::new("another password")?))?; |
| 8 | # drop(changed); |
| 9 | # Ok(()) } |
A Key holds no password; its key is cleared when its last clone drops. A Section
decodes into its Arena, which is not cleared; drop both when the section locks.
UnlockedSection owns its decoded buffers and clears them on drop; the views it lends
cannot outlive it, while copies of parsed strings, serialized models and exports are
plaintext with lifetimes of their own. CBC has no general
ciphertext-authentication guarantee; native read-only hashes and model validation
check the corresponding structure. Internal payloads are decoded; external payload
references remain references, and their protected decoding is not implemented.
For a deliberate plaintext diagnostic export, run the notebook exporter
(examples/document) with --password-file PATH after the source and optional
new output directory. The file contains exact UTF-8 password bytes; no newline is
removed or Unicode normalization applied. The exporter creates protected exports
under an owner-only directory on Unix and reports protected external payloads as
unsupported. It never rewrites the encrypted source.