Atlas Module

A lightweight document collection manager for V, inspired by doctree but simplified.

Features

• Simple Collection Scanning: Automatically find collections marked with .collection files
• Include Processing: Process !!include actions to embed content from other pages
• Easy Export: Copy files to destination with organized structure
• Optional Redis: Store metadata in Redis for quick lookups and caching
• Type-Safe Access: Get pages, images, and files with error handling
• Error Tracking: Built-in error collection and reporting with deduplication

Quick Start

put in .hero file and execute with hero or but shebang line on top of .hero script

Scan Parameters:

• name (optional, default: 'main') - Atlas instance name
• path (required when git_url not provided) - Directory path to scan
• git_url (alternative to path) - Git repository URL to clone/checkout
• git_root (optional when using git_url, default: ~/code) - Base directory for cloning
• meta_path (optional) - Directory to save collection metadata JSON
• ignore (optional) - List of directory names to skip during scan

most basic example

#!/usr/bin/env hero

!!atlas.scan git_url:"https://git.ourworld.tf/tfgrid/docs_tfgrid4/src/branch/main/collections/tests"

!!atlas.export destination: '/tmp/atlas_export' 

put this in .hero file

usage in herolib

import incubaid.herolib.data.atlas

// Create a new Atlas
mut a := atlas.new(name: 'my_docs')!

// Scan a directory for collections
a.scan(path: '/path/to/docs')!

// Export to destination
a.export(destination: '/path/to/output')!

Collections

Collections are directories marked with a .collection file.

.collection File Format

name:my_collection

Core Concepts

Collections

A collection is a directory containing:- A .collection file (marks the directory as a collection)

• Markdown pages (.md files)
• Images (.png, .jpg, .jpeg, .gif, .svg)
• Other files

Page Keys

Pages, images, and files are referenced using the format: collection:name

// Get a page
page := a.page_get('guides:introduction')!

// Get an image
img := a.image_get('guides:logo')!

// Get a file
file := a.file_get('guides:diagram')!

Usage Examples

Scanning for Collections

mut a := atlas.new()!
a.scan(path: './docs')!

Adding a Specific Collection

a.add_collection(name: 'guides', path: './docs/guides')!

Getting Pages

// Get a page
page := a.page_get('guides:introduction')!
content := page.content()!

// Check if page exists
if a.page_exists('guides:setup') {
    println('Setup guide found')
}

Getting Images and Files

// Get an image
img := a.image_get('guides:logo')!
println('Image path: ${img.path.path}')
println('Image type: ${img.ftype}')  // .image

// Get a file
file := a.file_get('guides:diagram')!
println('File name: ${file.file_name()}')

// Check existence
if a.image_exists('guides:screenshot') {
    println('Screenshot found')
}

Listing All Pages

pages_map := a.list_pages()
for col_name, page_names in pages_map {
    println('Collection: ${col_name}')
    for page_name in page_names {
        println('  - ${page_name}')
    }
}

Exporting

// Full export with all features
a.export(
    destination: './output'
    reset: true        // Clear destination before export
    include: true      // Process !!include actions
    redis: true        // Store metadata in Redis
)!

// Export without Redis
a.export(
    destination: './output'
    redis: false
)!

Error Handling

// Export and check for errors
a.export(destination: './output')!

// Errors are automatically printed during export
// You can also access them programmatically
for _, col in a.collections {
    if col.has_errors() {
        errors := col.get_errors()
        for err in errors {
            println('Error: ${err.str()}')
        }

        // Get error summary by category
        summary := col.error_summary()
        for category, count in summary {
            println('${category}: ${count} errors')
        }
    }
}

Include Processing

Atlas supports simple include processing using !!include actions:

// Export with includes processed (default)
a.export(
    destination: './output'
    include: true  // default
)!

// Export without processing includes
a.export(
    destination: './output'
    include: false
)!

Include Syntax

In your markdown files:

##
!!include collection:page_name

More content here

Or within the same collection:

!!include page_name

The !!include action will be replaced with the content of the referenced page during export.

Reading Pages with Includes

// Read with includes processed (default)
mut page := a.page_get('col:mypage')!
content := page.content(include: true)!

// Read raw content without processing includes
content := page.content()!

Git Integration

Atlas automatically detects the git repository URL for each collection and stores it for reference. This allows users to easily navigate to the source for editing.

Automatic Detection

When scanning collections, Atlas walks up the directory tree to find the .git directory and captures:- git_url: The remote origin URL

• git_branch: The current branch

Scanning from Git URL

You can scan collections directly from a git repository:

!!atlas.scan
    name: 'my_docs'
    git_url: 'https://github.com/myorg/docs.git'
    git_root: '~/code'  // optional, defaults to ~/code

The repository will be automatically cloned if it doesn't exist locally.

Accessing Edit URLs

mut page := atlas.page_get('guides:intro')!
edit_url := page.get_edit_url()!
println('Edit at: ${edit_url}')
// Output: Edit at: https://github.com/myorg/docs/edit/main/guides.md

Export with Source Information

When exporting, the git URL is displayed:

Collection guides source: https://github.com/myorg/docs.git (branch: main)

This allows published documentation to link back to the source repository for contributions.## Links

Atlas supports standard Markdown links with several formats for referencing pages within collections.

Link Formats

1. Explicit Collection Reference

Link to a page in a specific collection:

[Click here](guides:introduction)
[Click here](guides:introduction.md)

2. Same Collection Reference

Link to a page in the same collection (collection name omitted):

[Click here](introduction)

3. Path-Based Reference

Link using a path - only the filename is used for matching:

[Click here](some/path/introduction)
[Click here](/absolute/path/introduction)
[Click here](path/to/introduction.md)

Important: Paths are ignored during link resolution. Only the page name (filename) is used to find the target page within the same collection.

Link Processing

Validation

Check all links in your Atlas:

mut a := atlas.new()!
a.scan(path: './docs')!

// Validate all links
a.validate_links()!

// Check for errors
for _, col in a.collections {
    if col.has_errors() {
        col.print_errors()
    }
}

Fixing Links

Automatically rewrite links with correct relative paths:

mut a := atlas.new()!
a.scan(path: './docs')!

// Fix all links in place
a.fix_links()!

// Or fix links in a specific collection
mut col := a.get_collection('guides')!
col.fix_links()!

What fix_links() does:- Finds all local page links

• Calculates correct relative paths
• Rewrites links as [text](relative/path/pagename.md)
• Only fixes links within the same collection
• Preserves !!include actions unchanged
• Writes changes back to files

Example

Before fix:

##
[Introduction](introduction)
[Setup](/some/old/path/setup)
[Guide](guides:advanced)

After fix (assuming pages are in subdirectories):

##
[Introduction](../intro/introduction.md)
[Setup](setup.md)
[Guide](guides:advanced)  <!-- Cross-collection link unchanged -->

Link Rules

1. Name Normalization: All page names are normalized using name_fix() (lowercase, underscores, etc.)
2. Same Collection Only: fix_links() only rewrites links within the same collection
3. Cross-Collection Links: Links with explicit collection references (e.g., guides:page) are validated but not rewritten
4. External Links: HTTP(S), mailto, and anchor links are ignored
5. Error Reporting: Broken links are reported with file, line number, and link details

Export Directory Structure

When you export an Atlas, the directory structure is organized as:

$$\text{export_dir}/ \begin{cases} \text{content/} \ \quad \text{collection_name/} \ \quad \quad \text{page1.md} \ \quad \quad \text{page2.md} \ \quad \quad \text{img/} & \text{(images)} \ \quad \quad \quad \text{logo.png} \ \quad \quad \quad \text{banner.jpg} \ \quad \quad \text{files/} & \text{(other files)} \ \quad \quad \quad \text{data.csv} \ \quad \quad \quad \text{document.pdf} \ \text{meta/} & \text{(metadata)} \ \quad \text{collection_name.json} \end{cases}$$

• Pages: Markdown files directly in collection directory
• Images: Stored in img/ subdirectory
• Files: Other resources stored in files/ subdirectory
• Metadata: JSON files in meta/ directory with collection information

Redis Integration

Atlas uses Redis to store metadata about collections, pages, images, and files for fast lookups and caching.

Redis Data Structure

When redis: true is set during export, Atlas stores:

1. Collection Paths - Hash: atlas:path

• Key: collection name
• Value: exported collection directory path

2. Collection Contents - Hash: atlas:<collection_name>

• Pages: page_name → page_name.md
• Images: image_name.ext → img/image_name.ext
• Files: file_name.ext → files/file_name.ext

Redis Usage Examples

import incubaid.herolib.data.atlas
import incubaid.herolib.core.base

// Export with Redis metadata (default)
mut a := atlas.new(name: 'docs')!
a.scan(path: './docs')!
a.export(
    destination: './output'
    redis: true  // Store metadata in Redis
)!

// Later, retrieve metadata from Redis
mut context := base.context()!
mut redis := context.redis()!

// Get collection path
col_path := redis.hget('atlas:path', 'guides')!
println('Guides collection exported to: ${col_path}')

// Get page location
page_path := redis.hget('atlas:guides', 'introduction')!
println('Introduction page: ${page_path}')  // Output: introduction.md

// Get image location
img_path := redis.hget('atlas:guides', 'logo.png')!
println('Logo image: ${img_path}')  // Output: img/logo.png

Saving Collections (Beta)

Status: Basic save functionality is implemented. Load functionality is work-in-progress.

Saving to JSON

Save collection metadata to JSON files for archival or cross-tool compatibility:

import incubaid.herolib.data.atlas

mut a := atlas.new(name: 'my_docs')!
a.scan(path: './docs')!

// Save all collections to a specified directory
// Creates: ${save_path}/${collection_name}.json
a.save('./metadata')!

What Gets Saved

Each .json file contains:- Collection metadata (name, path, git URL, git branch)

• All pages (with paths and collection references)
• All images and files (with paths and types)
• All errors (category, page_key, message, file)

Storage Location

save_path/
├── collection1.json
├── collection2.json
└── collection3.json

HeroScript Integration

Atlas integrates with HeroScript, allowing you to define Atlas operations in .vsh or playbook files.

Using in V Scripts

Create a .vsh script to process Atlas operations:

#!/usr/bin/env -S v -n -w -cg -gc none -cc tcc -d use_openssl -enable-globals run

import incubaid.herolib.core.playbook
import incubaid.herolib.data.atlas

// Define your HeroScript content
heroscript := "
!!atlas.scan path: './docs'

!!atlas.export destination: './output' include: true
"

// Create playbook from text
mut plbook := playbook.new(text: heroscript)!

// Execute atlas actions
atlas.play(mut plbook)!

println('Atlas processing complete!')

Using in Playbook Files

Create a docs.play file:

!!atlas.scan
    name: 'main'
    path: '~/code/docs'

!!atlas.export
    destination: '~/code/output'
    reset: true
    include: true
    redis: true

Execute it:

vrun process_docs.vsh

Where process_docs.vsh contains:

#!/usr/bin/env -S v -n -w -cg -gc none -cc tcc -d use_openssl -enable-globals run

import incubaid.herolib.core.playbook
import incubaid.herolib.core.playcmds

// Load and execute playbook
mut plbook := playbook.new(path: './docs.play')!
playcmds.run(mut plbook)!

Error Handling

Errors are automatically collected and reported:

!!atlas.scan
 path: './docs'

##!!atlas.export
 destination: './output'

Errors are shown in the console:

Collection guides - Errors (2)
  [invalid_page_reference] [guides:intro]: Broken link to `guides:setup` at line 5
  [missing_include] [guides:advanced]: Included page `guides:examples` not found

Auto-Export Behavior

If you use !!atlas.scan without an explicit !!atlas.export, Atlas will automatically export to the default location (current directory).

To disable auto-export, include an explicit (empty) export action or simply don't include any scan actions.

Best Practices

1. Always validate before export: Use !!atlas.validate to catch broken links early
2. Use named instances: When working with multiple documentation sets, use the name parameter
3. Enable Redis for production: Use redis: true for web deployments to enable fast lookups
4. Process includes during export: Keep include: true to embed referenced content in exported files

Roadmap - Not Yet Implemented

The following features are planned but not yet available:

• Load collections from .collection.json files
• Python API for reading collections
• atlas.validate playbook action
• atlas.fix_links playbook action
• Auto-save on collection modifications
• Collection version control

fn exists(name string) bool

Check if Atlas exists

fn get(name string) !&Atlas

Get Atlas from global map

fn list() []string

List all Atlas names

fn new(args AtlasNewArgs) !&Atlas

Create a new Atlas

fn new_group(args GroupNewArgs) !Group

Create a new Group

fn play(mut plbook PlayBook) !

Play function to process HeroScript actions for Atlas

enum CollectionErrorCategory {
	circular_include
	missing_include
	include_syntax_error
	invalid_page_reference
	invalid_file_reference
	file_not_found
	invalid_collection
	general_error
	acl_denied // NEW: Access denied by ACL
}

enum FileType {
	file
	image
}

enum LinkFileType {
	page  // Default: link to another page
	file  // Link to a non-image file
	image // Link to an image file
}

enum LinkStatus {
	init
	external
	found
	not_found
	anchor
	error
}

struct Atlas {
pub mut:
	name        string
	collections map[string]&Collection
	groups      map[string]&Group // name -> Group mapping
}

fn (mut a Atlas) export(args ExportArgs) !

Export all collections

fn (a Atlas) file_exists(key string) !bool

Check if file exists

fn (a Atlas) file_get(key string) !&File

Get a file from any collection using format "collection:file"

fn (a Atlas) file_or_image_exists(key string) !bool

fn (a Atlas) file_or_image_get(key string) !&File

Get a file (can be image) from any collection using format "collection:file"

fn (mut a Atlas) fix_links() !

Fix all links in all collections (rewrite source files)

fn (a Atlas) get_collection(name string) !&Collection

Get a collection by name

fn (mut a Atlas) group_add(mut group Group) !

Add a group to the atlas

fn (a Atlas) group_get(name string) !&Group

Get a group by name

fn (a Atlas) groups_get(session Session) []&Group

Get all groups matching a session's email

fn (a Atlas) image_exists(key string) !bool

Check if image exists

fn (a Atlas) image_get(key string) !&File

Get an image from any collection using format "collection:image"

fn (mut a Atlas) init_post() !

Validate all links in all collections

fn (a Atlas) list_pages() map[string][]string

List all pages in Atlas

fn (a Atlas) page_exists(key string) !bool

Check if page exists

fn (a Atlas) page_get(key string) !&Page

Get a page from any collection using format "collection:page"

fn (mut a Atlas) scan(args ScanArgs) !

fn (mut a Atlas) validate_links() !

Validate all links in all collections

struct AtlasNewArgs {
pub mut:
	name string = 'default'
}

struct Collection {
pub mut:
	name        string
	path        string // absolute path
	pages       map[string]&Page
	files       map[string]&File
	atlas       &Atlas @[skip; str: skip]
	errors      []CollectionError
	error_cache map[string]bool
	git_url     string
	acl_read    []string // Group names allowed to read (lowercase)
	acl_write   []string // Group names allowed to write (lowercase)
}

fn (c Collection) can_read(session Session) bool

Check if session can read this collection

fn (c Collection) can_write(session Session) bool

Check if session can write this collection

fn (mut c Collection) clear_errors()

Clear all errors

fn (mut c Collection) error(args CollectionErrorArgs)

Report an error, avoiding duplicates based on hash

fn (c Collection) error_summary() map[CollectionErrorCategory]int

Get error summary by category

fn (mut c Collection) export(args CollectionExportArgs) !

Export a single collection Export a single collection with recursive link processing

fn (c Collection) file_exists(name_ string) !bool

Check if file exists

fn (c Collection) file_get(name_ string) !&File

Get a file by name

fn (c Collection) file_or_image_exists(name_ string) !bool

fn (c Collection) file_or_image_get(name_ string) !&File

fn (mut c Collection) fix_links() !

Fix all links in collection (rewrite files)

fn (c Collection) get_errors() []CollectionError

Get all errors

fn (c Collection) has_errors() bool

Check if collection has errors

fn (c Collection) image_exists(name_ string) !bool

Check if image exists

fn (c Collection) image_get(name_ string) !&File

Get an image by name

fn (c Collection) page_exists(name_ string) !bool

Check if page exists

fn (c Collection) page_get(name_ string) !&Page

Get a page by name

fn (mut c Collection) path() !pathlib.Path

Read content without processing includes

fn (c Collection) print_errors()

Print all errors to console

fn (mut c Collection) scan_groups() !

scan_groups scans the collection's directory for .group files and loads them into memory.

fn (mut c Collection) validate_links() !

Validate all links in collection

struct CollectionError {
pub mut:
	category CollectionErrorCategory
	page_key string // Format: "collection:page" or just collection name
	message  string
	file     string // Optional: specific file path if relevant
}

fn (e CollectionError) hash() string

Generate MD5 hash for error deduplication Hash is based on category + page_key (or file if page_key is empty)

fn (e CollectionError) str() string

Get human-readable error message

fn (e CollectionError) category_str() string

Get category as string

struct CollectionErrorArgs {
pub mut:
	category     CollectionErrorCategory @[required]
	message      string                  @[required]
	page_key     string
	file         string
	show_console bool // Show error in console immediately
	log_error    bool = true // Log to errors array (default: true)
}

struct CollectionExportArgs {
pub mut:
	destination pathlib.Path @[required]
	reset       bool = true
	include     bool = true // process includes during export
	redis       bool = true
}

struct CollectionNotFound {
	Error
pub:
	name string
	msg  string
}

fn (err CollectionNotFound) msg() string

struct ExportArgs {
pub mut:
	destination string @[requireds]
	reset       bool = true
	include     bool = true
	redis       bool = true
}

struct File {
pub mut:
	name       string   // name with extension
	path       string   // relative path of file in the collection
	ftype      FileType // file or image
	collection &Collection @[skip; str: skip] // Reference to parent collection
}

fn (mut f File) path() !pathlib.Path

Read content without processing includes

fn (f File) is_image() bool

fn (f File) ext() string

struct FileNotFound {
	Error
pub:
	collection string
	file       string
}

fn (err FileNotFound) msg() string

struct FixLinksArgs {
	include          bool // Process includes before fixing links
	cross_collection bool // Process cross-collection links (for export)
	export_mode      bool // Use export-style simple paths instead of filesystem paths
}

//////////////FIX PAGES FOR THE LINKS///////////////////////

struct Group {
pub mut:
	name     string   // normalized to lowercase
	patterns []string // email patterns, normalized to lowercase
}

fn (g Group) matches(email string) bool

Check if email matches any pattern in this group

struct GroupNewArgs {
pub mut:
	name     string   @[required]
	patterns []string @[required]
}

struct Link {
pub mut:
	src                    string // Source content where link was found (what to replace)
	text                   string // Link text [text]
	target                 string // Original link target (the source text)
	line                   int    // Line number where link was found (1-based)
	pos                    int    // Character position in line where link starts (0-based)
	target_collection_name string
	target_item_name       string
	status                 LinkStatus
	file_type              LinkFileType // Type of the link target: file, image, or page (default)
	page                   &Page @[skip; str: skip] // Reference to page where this link is found
}

Link represents a markdown link found in content

fn (mut self Link) target_page() !&Page

Get the target page this link points to

fn (mut self Link) target_file() !&File

Get the target file this link points to

struct NewPageArgs {
pub:
	name            string      @[required]
	path            string      @[required]
	collection_name string      @[required]
	collection      &Collection @[required]
}

struct Page {
pub mut:
	name            string
	path            string // in collection
	collection_name string
	links           []Link
	// macros          []Macro
	collection &Collection @[skip; str: skip] // Reference to parent collection
}

fn (mut p Page) path() !pathlib.Path

Read content without processing includes

fn (mut p Page) content(args ReadContentArgs) !string

Read content without processing includes

fn (p Page) key() string

struct PageNotFound {
	Error
pub:
	collection string
	page       string
}

fn (err PageNotFound) msg() string

struct ReadContentArgs {
pub mut:
	include bool
}

Read content with includes processed (default behavior)

struct ScanArgs {
pub mut:
	path   string @[required]
	ignore []string // list of directory names to ignore
}

Scan a path for collections

struct Session {
pub mut:
	user   string // username
	email  string // user's email (lowercase internally)
	params Params // additional context from request/webserver
}