Module @memberjunction/metadata-sync

MemberJunction Metadata Sync

A library for synchronizing MemberJunction database metadata with local file system representations. This library is integrated into the MemberJunction CLI (mj) and is accessed through mj sync commands. It enables developers and non-technical users to manage MJ metadata using their preferred editors and version control systems while maintaining the database as the source of truth.

Installation

MetadataSync is included with the MemberJunction CLI. Install the CLI globally:

npm install -g @memberjunction/cli

Then use the sync commands:

mj sync --help

Purpose

MemberJunction is a powerful metadata-driven system where configuration, business logic, AI prompts, templates, and more are stored as metadata in the database. This approach provides tremendous flexibility and runtime configurability, but it can create friction in modern development workflows.

Why This Tool Matters

For Developers:

  • Full IDE Support: Edit complex prompts and templates with syntax highlighting, IntelliSense, and all your favorite editor features
  • Version Control: Track every change with Git - see diffs, blame, history, and collaborate through pull requests
  • Branch-based Development: Work on features in isolation, test changes, and merge when ready
  • CI/CD Integration: Automatically deploy metadata changes as code moves through environments
  • Bulk Operations: Use familiar command-line tools (grep, sed, find) to make sweeping changes
  • Offline Development: Work on metadata without database connectivity

For Non-Technical Users:

  • Familiar Tools: Edit prompts in Word, Notepad++, or any text editor
  • No Database Access Needed: IT can set up sync, users just edit files
  • Folder Organization: Intuitive file/folder structure instead of database IDs
  • Easy Sharing: Send prompt files via email or shared drives
  • Simple Backups: Copy/paste folders for personal backups

For Organizations:

  • Migration Path: Metadata flows naturally from dev → staging → production with code
  • Compliance: Full audit trail through version control
  • Collaboration: Multiple team members can work on different metadata simultaneously
  • Disaster Recovery: File-based backups complement database backups
  • Cross-System Sync: Export from one MJ instance, import to another

The Best of Both Worlds

This tool preserves the power of MJ's metadata-driven architecture while adding the convenience of file-based workflows. The database remains the source of truth for runtime operations, while files become the medium for creation, editing, and deployment.

Overview

The Metadata Sync tool bridges the gap between database-stored metadata and file-based workflows by:

  • Pulling metadata entities from database to JSON files with external file support
  • Pushing local file changes back to the database
  • Supporting embedded collections for related entities
  • Enabling version control for all MJ metadata through Git
  • Supporting CI/CD workflows for metadata deployment
  • Providing a familiar file-based editing experience

Key Features

Hybrid File Storage

  • JSON files: Store structured metadata for entities
  • External files: Store large text fields (prompts, templates, etc.) in appropriate formats (.md, .html, .sql)
  • File references: Use @file:filename.ext to link external files from JSON

Embedded Collections

  • Related Entities: Store related records as arrays within parent JSON files
  • Hierarchical References: Use @parent: and @root: to reference parent/root entity fields
  • Automatic Metadata: Related entities maintain their own primaryKey and sync metadata
  • Nested Support: Support for multiple levels of nested relationships

Synchronization Operations

  • Pull: Download metadata from database to local files
    • Optionally pull related entities based on configuration
    • Filter support for selective pulling
  • Push: Upload local file changes to database
    • Process embedded collections automatically
    • Verbose mode (-v) for detailed output
  • Status: Show what would change without making modifications

Development Workflow Integration

  • Watch mode for automatic syncing during development
  • Dry-run mode to preview changes
  • CI/CD mode for automated deployments
  • Integration with existing mj.config.cjs configuration

Supported Entities

The tool works with any MemberJunction entity - both core system entities and user-created entities. Each entity type can have its own directory structure, file naming conventions, and related entity configurations.

Important Limitation: Database-Reflected Metadata

This tool should NOT be used to modify metadata that is reflected from the underlying database catalog. Examples include:

  • Entity field data types
  • Column lengths/precision
  • Primary key definitions
  • Foreign key relationships
  • Table/column existence

These properties are designed to flow from the database catalog up into MJ metadata, not the other way around. Attempting to modify these via file sync could create inconsistencies between the metadata and actual database schema.

The tool is intended for managing business-level metadata such as:

  • Descriptions and documentation
  • Display names and user-facing text
  • Categories and groupings
  • Custom properties and settings
  • AI prompts, templates, and other content
  • Permissions and security settings
  • Any other data that is not reflected up from the underlying system database catalogs

For more information about how CodeGen reflects system-level data from the database into the MJ metadata layer, see the CodeGen documentation.

Creating Error-Free Entity Files

Quick Start Checklist

Before creating entity JSON files, follow this checklist to avoid common mistakes:

✅ 1. Find the Entity Definition

  • Open packages/MJCoreEntities/src/generated/entity_subclasses.ts or packages/GeneratedEntities/src/generated/entity_subclasses.ts
  • Search for class [EntityName]Entity (e.g., class TemplateEntity)
  • Review JSDoc comments and property definitions to identify required vs optional fields

✅ 2. Check Required Fields

  • Look for JSDoc comments with @required annotations
  • Fields without ? in TypeScript definitions are typically required
  • Always include Name (almost always required)
  • Always include UserID (use System User ID: ECAFCCEC-6A37-EF11-86D4-000D3A4E707E)

✅ 3. Validate Field Names

  • Use exact field names from the BaseEntity class definition
  • Field names are case-sensitive
  • Don't assume fields exist (e.g., not all entities have Status)

✅ 4. Use Correct File Naming

  • Configuration files (.mj-sync.json, .mj-folder.json) must start with dot
  • Metadata files follow the filePattern in your .mj-sync.json
  • Most common: "filePattern": "*.json" (matches any .json file)
  • Alternative: "filePattern": ".*.json" (matches dot-prefixed .json files)

✅ 5. Set Up Directory Structure

  • Create .mj-sync.json in the entity directory
  • Use glob patterns: "filePattern": "*.json" (not regex: ".*.json")

Discovering Entity Structure

CRITICAL: Before creating entity files, you must understand the entity's field structure. Most errors occur because users are unfamiliar with the required fields, data types, and constraints.

Finding Entity Definitions

The approach depends on whether you're working inside or outside the MemberJunction monorepo:

Working Inside MJ Monorepo

Entity classes are located in:

  • Core MJ Entities: packages/MJCoreEntities/src/generated/entity_subclasses.ts

    • System entities like Users, Roles, EntityFields, etc.
    • AI-related entities like AI Prompts, AI Models, etc.

  • Custom Entities: packages/GeneratedEntities/src/generated/entity_subclasses.ts

    • Your application-specific entities
    • Business domain entities
Working Outside MJ Monorepo (In Your Own Project)

Entity classes are located in:

  • Core MJ Entities: node_modules/@memberjunction/core-entities/dist/generated/entity_subclasses.js

    • Note: This is compiled JavaScript, but your IDE should provide IntelliSense
    • For TypeScript definitions: node_modules/@memberjunction/core-entities/dist/generated/entity_subclasses.d.ts

  • Custom Entities: Your project's generated entities location (varies by project structure)

    • Common locations: src/generated/, packages/entities/, or similar
    • Look for files containing your custom entity classes
Best Practice: Use Your IDE's IntelliSense

Recommended approach for all scenarios:

  1. Import the entity class in your IDE:

    import { TemplateEntity } from '@memberjunction/core-entities';

  2. Create an instance and explore with IntelliSense:

    const template = new TemplateEntity();
    // Type "template." and let your IDE show available properties

  3. Check the class definition (F12 or "Go to Definition") to see:

    • JSDoc comments with field descriptions
    • Required vs optional fields
    • Field types and validation rules
    • Relationships and constraints

How to Find Required Fields

  1. Use IDE IntelliSense (Recommended):

    • Import the entity class
    • Create an instance: const entity = new TemplateEntity();
    • Use "Go to Definition" (F12) to see the BaseEntity class
    • Look for JSDoc comments and field definitions

  2. Examine the BaseEntity Class:

    • Find the entity class (e.g., class TemplateEntity)
    • Look at property declarations with JSDoc comments
    • Check for required vs optional field annotations
    • Review any validation methods or constraints

  3. Runtime Metadata Discovery:

    import { Metadata } from '@memberjunction/core';

    const md = new Metadata();
    const entityInfo = md.EntityByName('Templates');
    console.log('Required fields:', entityInfo.Fields.filter(f => !f.AllowsNull));

Example: Templates Entity Structure

// BaseEntity class (accessible via IDE IntelliSense)
export class TemplateEntity extends BaseEntity {
  /**
   * Primary key - auto-generated GUID
   */
  ID: string;
  
  /**
   * Template name - REQUIRED
   * @required
   */
  Name: string;
  
  /**
   * Template description - optional
   */
  Description?: string;
  
  /**
   * User who created this template - REQUIRED
   * Must be a valid User ID
   * @required
   * @foreignKey Users.ID
   */
  UserID: string;
  
  /**
   * Category for organizing templates - optional
   * @foreignKey TemplateCategories.ID
   */
  CategoryID?: string;
  
  // Note: Status field may not exist on all entities!
}

Common Required Fields Pattern

Most MJ entities follow these patterns:

Always Required:

  • ID - Primary key (GUID) - auto-generated if not provided
  • Name - Human-readable name
  • UserID - Creator/owner (use System User: ECAFCCEC-6A37-EF11-86D4-000D3A4E707E)

Often Required:

  • Description - Usually optional but recommended
  • Foreign key fields ending in ID - Check if they have .optional()

Be Careful With:

  • Status fields - Some entities have them, others don't
  • Enum fields - Must match exact values from database
  • DateTime fields - Use ISO format: 2024-01-15T10:30:00Z

Common Mistakes and Solutions

❌ Mistake 1: Using Non-Existent Fields

{
  "fields": {
    "Name": "My Template",
    "Status": "Active"  // ❌ Templates entity may not have Status field
  }
}

✅ Solution: Check the BaseEntity class first

// In entity_subclasses.ts - if you don't see Status here, don't use it
export class TemplateEntity extends BaseEntity {
  Name: string;  // Required
  Description?: string;  // Optional (note the ?)
  // No Status field defined
}

❌ Mistake 2: Missing Required Fields

{
  "fields": {
    "Name": "My Template"
    // ❌ Missing required UserID
  }
}

✅ Solution: Include all required fields

{
  "fields": {
    "Name": "My Template",
    "UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"
  }
}

❌ Mistake 3: Wrong File Pattern in .mj-sync.json

{
  "entity": "Templates",
  "filePattern": ".*.json"  // ❌ This is regex, not glob
}

✅ Solution: Use glob patterns

{
  "entity": "Templates", 
  "filePattern": "*.json"  // ✅ Correct glob pattern
}

❌ Mistake 4: Incorrect Data Types

{
  "fields": {
    "Name": "My Template",
    "CreatedAt": "2024-01-15",  // ❌ Wrong datetime format
    "Priority": "1"  // ❌ Should be number, not string
  }
}

✅ Solution: Use correct data types

{
  "fields": {
    "Name": "My Template",
    "CreatedAt": "2024-01-15T10:30:00Z",  // ✅ ISO format
    "Priority": 1  // ✅ Number type
  }
}

❌ Mistake 5: Files Not Being Detected

mydir/
├── .mj-sync.json (with "filePattern": "*.json")
├── template1.txt  // ❌ Wrong extension
└── .template2.json  // ❌ Dot prefix when pattern is "*.json"

✅ Solution: Match your filePattern

mydir/
├── .mj-sync.json (with "filePattern": "*.json")
├── template1.json  // ✅ Matches *.json pattern
└── template2.json  // ✅ Matches *.json pattern

Step-by-Step Entity File Creation

Step 1: Research the Entity

# Open in your IDE:
packages/MJCoreEntities/src/generated/entity_subclasses.ts

# Search for your entity class (Ctrl+F):
class TemplateEntity

# Note the required vs optional fields:
Name: string;           // Required (no ?)
UserID: string;         // Required (no ?)  
Description?: string;   // Optional (note the ?)

Step 2: Create Directory Structure

mkdir templates
cd templates

# Create entity config (dot-prefixed configuration file)
echo '{
  "entity": "Templates",
  "filePattern": "*.json"
}' > .mj-sync.json

Step 3: Create Your First Entity File

# Create metadata file (follows filePattern from .mj-sync.json)
echo '{
  "fields": {
    "Name": "My First Template",
    "Description": "A test template",
    "UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"
  }
}' > my-first-template.json

Step 4: Test and Validate

# Dry run to check for errors
mj sync push --dir="templates" --dry-run

# If successful, do actual push
mj sync push --dir="templates"

AI/LLM Guidelines

When using AI tools (like Claude, ChatGPT, etc.) to generate entity files:

🤖 For AI Assistants:

  1. Always check entity definitions first - Never assume field names or requirements
  2. Look up the exact BaseEntity class in the generated entity files
  3. Use the System User ID (ECAFCCEC-6A37-EF11-86D4-000D3A4E707E) for UserID fields
  4. Include only fields that exist in the entity definition
  5. Use proper data types as defined in the BaseEntity class
  6. Remember file naming rules:
    • Configuration files (.mj-sync.json) must have dot prefix
    • Metadata files follow the filePattern in .mj-sync.json
  7. Use glob patterns in .mj-sync.json, not regex patterns

📝 Prompt Template for AI:

I need to create entity files for the [EntityName] entity in MemberJunction.

Please:
1. First, check the entity definition in packages/MJCoreEntities/src/generated/entity_subclasses.ts
2. Find the class [EntityName]Entity (e.g., class TemplateEntity)
3. Review JSDoc comments and property definitions to identify required vs optional fields
4. Create a .mj-sync.json file with correct glob pattern
5. Create sample metadata JSON files following the filePattern
6. Use UserID: "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E" for required UserID fields
7. Follow the exact field names and data types from the BaseEntity class definition

CRITICAL: Configuration files (.mj-sync.json) must start with dot, but metadata files follow the filePattern specified in the configuration.

Understanding File Naming Rules

Configuration Files (Always Dot-Prefixed):

  • ✅ .mj-sync.json - Entity configuration
  • ✅ .mj-folder.json - Folder defaults
  • ❌ mj-sync.json - Won't be recognized

Metadata Files (Follow filePattern): With "filePattern": "*.json":

  • ✅ my-template.json - Will be processed
  • ✅ greeting.json - Will be processed
  • ❌ .my-template.json - Won't match pattern
  • ❌ package.json - Will be ignored (add to ignore list if needed)

With "filePattern": ".*.json":

  • ✅ .my-template.json - Will be processed
  • ✅ .greeting.json - Will be processed
  • ❌ my-template.json - Won't match pattern
  • ❌ package.json - Won't match pattern

Troubleshooting Quick Reference

Error Message Cause Solution
No entity directories found Missing .mj-sync.json or wrong filePattern Check .mj-sync.json exists and uses "*.json"
Field 'X' does not exist on entity 'Y' Using non-existent field Check BaseEntity class in entity_subclasses.ts
User ID cannot be null Missing required UserID Add "UserID": "ECAFCCEC-6A37-EF11-86D4-000D3A4E707E"
Processing 0 records Files don't match filePattern Check files match pattern in .mj-sync.json
Failed validation Wrong data type or format Check BaseEntity class for field types

System User ID Reference

Always use this GUID for UserID fields:

ECAFCCEC-6A37-EF11-86D4-000D3A4E707E

This is the System User ID that should be used when creating entity records through the MetadataSync tool. Using any other ID or leaving it null will cause validation errors.

File Structure

The tool uses a hierarchical directory structure with cascading defaults:

  • Each top-level directory represents an entity type
  • .mj-sync.json files define entities and base defaults
  • .mj-folder.json files define folder-specific defaults (optional)
  • Only dot-prefixed JSON files (e.g., .prompt-template.json, .category.json) are treated as metadata records
  • Regular JSON files without the dot prefix are ignored, allowing package.json and other config files to coexist
  • External files (.md, .html, etc.) are referenced from the JSON files
  • Defaults cascade down through the folder hierarchy

File Naming Convention

Metadata files must be prefixed with a dot (.) to be recognized by the sync tool. This convention:

  • Clearly distinguishes metadata files from regular configuration files
  • Allows package.json, tsconfig.json and other standard files to coexist without being processed
  • Follows established patterns like .gitignore and .eslintrc.json

Examples:

  • ✅ .greeting.json - Will be processed as metadata
  • ✅ .customer-prompt.json - Will be processed as metadata
  • ❌ greeting.json - Will be ignored
  • ❌ package.json - Will be ignored

File Format Options

Single Record per File (Default)

Each JSON file contains one record:

{
  "fields": { ... },
  "relatedEntities": { ... }
}

Multiple Records per File

JSON files can contain arrays of records:

[
  {
    "fields": { ... },
    "relatedEntities": { ... }
  },
  {
    "fields": { ... },
    "relatedEntities": { ... }
  }
]

This is useful for:

  • Grouping related records in a single file
  • Reducing file clutter for entities with many small records
  • Maintaining logical groupings while using @file: references for large content

Example Structure

metadata/
├── .mj-sync.json                    # Global sync configuration
├── ai-prompts/
│   ├── .mj-sync.json               # Defines entity: "AI Prompts"
│   ├── customer-service/
│   │   ├── .mj-folder.json         # Folder metadata (CategoryID, etc.)
│   │   ├── .greeting.json          # AI Prompt record with embedded models
│   │   ├── greeting.prompt.md      # Prompt content (referenced)
│   │   └── greeting.notes.md       # Notes field (referenced)
│   └── analytics/
│       ├── .mj-folder.json         # Folder metadata (CategoryID, etc.)
│       ├── .daily-report.json      # AI Prompt record
│       └── daily-report.prompt.md  # Prompt content (referenced)
├── templates/                       # Reusable JSON templates
│   ├── standard-prompt-settings.json # Common prompt configurations
│   ├── standard-ai-models.json     # Standard model configurations
│   ├── high-performance-models.json # High-power model configurations
│   └── customer-service-defaults.json # CS-specific defaults
└── template-entities/
    ├── .mj-sync.json               # Defines entity: "Templates"
    ├── email/
    │   ├── .mj-folder.json         # Folder metadata
    │   ├── .welcome.json           # Template record (dot-prefixed)
    │   └── welcome.template.html   # Template content (referenced)
    └── reports/
        ├── .mj-folder.json         # Folder metadata
        ├── .invoice.json           # Template record (dot-prefixed)
        └── invoice.template.html   # Template content (referenced)

JSON Metadata Format

Individual Record (e.g., ai-prompts/customer-service/.greeting.json)

{
  "fields": {
    "Name": "Customer Greeting",
    "Description": "Friendly customer service greeting",
    "TypeID": "@lookup:AI Prompt Types.Name=Chat",
    "CategoryID": "@lookup:AI Prompt Categories.Name=Customer Service",
    "Temperature": 0.7,
    "MaxTokens": 1000,
    "Prompt": "@file:greeting.prompt.md",
    "Notes": "@file:../shared/notes/greeting-notes.md",
    "SystemPrompt": "@url:https://raw.githubusercontent.com/company/prompts/main/system/customer-service.md"
  },
  "primaryKey": {
    "ID": "550e8400-e29b-41d4-a716-446655440000"
  },
  "sync": {
    "lastModified": "2024-01-15T10:30:00Z",
    "checksum": "sha256:abcd1234..."
  }
}

Record with Embedded Collections

{
  "fields": {
    "Name": "Customer Service Chat",
    "Description": "Main customer service prompt",
    "TypeID": "@lookup:AI Prompt Types.Name=Chat",
    "TemplateText": "@file:customer-service.md",
    "Status": "Active"
  },
  "relatedEntities": {
    "MJ: AI Prompt Models": [
      {
        "fields": {
          "PromptID": "@parent:ID",
          "ModelID": "@lookup:AI Models.Name=GPT 4.1",
          "VendorID": "@lookup:MJ: AI Vendors.Name=OpenAI",
          "Priority": 1,
          "Status": "Active"
        },
        "primaryKey": {
          "ID": "BFA2433E-F36B-1410-8DB0-00021F8B792E"
        },
        "sync": {
          "lastModified": "2025-06-07T17:18:31.687Z",
          "checksum": "a642ebea748cb1f99467af2a7e6f4ffd3649761be27453b988af973bed57f070"
        }
      },
      {
        "fields": {
          "PromptID": "@parent:ID",
          "ModelID": "@lookup:AI Models.Name=Claude 4 Sonnet",
          "Priority": 2,
          "Status": "Active"
        }
      }
    ]
  },
  "primaryKey": {
    "ID": "C2A1433E-F36B-1410-8DB0-00021F8B792E"
  },
  "sync": {
    "lastModified": "2025-06-07T17:18:31.698Z",
    "checksum": "7cbd241cbf0d67c068c1434e572a78c87bb31751cbfe7734bfd32f8cea17a2c9"
  }
}

Composite Primary Key Example

{
  "primaryKey": {
    "UserID": "550e8400-e29b-41d4-a716-446655440000",
    "RoleID": "660f9400-f39c-51e5-b827-557766551111"
  },
  "fields": {
    "GrantedAt": "2024-01-15T10:30:00Z",
    "GrantedBy": "@lookup:Users.Email=admin@company.com",
    "ExpiresAt": "2025-01-15T10:30:00Z",
    "Notes": "@file:user-role-notes.md"
  },
  "sync": {
    "lastModified": "2024-01-15T10:30:00Z",
    "checksum": "sha256:abcd1234..."
  }
}

Default Value Inheritance

The tool implements a cascading inheritance system for field defaults, similar to CSS or OOP inheritance:

  1. Entity-level defaults (in .mj-sync.json) - Base defaults for all records
  2. Folder-level defaults (in .mj-folder.json) - Override/extend entity defaults
  3. Nested folder defaults - Override/extend parent folder defaults
  4. Record-level values - Override all inherited defaults

Inheritance Example

ai-prompts/.mj-sync.json         → Temperature: 0.7, MaxTokens: 1500
├── customer-service/.mj-folder.json → Temperature: 0.8 (overrides)
│   ├── greeting.json            → Uses Temperature: 0.8, MaxTokens: 1500
│   └── escalation/.mj-folder.json → Temperature: 0.6 (overrides again)
│       └── urgent.json          → Temperature: 0.9 (record override)

Final values for urgent.json:

  • Temperature: 0.9 (from record)
  • MaxTokens: 1500 (from entity defaults)
  • All other fields from folder hierarchy

Special Conventions

The tool supports special reference types that can be used in ANY field that accepts text content. These references are processed during push/pull operations to handle external content, lookups, and environment-specific values.

Primary Key Handling

The tool automatically detects primary key fields from entity metadata:

  • Single primary keys: Most common, stored as {"ID": "value"} or {"CustomKeyName": "value"}
  • Composite primary keys: Multiple fields that together form the primary key
  • Auto-detection: Tool reads entity metadata to determine primary key structure
  • No hardcoding: Works with any primary key field name(s)

deleteRecord Directive

The tool now supports deleting records from the database using a special deleteRecord directive in JSON files. This allows you to remove obsolete records as part of your metadata sync workflow:

How to Delete a Record

  1. Add a deleteRecord section to any record JSON file
  2. Set delete: true to mark the record for deletion
  3. Run mj sync push to execute the deletion
  4. The tool will update the JSON with a deletion timestamp

Syntax

{
  "fields": {
    "Name": "Obsolete Prompt",
    "Description": "This prompt is no longer needed"
  },
  "primaryKey": {
    "ID": "550e8400-e29b-41d4-a716-446655440000"
  },
  "deleteRecord": {
    "delete": true
  }
}

After Deletion

After successfully deleting the record, the tool updates the JSON file:

{
  "fields": {
    "Name": "Obsolete Prompt",
    "Description": "This prompt is no longer needed"
  },
  "primaryKey": {
    "ID": "550e8400-e29b-41d4-a716-446655440000"
  },
  "deleteRecord": {
    "delete": true,
    "deletedAt": "2024-01-15T14:30:00.000Z"
  }
}

Important Notes

  • Primary key required: You must specify the primaryKey to identify which record to delete
  • One-time operation: Once deletedAt is set, the deletion won't be attempted again
  • SQL logging: Delete operations are included in SQL logs when enabled
  • Foreign key constraints: Deletions may fail if other records reference this record
  • Dry-run support: Use --dry-run to preview what would be deleted
  • Takes precedence: If deleteRecord is present, normal create/update operations are skipped

Use Cases

  • Removing deprecated prompts, templates, or configurations
  • Cleaning up test data
  • Synchronizing deletions across environments
  • Maintaining clean metadata through version control

@file: References

When a field value starts with @file:, the tool will:

  1. Read content from the specified file for push operations
  2. Write content to the specified file for pull operations
  3. Track both files for change detection
  4. For JSON files: Automatically process any @include directives within them

Examples:

  • @file:greeting.prompt.md - File in same directory as JSON
  • @file:./shared/common-prompt.md - Relative path
  • @file:../templates/standard-header.md - Parent directory reference
  • @file:spec.json - JSON file with @include directives (processed automatically)

@url: References

When a field value starts with @url:, the tool will:

  1. Fetch content from the URL during push operations
  2. Cache the content with appropriate headers
  3. Support both HTTP(S) and file:// protocols

Examples:

  • @url:https://example.com/prompts/greeting.md - Remote content
  • @url:https://raw.githubusercontent.com/company/prompts/main/customer.md - GitHub raw content
  • @url:file:///shared/network/drive/prompts/standard.md - Local file URL

@lookup: References (ENHANCED)

Enable entity relationships using human-readable values:

  • Basic syntax: @lookup:EntityName.FieldName=Value
  • Multi-field syntax: @lookup:EntityName.Field1=Value1&Field2=Value2
  • Auto-create syntax: @lookup:EntityName.FieldName=Value?create
  • With additional fields: @lookup:EntityName.FieldName=Value?create&Field2=Value2

Examples:

  • @lookup:AI Prompt Types.Name=Chat - Single field lookup, fails if not found
  • @lookup:Users.Email=john@example.com&Department=Sales - Multi-field lookup for precise matching
  • @lookup:AI Prompt Categories.Name=Examples?create - Creates if missing
  • @lookup:AI Prompt Categories.Name=Examples?create&Description=Example prompts - Creates with description

Multi-Field Lookups

When you need to match records based on multiple criteria, use the multi-field syntax:

{
  "CategoryID": "@lookup:AI Prompt Categories.Name=Actions&Status=Active",
  "ManagerID": "@lookup:Users.Email=manager@company.com&Department=Engineering&Status=Active"
}

This ensures you get the exact record you want when multiple records might have the same value in a single field.

@parent: References

Reference fields from the immediate parent entity in embedded collections:

  • @parent:ID - Get the parent's ID field
  • @parent:Name - Get the parent's Name field
  • Works with any field from the parent entity

@root: References

Reference fields from the root entity in nested structures:

  • @root:ID - Get the root entity's ID
  • @root:CategoryID - Get the root's CategoryID
  • Useful for deeply nested relationships

@env: References

Support environment-specific values:

  • @env:VARIABLE_NAME
  • Useful for different environments (dev/staging/prod)

Automatic JSON Stringification

When a field value is an array or object, the tool automatically converts it to a JSON string for database storage:

  • Arrays and objects are detected and stringified with pretty formatting (2-space indentation)
  • Maintains clean, readable JSON in source files while storing as strings in database
  • Works seamlessly with all field types that accept text content

Examples:

{
  "fields": {
    "Name": "My Entity",
    "Configuration": {
      "setting1": "value1",
      "setting2": {
        "nested": true,
        "items": [1, 2, 3]
      }
    },
    "Tags": ["tag1", "tag2", "tag3"],
    "Metadata": {
      "created": "2024-01-15",
      "author": "John Doe"
    }
  }
}

The Configuration, Tags, and Metadata fields will automatically be converted to JSON strings when pushed to the database, while maintaining their structured format in your source files.

{@include } References in Files

Enable content composition within non-JSON files (like .md, .html, .txt) using JSDoc-style include syntax:

  • Pattern: {@include path/to/file.ext}
  • Supports relative paths from the containing file
  • Recursive includes (includes within includes)
  • Circular reference detection prevents infinite loops
  • Works seamlessly with @file: references

How It Works

When a JSON metadata file uses @file: to reference an external file, the MetadataSync tool:

  1. Loads the referenced file
  2. Scans for {@include} patterns
  3. Recursively resolves all includes
  4. Returns the fully composed content

Example Usage

# My Prompt Template

## System Instructions
{@include ./shared/system-instructions.md}

## Context
{@include ../common/context-header.md}

## Task
Please analyze the following...

Complex Example with Nested Includes

Directory structure:

prompts/
├── customer-service/
│   ├── greeting.json          # Uses @file:greeting.md
│   ├── greeting.md            # Contains {@include} references
│   └── shared/
│       ├── tone.md
│       └── guidelines.md
└── common/
    ├── company-info.md
    └── legal-disclaimer.md

greeting.json:

{
  "fields": {
    "Name": "Customer Greeting",
    "Prompt": "@file:greeting.md"
  }
}

greeting.md:

# Customer Service Greeting

{@include ./shared/tone.md}

## Guidelines
{@include ./shared/guidelines.md}

## Company Information
{@include ../common/company-info.md}

## Legal
{@include ../common/legal-disclaimer.md}

The final content pushed to the database will have all includes fully resolved.

Benefits:

  • DRY Principle: Share common content across multiple files
  • Maintainability: Update shared content in one place
  • Flexibility: Build complex documents from modular parts
  • Validation: Automatic checking of included file existence and circular references

Settings

Member Visibility

Theme

On This Page

MemberJunction Metadata Sync