The MemberJunction AI Vector Database package provides a standardized interface and base classes for working with vector databases in the MemberJunction ecosystem. This package serves as a common abstraction layer that allows different vector database implementations to be used interchangeably.
npm install @memberjunction/ai-vectordb
The abstract base class that all vector database providers must implement:
export abstract class VectorDBBase {
// Protected getter for API key access by subclasses
protected get apiKey(): string;
constructor(apiKey: string);
// Index operations
abstract listIndexes(): IndexList | Promise<IndexList>;
abstract getIndex(params: BaseRequestParams): BaseResponse | Promise<BaseResponse>;
abstract createIndex(params: CreateIndexParams): BaseResponse | Promise<BaseResponse>;
abstract deleteIndex(params: BaseRequestParams): BaseResponse | Promise<BaseResponse>;
abstract editIndex(params: EditIndexParams): BaseResponse | Promise<BaseResponse>;
abstract queryIndex(params: QueryOptions): BaseResponse | Promise<BaseResponse>;
// Record operations
abstract createRecord(record: VectorRecord): BaseResponse | Promise<BaseResponse>;
abstract createRecords(record: VectorRecord[]): BaseResponse | Promise<BaseResponse>;
abstract getRecord(param: BaseRequestParams): BaseResponse | Promise<BaseResponse>;
abstract getRecords(params: BaseRequestParams): BaseResponse | Promise<BaseResponse>;
abstract updateRecord(record: UpdateOptions): BaseResponse | Promise<BaseResponse>;
abstract updateRecords(records: UpdateOptions): BaseResponse | Promise<BaseResponse>;
abstract deleteRecord(record: VectorRecord): BaseResponse | Promise<BaseResponse>;
abstract deleteRecords(records: VectorRecord[]): BaseResponse | Promise<BaseResponse>;
}
export type VectorRecord<T extends RecordMetadata = RecordMetadata> = {
id: string; // Unique identifier for the record
values: RecordValues; // Vector embedding values (array of numbers)
sparseValues?: RecordSparseValues; // Optional sparse representation for hybrid search
metadata?: T; // Optional metadata for filtering and identification
};
export type RecordValues = Array<number>;
export type RecordSparseValues = {
indices: Array<number>; // List of indices where non-zero values are present
values: Array<number>; // The values that correspond to the positions in indices
};
export type RecordMetadataValue = string | boolean | number | Array<string>;
export type RecordMetadata = Record<string, RecordMetadataValue>;
export type IndexDescription = {
name: string; // Name of the index
dimension: number; // Vector dimension size
metric: IndexModelMetricEnum; // Distance metric: 'cosine', 'euclidean', or 'dotproduct'
host: string; // Host where the index is located
};
For similarity search in vector databases:
// Base query parameters
export type QueryParamsBase = {
topK: number; // Number of results to return
includeValues?: boolean; // Whether to include vector values in results
includeMetadata?: boolean; // Whether to include metadata in results
filter?: object; // Metadata filter to apply
};
// Query by vector values
export type QueryByVectorValues = QueryParamsBase & {
vector: RecordValues; // The query vector to find similar vectors for
};
// Query by record ID
export type QueryByRecordId = QueryParamsBase & {
id: string; // Use an existing record's vector to query
};
// Combined query options type
export type QueryOptions = QueryByRecordId | QueryByVectorValues;
export type QueryResponse<T extends RecordMetadata = RecordMetadata> = {
matches: Array<ScoredRecord<T>>; // Results sorted by similarity
namespace: string; // Namespace where query was executed
usage?: OperationUsage; // Usage information
};
export interface ScoredRecord<T extends RecordMetadata = RecordMetadata> extends VectorRecord<T> {
score?: number; // Similarity score (interpretation depends on metric)
}
export type OperationUsage = {
readUnits?: number; // Number of read units consumed by operation
};
While this package primarily provides interfaces and base classes, here's how it would be used with a concrete implementation:
Create a provider by implementing the VectorDBBase abstract class:
import { VectorDBBase, VectorRecord, BaseResponse, CreateIndexParams } from '@memberjunction/ai-vectordb';
class MyVectorDBProvider extends VectorDBBase {
constructor(apiKey: string) {
super(apiKey);
}
async listIndexes() {
// Implementation for listing indexes
return {
indexes: [
// Index descriptions
]
};
}
async createIndex(params: CreateIndexParams): Promise<BaseResponse> {
// Implementation for creating an index
try {
// Provider-specific code
return {
success: true,
message: 'Index created successfully',
data: { /* index info */ }
};
} catch (error) {
return {
success: false,
message: error.message,
data: null
};
}
}
// Implement remaining methods...
}
Once a provider is implemented, you can use it with a consistent API:
import { VectorDBBase, VectorRecord } from '@memberjunction/ai-vectordb';
import { MyVectorDBProvider } from './my-vector-db-provider';
async function workWithVectors() {
// Initialize the provider
const vectorDB: VectorDBBase = new MyVectorDBProvider('your-api-key');
// Create an index
const createResult = await vectorDB.createIndex({
id: 'my-index',
dimension: 1536,
metric: 'cosine' as IndexModelMetricEnum
});
if (createResult.success) {
console.log('Index created:', createResult.data);
}
// Insert a vector
const insertResult = await vectorDB.createRecord({
id: 'record-1',
values: [0.1, 0.2, 0.3, /* ... */],
metadata: {
category: 'document',
title: 'Sample Document'
}
});
// Query for similar vectors
const queryResult = await vectorDB.queryIndex({
vector: [0.1, 0.2, 0.3, /* ... */],
topK: 5,
includeMetadata: true,
filter: {
category: 'document'
}
});
if (queryResult.success) {
console.log('Similar vectors:', queryResult.data.matches);
}
}
MemberJunction provides implementations for popular vector databases:
@memberjunction/ai-vectors-pinecone - Implementation for Pinecone vector databaseYou can also create your own implementation for any vector database service by extending the VectorDBBase class.
This package integrates with the broader MemberJunction AI vector ecosystem:
@memberjunction/ai-vectors - Core vector functionality@memberjunction/ai-vectors-sync - Synchronize entity data to vector databases@memberjunction/ai-vectors-dupe - Duplicate detection using vector similarityThe VectorDBBase constructor validates the API key and throws an error if it's empty or invalid:
try {
const vectorDB = new MyVectorDBProvider('');
} catch (error) {
// Error: API key cannot be empty
}
All methods return a standardized BaseResponse format:
export type BaseResponse = {
success: boolean; // Whether the operation succeeded
message: string; // Human-readable message about the operation
data: any; // Operation-specific response data
};
export type BaseRequestParams = {
id: string;
data?: any;
};
export type CreateIndexParams = BaseRequestParams & {
dimension: number;
metric: IndexModelMetricEnum;
additionalParams?: any;
};
export type EditIndexParams = BaseRequestParams & {
// Additional fields specific to the provider
};
@memberjunction/core: ^2.43.0 - MemberJunction core library@memberjunction/global: ^2.43.0 - MemberJunction global utilitiesdotenv: ^16.4.1 - Environment variable management@types/node: 20.14.2ts-node-dev: ^2.0.0typescript: ^5.4.5npm run build - Compile TypeScript to JavaScriptnpm run start - Run the development server with ts-node-devnpm test - Run tests (currently not implemented)ISC