Class WebPageContentAction

Action that retrieves and processes web content in various formats Supports JSON, PDF, DOCX, XML, CSV, HTML and more - similar to Claude's web reading capabilities

Example

// Get JSON API response
await runAction({
  ActionName: 'Web Page Content',
  Params: [{
    Name: 'URL',
    Value: 'https://api.example.com/data'
  }, {
    Name: 'ContentType',
    Value: 'json'
  }]
});

// Get PDF content as text
await runAction({
  ActionName: 'Web Page Content',
  Params: [{
    Name: 'URL',
    Value: 'https://example.com/document.pdf'
  }, {
    Name: 'ContentType',
    Value: 'text'
  }]
});

// Get DOCX content as markdown
await runAction({
  ActionName: 'Web Page Content',
  Params: [{
    Name: 'URL',
    Value: 'https://example.com/document.docx'
  }, {
    Name: 'ContentType',
    Value: 'markdown'
  }]
});

Hierarchy (view full)

BaseAction
- WebPageContentAction

Constructors

constructor

new WebPageContentAction(): WebPageContentAction
Returns WebPageContentAction
Overrides BaseAction.constructor
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:62

Properties

`Private` `Readonly` MAX_CONTENT_SIZE

MAX_CONTENT_SIZE: number = ...

`Private` `Readonly` MAX_IMAGE_SIZE

MAX_IMAGE_SIZE: number = ...

`Private` turndown

turndown: TurndownService

Methods

`Protected` InternalRunAction

InternalRunAction(params): Promise<ActionResultSimple>
Executes web content retrieval and processing
Parameters
- params: RunActionParams<any>
  The action parameters containing:
  - URL: Web resource URL to fetch (required)
  - ContentType: Output format - 'auto', 'text', 'html', 'markdown', 'json' (default: 'auto')
  - ExtractMainContent: Extract only main content vs full page (default: false)
  - IncludeMetadata: Include resource metadata in response (default: true)
  - MaxContentLength: Maximum content length to return (default: 100000 chars)
Returns Promise<ActionResultSimple>
Processed content in the specified format
Overrides BaseAction.InternalRunAction
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:83

Run

Run(params): Promise<ActionResultSimple>
Executes the action with the provided parameters.
Parameters
- params: RunActionParams<any>
  The action execution parameters including context
Returns Promise<ActionResultSimple>
Promise resolving to the action result
Inherited from BaseAction.Run
- Defined in Engine/dist/generic/BaseAction.d.ts:37

`Private` csvToMarkdownTable

csvToMarkdownTable(data, fields): string
Converts CSV data to markdown table
Parameters
- data: Record<string, string | number | boolean>[]
- fields: string[]
Returns string
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:671

`Private` delay

delay(ms): Promise<void>
Delays execution for specified milliseconds
Parameters
- ms: number
Returns Promise<void>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:804

`Private` detectContentType

detectContentType(contentType, pathname): string
Detects content type from response headers and URL
Parameters
- contentType: string
- pathname: string
Returns string
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:189

`Private` extractHtmlMetadata

extractHtmlMetadata(html): Record<string, unknown>
Extracts HTML metadata
Parameters
- html: string
Returns Record<string, unknown>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:566

`Private` extractMainContent

extractMainContent(html): string
Attempts to extract main content from HTML
Parameters
- html: string
Returns string
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:599

`Private` extractStructure

extractStructure(doc): Record<string, unknown>
Extracts document structure from HTML
Parameters
- doc: Document
Returns Record<string, unknown>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:637

`Private` fetchWithRetry

fetchWithRetry(url, options, maxRetries?): Promise<Response>
Fetches URL with retry logic for transient failures
Parameters
- url: string
- options: RequestInit
- maxRetries: number = 3
Returns Promise<Response>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:730

`Private` getBrowserHeaders

getBrowserHeaders(hostname): Record<string, string>
Returns browser-like headers to avoid bot detection
Parameters
- hostname: string
Returns Record<string, string>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:708

`Private` getRetryDelay

getRetryDelay(attempt): number
Calculates exponential backoff delay for retries
Parameters
- attempt: number
Returns number
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:796

`Private` htmlToText

htmlToText(html): string
Converts HTML to plain text
Parameters
- html: string
Returns string
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:624

`Private` inferOutputType

inferOutputType(detectedType): string
Infers best output type based on detected content type
Parameters
- detectedType: string
Returns string
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:263

`Private` isRetryableError

isRetryableError(error): boolean
Determines if an error is retryable
Parameters
- error: Error
Returns boolean
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:781

`Private` processCsv

processCsv(response, outputType, includeMetadata): Promise<Record<string, unknown>>
Process CSV content
Parameters
- response: Response
- outputType: string
- includeMetadata: boolean
Returns Promise<Record<string, unknown>>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:390

`Private` processDocx

processDocx(response, outputType, includeMetadata, maxLength): Promise<Record<string, unknown>>
Process DOCX content
Parameters
- response: Response
- outputType: string
- includeMetadata: boolean
- maxLength: number
Returns Promise<Record<string, unknown>>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:349

`Private` processHtml

processHtml(response, outputType, extractMain, includeMetadata, maxLength): Promise<Record<string, unknown>>
Process HTML content
Parameters
- response: Response
- outputType: string
- extractMain: boolean
- includeMetadata: boolean
- maxLength: number
Returns Promise<Record<string, unknown>>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:461

`Private` processImage

processImage(response, includeMetadata): Promise<Record<string, unknown>>
Process image content
Parameters
- response: Response
- includeMetadata: boolean
Returns Promise<Record<string, unknown>>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:510

`Private` processJson

processJson(response, outputType, includeMetadata): Promise<Record<string, unknown>>
Process JSON content
Parameters
- response: Response
- outputType: string
- includeMetadata: boolean
Returns Promise<Record<string, unknown>>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:282

`Private` processPdf

processPdf(response, outputType, includeMetadata, maxLength): Promise<Record<string, unknown>>
Process PDF content
Parameters
- response: Response
- outputType: string
- includeMetadata: boolean
- maxLength: number
Returns Promise<Record<string, unknown>>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:314

`Private` processResponse

processResponse(response, detectedType, requestedType, extractMain, includeMetadata, maxLength): Promise<Record<string, unknown>>
Processes response based on detected content type
Parameters
- response: Response
- detectedType: string
- requestedType: string
- extractMain: boolean
- includeMetadata: boolean
- maxLength: number
Returns Promise<Record<string, unknown>>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:221

`Private` processText

processText(response, outputType, maxLength): Promise<Record<string, unknown>>
Process plain text content
Parameters
- response: Response
- outputType: string
- maxLength: number
Returns Promise<Record<string, unknown>>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:543

`Private` processXml

processXml(response, outputType, includeMetadata): Promise<Record<string, unknown>>
Process XML content
Parameters
- response: Response
- outputType: string
- includeMetadata: boolean
Returns Promise<Record<string, unknown>>
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:425

`Private` shouldRetry

shouldRetry(status): boolean
Determines if HTTP status code warrants a retry
Parameters
- status: number
Returns boolean
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:773

`Private` truncateContent

truncateContent(content, maxLength): string
Truncates content to specified length
Parameters
- content: string
- maxLength: number
Returns string
- Defined in CoreActions/src/custom/web/web-page-content.action.ts:698

Class WebPageContentAction

Example

Hierarchy (view full)

Index

Constructors

Properties

Methods

Constructors

constructor

Returns WebPageContentAction

Properties

Private Readonly MAX_CONTENT_SIZE

Private Readonly MAX_IMAGE_SIZE

Private turndown

Methods

Protected InternalRunAction

Parameters

Returns Promise<ActionResultSimple>

Run

Parameters

Returns Promise<ActionResultSimple>

Private csvToMarkdownTable

Parameters

Returns string

Private delay

Parameters

Returns Promise<void>

Private detectContentType

Parameters

Returns string

Private extractHtmlMetadata

Parameters

Returns Record<string, unknown>

Private extractMainContent

Parameters

Returns string

Private extractStructure

Parameters

Returns Record<string, unknown>

Private fetchWithRetry

Parameters

Returns Promise<Response>

Private getBrowserHeaders

Parameters

Returns Record<string, string>

Private getRetryDelay

Parameters

Returns number

Private htmlToText

Parameters

Returns string

Private inferOutputType

Parameters

Returns string

Private isRetryableError

Parameters

Returns boolean

Private processCsv

Parameters

Returns Promise<Record<string, unknown>>

Private processDocx

Parameters

Returns Promise<Record<string, unknown>>

Private processHtml

Parameters

Returns Promise<Record<string, unknown>>

Private processImage

Parameters

Returns Promise<Record<string, unknown>>

Private processJson

Parameters

Returns Promise<Record<string, unknown>>

Private processPdf

Parameters

Returns Promise<Record<string, unknown>>

Private processResponse

Parameters

Returns Promise<Record<string, unknown>>

Private processText

Parameters

`Private` `Readonly` MAX_CONTENT_SIZE

`Private` `Readonly` MAX_IMAGE_SIZE

`Private` turndown

`Protected` InternalRunAction

`Private` csvToMarkdownTable

`Private` delay

`Private` detectContentType

`Private` extractHtmlMetadata

`Private` extractMainContent

`Private` extractStructure

`Private` fetchWithRetry

`Private` getBrowserHeaders

`Private` getRetryDelay

`Private` htmlToText

`Private` inferOutputType

`Private` isRetryableError

`Private` processCsv

`Private` processDocx

`Private` processHtml

`Private` processImage

`Private` processJson

`Private` processPdf

`Private` processResponse

`Private` processText

`Private` processXml

`Private` shouldRetry

`Private` truncateContent