Sync website crawl deltas into Qdrant with Apify and OpenAI embeddings

{
  "_source": "n8n.io",
  "workflow": {
    "id": 19911,
    "name": "Sync website crawl deltas into Qdrant with Apify and OpenAI embeddings",
    "totalViews": 14,
    "price": null,
    "purchaseUrl": null,
    "recentViews": 2,
    "createdAt": "2026-09-24T14:26:30.721Z",
    "user": {
      "username": "apify",
      "verified": true
    },
    "readyToDemo": null,
    "nodes": [
      {
        "id": 19,
        "icon": "file:httprequest.svg",
        "name": "n8n-nodes-base.httpRequest",
        "codex": {
          "data": {
            "alias": [
              "API",
              "Request",
              "URL",
              "Build",
              "cURL"
            ],
            "resources": {
              "generic": [
                {
                  "url": "https://n8n.io/blog/2021-the-year-to-automate-the-new-you-with-n8n/",
                  "icon": "\u2600\ufe0f",
                  "label": "2021: The Year to Automate the New You with n8n"
                },
                {
                  "url": "https://n8n.io/blog/why-business-process-automation-with-n8n-can-change-your-daily-life/",
                  "icon": "\ud83e\uddec",
                  "label": "Why business process automation with n8n can change your daily life"
                },
                {
                  "url": "https://n8n.io/blog/automatically-pulling-and-visualizing-data-with-n8n/",
                  "icon": "\ud83d\udcc8",
                  "label": "Automatically pulling and visualizing data with n8n"
                },
                {
                  "url": "https://n8n.io/blog/learn-how-to-automatically-cross-post-your-content-with-n8n/",
                  "icon": "\u270d\ufe0f",
                  "label": "Learn how to automatically cross-post your content with n8n"
                },
                {
                  "url": "https://n8n.io/blog/automatically-adding-expense-receipts-to-google-sheets-with-telegram-mindee-twilio-and-n8n/",
                  "icon": "\ud83e\uddfe",
                  "label": "Automatically Adding Expense Receipts to Google Sheets with Telegram, Mindee, Twilio, and n8n"
                },
                {
                  "url": "https://n8n.io/blog/running-n8n-on-ships-an-interview-with-maranics/",
                  "icon": "\ud83d\udef3",
                  "label": "Running n8n on ships: An interview with Maranics"
                },
                {
                  "url": "https://n8n.io/blog/what-are-apis-how-to-use-them-with-no-code/",
                  "icon": " \ud83e\udea2",
                  "label": "What are APIs and how to use them with no code"
                },
                {
                  "url": "https://n8n.io/blog/5-tasks-you-can-automate-with-notion-api/",
                  "icon": "\u26a1\ufe0f",
                  "label": "5 tasks you can automate with the new Notion API "
                },
                {
                  "url": "https://n8n.io/blog/world-poetry-day-workflow/",
                  "icon": "\ud83d\udcdc",
                  "label": "Celebrating World Poetry Day with a daily poem in Telegram"
                },
                {
                  "url": "https://n8n.io/blog/automate-google-apps-for-productivity/",
                  "icon": "\ud83d\udca1",
                  "label": "15 Google apps you can combine and automate to increase productivity"
                },
                {
                  "url": "https://n8n.io/blog/automate-designs-with-bannerbear-and-n8n/",
                  "icon": "\ud83c\udfa8",
                  "label": "Automate Designs with Bannerbear and n8n"
                },
                {
                  "url": "https://n8n.io/blog/how-uproc-scraped-a-multi-page-website-with-a-low-code-workflow/",
                  "icon": " \ud83d\udd78\ufe0f",
                  "label": "How uProc scraped a multi-page website with a low-code workflow"
                },
                {
                  "url": "https://n8n.io/blog/building-an-expense-tracking-app-in-10-minutes/",
                  "icon": "\ud83d\udcf1",
                  "label": "Building an expense tracking app in 10 minutes"
                },
                {
                  "url": "https://n8n.io/blog/5-workflow-automations-for-mattermost-that-we-love-at-n8n/",
                  "icon": "\ud83e\udd16",
                  "label": "5 workflow automations for Mattermost that we love at n8n"
                },
                {
                  "url": "https://n8n.io/blog/how-to-use-the-http-request-node-the-swiss-army-knife-for-workflow-automation/",
                  "icon": "\ud83e\uddf0",
                  "label": "How to use the HTTP Request Node - The Swiss Army Knife for Workflow Automation"
                },
                {
                  "url": "https://n8n.io/blog/learn-how-to-use-webhooks-with-mattermost-slash-commands/",
                  "icon": "\ud83e\udd84",
                  "label": "Learn how to use webhooks with Mattermost slash commands"
                },
                {
                  "url": "https://n8n.io/blog/how-a-membership-development-manager-automates-his-work-and-investments/",
                  "icon": "\ud83d\udcc8",
                  "label": "How a Membership Development Manager automates his work and investments"
                },
                {
                  "url": "https://n8n.io/blog/a-low-code-bitcoin-ticker-built-with-questdb-and-n8n-io/",
                  "icon": "\ud83d\udcc8",
                  "label": "A low-code bitcoin ticker built with QuestDB and n8n.io"
                },
                {
                  "url": "https://n8n.io/blog/how-to-set-up-a-ci-cd-pipeline-with-no-code/",
                  "icon": "\ud83c\udfa1",
                  "label": "How to set up a no-code CI/CD pipeline with GitHub and TravisCI"
                },
                {
                  "url": "https://n8n.io/blog/automations-for-activists/",
                  "icon": "\u2728",
                  "label": "How Common Knowledge use workflow automation for activism"
                },
                {
                  "url": "https://n8n.io/blog/creating-scheduled-text-affirmations-with-n8n/",
                  "icon": "\ud83e\udd1f",
                  "label": "Creating scheduled text affirmations with n8n"
                },
                {
                  "url": "https://n8n.io/blog/how-goomer-automated-their-operations-with-over-200-n8n-workflows/",
                  "icon": "\ud83d\udef5",
                  "label": "How Goomer automated their operations with over 200 n8n workflows"
                },
                {
                  "url": "https://n8n.io/blog/aws-workflow-automation/",
                  "label": "7 no-code workflow automations for Amazon Web Services"
                }
              ],
              "primaryDocumentation": [
                {
                  "url": "https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/"
                }
              ]
            },
            "categories": [
              "Development",
              "Core Nodes"
            ],
            "nodeVersion": "1.0",
            "codexVersion": "1.0",
            "subcategories": {
              "Core Nodes": [
                "Helpers"
              ]
            }
          }
        },
        "group": "[\"output\"]",
        "defaults": {
          "name": "HTTP Request",
          "color": "#0004F5"
        },
        "iconData": {
          "type": "file",
          "fileBuffer": "data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDAiIGhlaWdodD0iNDAiIHZpZXdCb3g9IjAgMCA0MCA0MCIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPHBhdGggZmlsbC1ydWxlPSJldmVub2RkIiBjbGlwLXJ1bGU9ImV2ZW5vZGQiIGQ9Ik00MCAyMEM0MCA4Ljk1MzE0IDMxLjA0NjkgMCAyMCAwQzguOTUzMTQgMCAwIDguOTUzMTQgMCAyMEMwIDMxLjA0NjkgOC45NTMxNCA0MCAyMCA0MEMzMS4wNDY5IDQwIDQwIDMxLjA0NjkgNDAgMjBaTTIwIDM2Ljk0NThDMTguODg1MiAzNi45NDU4IDE3LjEzNzggMzUuOTY3IDE1LjQ5OTggMzIuNjk4NUMxNC43OTY0IDMxLjI5MTggMTQuMTk2MSAyOS41NDMxIDEzLjc1MjYgMjcuNjg0N0gyNi4xODk4QzI1LjgwNDUgMjkuNTQwMyAyNS4yMDQ0IDMxLjI5MDEgMjQuNTAwMiAzMi42OTg1QzIyLjg2MjIgMzUuOTY3IDIxLjExNDggMzYuOTQ1OCAyMCAzNi45NDU4Wk0xMi45MDY0IDIwQzEyLjkwNjQgMjEuNjA5NyAxMy4wMDg3IDIzLjE2NCAxMy4yMDAzIDI0LjYzMDVIMjYuNzk5N0MyNi45OTEzIDIzLjE2NCAyNy4wOTM2IDIxLjYwOTcgMjcuMDkzNiAyMEMyNy4wOTM2IDE4LjM5MDMgMjYuOTkxMyAxNi44MzYgMjYuNzk5NyAxNS4zNjk1SDEzLjIwMDNDMTMuMDA4NyAxNi44MzYgMTIuOTA2NCAxOC4zOTAzIDEyLjkwNjQgMjBaTTIwIDMuMDU0MTlDMjEuMTE0OSAzLjA1NDE5IDIyLjg2MjIgNC4wMzA3OCAyNC41MDAxIDcuMzAwMzlDMjUuMjA2NiA4LjcxNDA4IDI1LjgwNzIgMTAuNDA2NyAyNi4xOTIgMTIuMzE1M0gxMy43NTAxQzE0LjE5MzMgMTAuNDA0NyAxNC43OTQyIDguNzEyNTQgMTUuNDk5OCA3LjMwMDY0QzE3LjEzNzcgNC4wMzA4MyAxOC44ODUxIDMuMDU0MTkgMjAgMy4wNTQxOVpNMzAuMTQ3OCAyMEMzMC4xNDc4IDE4LjQwOTkgMzAuMDU0MyAxNi44NjE3IDI5LjgyMjcgMTUuMzY5NUgzNi4zMDQyQzM2LjcyNTIgMTYuODQyIDM2Ljk0NTggMTguMzk2NCAzNi45NDU4IDIwQzM2Ljk0NTggMjEuNjAzNiAzNi43MjUyIDIzLjE1OCAzNi4zMDQyIDI0LjYzMDVIMjkuODIyN0MzMC4wNTQzIDIzLjEzODMgMzAuMTQ3OCAyMS41OTAxIDMwLjE0NzggMjBaTTI2LjI3NjcgNC4yNTUxMkMyNy42MzY1IDYuMzYwMTkgMjguNzExIDkuMTMyIDI5LjM3NzQgMTIuMzE1M0gzNS4xMDQ2QzMzLjI1MTEgOC42NjggMzAuMTA3IDUuNzgzNDYgMjYuMjc2NyA0LjI1NTEyWk0xMC42MjI2IDEyLjMxNTNINC44OTI5M0M2Ljc1MTQ3IDguNjY3ODQgOS44OTM1MSA1Ljc4MzQxIDEzLjcyMzIgNC4yNTUxM0MxMi4zNjM1IDYuMzYwMjEgMTEuMjg5IDkuMTMyMDEgMTAuNjIyNiAxMi4zMTUzWk0zLjA1NDE5IDIwQzMuMDU0MTkgMjEuNjAzIDMuMjc3NDMgMjMuMTU3NSAzLjY5NDg0IDI0LjYzMDVIMTAuMTIxN0M5Ljk0NjE5IDIzLjE0MiA5Ljg1MjIyIDIxLjU5NDMgOS44NTIyMiAyMEM5Ljg1MjIyIDE4LjQwNTcgOS45NDYxOSAxNi44NTggMTAuMTIxNyAxNS4zNjk1SDMuNjk0ODRDMy4yNzc0MyAxNi44NDI1IDMuMDU0MTkgMTguMzk3IDMuMDU0MTkgMjBaTTI2LjI3NjYgMzUuNzQyN0MyNy42MzY1IDMzLjYzOTMgMjguNzExIDMwLjg2OCAyOS4zNzc0IDI3LjY4NDdIMzUuMTA0NkMzMy4yNTEgMzEuMzMyMiAzMC4xMDY4IDM0LjIxNzkgMjYuMjc2NiAzNS43NDI3Wk0xMy43MjM0IDM1Ljc0MjdDOS44OTM2OSAzNC4yMTc5IDYuNzUxNTUgMzEuMzMyNCA0Ljg5MjkzIDI3LjY4NDdIMTAuNjIyNkMxMS4yODkgMzAuODY4IDEyLjM2MzUgMzMuNjM5MyAxMy43MjM0IDM1Ljc0MjdaIiBmaWxsPSIjM0E0MkU5Ii8+Cjwvc3ZnPgo="
        },
        "displayName": "HTTP Request",
        "typeVersion": 4,
        "nodeCategories": [
          {
            "id": 5,
            "name": "Development"
          },
          {
            "id": 9,
            "name": "Core Nodes"
          }
        ]
      },
      {
        "id": 514,
        "icon": "fa:pause-circle",
        "name": "n8n-nodes-base.wait",
        "codex": {
          "data": {
            "alias": [
              "pause",
              "sleep",
              "delay",
              "timeout"
            ],
            "resources": {
              "generic": [
                {
                  "url": "https://n8n.io/blog/how-to-get-started-with-crm-automation-and-no-code-workflow-ideas/",
                  "icon": "\ud83d\udc65",
                  "label": "How to get started with CRM automation (with 3 no-code workflow ideas"
                },
                {
                  "url": "https://n8n.io/blog/aws-workflow-automation/",
                  "label": "7 no-code workflow automations for Amazon Web Services"
                }
              ],
              "primaryDocumentation": [
                {
                  "url": "https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.wait/"
                }
              ]
            },
            "categories": [
              "Core Nodes"
            ],
            "nodeVersion": "1.0",
            "codexVersion": "1.0",
            "subcategories": {
              "Core Nodes": [
                "Helpers",
                "Flow"
              ]
            }
          }
        },
        "group": "[\"organization\"]",
        "defaults": {
          "name": "Wait",
          "color": "#804050"
        },
        "iconData": {
          "icon": "pause-circle",
          "type": "icon"
        },
        "displayName": "Wait",
        "typeVersion": 1,
        "nodeCategories": [
          {
            "id": 9,
            "name": "Core Nodes"
          }
        ]
      },
      {
        "id": 565,
        "icon": "fa:sticky-note",
        "name": "n8n-nodes-base.stickyNote",
        "codex": {
          "data": {
            "alias": [
              "Comments",
              "Notes",
              "Sticky"
            ],
            "categories": [
              "Core Nodes"
            ],
            "nodeVersion": "1.0",
            "codexVersion": "1.0",
            "subcategories": {
              "Core Nodes": [
                "Helpers"
              ]
            }
          }
        },
        "group": "[\"input\"]",
        "defaults": {
          "name": "Sticky Note",
          "color": "#FFD233"
        },
        "iconData": {
          "icon": "sticky-note",
          "type": "icon"
        },
        "displayName": "Sticky Note",
        "typeVersion": 1,
        "nodeCategories": [
          {
            "id": 9,
            "name": "Core Nodes"
          }
        ]
      },
      {
        "id": 834,
        "icon": "file:code.svg",
        "name": "n8n-nodes-base.code",
        "codex": {
          "data": {
            "alias": [
              "cpde",
              "Javascript",
              "JS",
              "Python",
              "Script",
              "Custom Code",
              "Function"
            ],
            "details": "The Code node allows you to execute JavaScript in your workflow.",
            "resources": {
              "primaryDocumentation": [
                {
                  "url": "https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code/"
                }
              ]
            },
            "categories": [
              "Development",
              "Core Nodes"
            ],
            "nodeVersion": "1.0",
            "codexVersion": "1.0",
            "subcategories": {
              "Core Nodes": [
                "Helpers",
                "Data Transformation"
              ]
            }
          }
        },
        "group": "[\"transform\"]",
        "defaults": {
          "name": "Code"
        },
        "iconData": {
          "type": "file",
          "fileBuffer": "data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNTEyIiBoZWlnaHQ9IjUxMiIgdmlld0JveD0iMCAwIDUxMiA1MTIiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxnIGNsaXAtcGF0aD0idXJsKCNjbGlwMF8xMTcxXzQ0MSkiPgo8cGF0aCBkPSJNMTcwLjI4MyA0OEgxOTYuNUMyMDMuMTI3IDQ4IDIwOC41IDQyLjYyNzQgMjA4LjUgMzZWMTJDMjA4LjUgNS4zNzI1OCAyMDMuMTI3IDAgMTk2LjUgMEgxNzAuMjgzQzEyNi4xIDAgOTAuMjgzIDM1LjgxNzIgOTAuMjgzIDgwVjE3NkM5MC4yODMgMjA2LjkyOCA2NS4yMTA5IDIzMiAzNC4yODMgMjMySDIzQzE2LjM3MjYgMjMyIDExIDIzNy4zNzIgMTEgMjQ0VjI2OEMxMSAyNzQuNjI3IDE2LjM3MjQgMjgwIDIyLjk5OTYgMjgwTDM0LjI4MyAyODBDNjUuMjEwOSAyODAgOTAuMjgzIDMwNS4wNzIgOTAuMjgzIDMzNlY0NDBDOTAuMjgzIDQ3OS43NjQgMTIyLjUxOCA1MTIgMTYyLjI4MyA1MTJIMTk2LjVDMjAzLjEyNyA1MTIgMjA4LjUgNTA2LjYyNyAyMDguNSA1MDBWNDc2QzIwOC41IDQ2OS4zNzMgMjAzLjEyNyA0NjQgMTk2LjUgNDY0SDE2Mi4yODNDMTQ5LjAyOCA0NjQgMTM4LjI4MyA0NTMuMjU1IDEzOC4yODMgNDQwVjMzNkMxMzguMjgzIDMwOS4wMjIgMTI4LjAxMSAyODQuNDQzIDExMS4xNjQgMjY1Ljk2MUMxMDYuMTA5IDI2MC40MTYgMTA2LjEwOSAyNTEuNTg0IDExMS4xNjQgMjQ2LjAzOUMxMjguMDExIDIyNy41NTcgMTM4LjI4MyAyMDIuOTc4IDEzOC4yODMgMTc2VjgwQzEzOC4yODMgNjIuMzI2OSAxNTIuNjEgNDggMTcwLjI4MyA0OFoiIGZpbGw9IiNGRjk5MjIiLz4KPHBhdGggZD0iTTMwNSAzNkMzMDUgNDIuNjI3NCAzMTAuMzczIDQ4IDMxNyA0OEgzNDIuOTc5QzM2MC42NTIgNDggMzc0Ljk3OCA2Mi4zMjY5IDM3NC45NzggODBWMTc2QzM3NC45NzggMjAyLjk3OCAzODUuMjUxIDIyNy41NTcgNDAyLjA5OCAyNDYuMDM5QzQwNy4xNTMgMjUxLjU4NCA0MDcuMTUzIDI2MC40MTYgNDAyLjA5OCAyNjUuOTYxQzM4NS4yNTEgMjg0LjQ0MyAzNzQuOTc4IDMwOS4wMjIgMzc0Ljk3OCAzMzZWNDMyQzM3NC45NzggNDQ5LjY3MyAzNjAuNjUyIDQ2NCAzNDIuOTc5IDQ2NEgzMTdDMzEwLjM3MyA0NjQgMzA1IDQ2OS4zNzMgMzA1IDQ3NlY1MDBDMzA1IDUwNi42MjcgMzEwLjM3MyA1MTIgMzE3IDUxMkgzNDIuOTc5QzM4Ny4xNjEgNTEyIDQyMi45NzggNDc2LjE4MyA0MjIuOTc4IDQzMlYzMzZDNDIyLjk3OCAzMDUuMDcyIDQ0OC4wNTEgMjgwIDQ3OC45NzkgMjgwSDQ5MEM0OTYuNjI3IDI4MCA1MDIgMjc0LjYyOCA1MDIgMjY4VjI0NEM1MDIgMjM3LjM3MyA0OTYuNjI4IDIzMiA0OTAgMjMyTDQ3OC45NzkgMjMyQzQ0OC4wNTEgMjMyIDQyMi45NzggMjA2LjkyOCA0MjIuOTc4IDE3NlY4MEM0MjIuOTc4IDM1LjgxNzIgMzg3LjE2MSAwIDM0Mi45NzkgMEgzMTdDMzEwLjM3MyAwIDMwNSA1LjM3MjU4IDMwNSAxMlYzNloiIGZpbGw9IiNGRjk5MjIiLz4KPC9nPgo8ZGVmcz4KPGNsaXBQYXRoIGlkPSJjbGlwMF8xMTcxXzQ0MSI+CjxyZWN0IHdpZHRoPSI1MTIiIGhlaWdodD0iNTEyIiBmaWxsPSJ3aGl0ZSIvPgo8L2NsaXBQYXRoPgo8L2RlZnM+Cjwvc3ZnPgo="
        },
        "displayName": "Code",
        "typeVersion": 2,
        "nodeCategories": [
          {
            "id": 5,
            "name": "Development"
          },
          {
            "id": 9,
            "name": "Core Nodes"
          }
        ]
      }
    ]
  },
  "detail": {
    "description": "Quick overview This workflow triggers on a completed Apify Website Content Crawler run, syncs the crawl dataset into a Qdrant collection using OpenAI embeddings, then compares vector counts and parses the Apify run logs to report what chunks were added, refreshed, or deleted. How it works 1. Triggers when an Apify Website Content Crawler actor run finishes. 2. Calls the Qdrant Collections API to read the current point count in the target collection as a pre-sync baseline. 3. Starts an Apify Qdrant Integration actor run to upsert the crawl dataset into Qdrant, generating embeddings with OpenAI and applying delta updates plus deletion of expired chunks. 4. Waits 30 seconds to allow the Qdrant sync to finish settling. 5. Calls the Qdrant Collections API again to read the post-sync point count. 6. Fetches the Apify actor run logs and outputs a freshness report with vectors-before/after and the counts of chunks added, refreshed, and deleted. Setup 1. Create and select an Apify API credential for the Apify Trigger, the sync run request, and the log request. 2. Set environment variables QDRANTAPIKEY and OPENAIAPIKEY, and allow env access in nodes (for example by setting N8NBLOCKENVACCESSINNODE=false if required by your n8n deployment). 3. Replace https://YOUR-CLUSTER.qdrant.io:6333 with your Qdrant endpoint in all Qdrant URLs and in the sync request body, and ensure the docs collection name matches your intended collection (or allow auto-create). 4. Ensure the Apify Website Content Crawler actor is available in your Apify account and run it once so the trigger can be configured and tested."
  }
}