SFTP Flow

​

SFTP Integration

​

Core Concepts

​

1. Platform Overview

• Create dedicated SFTP spaces for each client or data source
• Define schemas that validate incoming data files
• Build automated pipelines that transform files into standardized formats
• Send processed data to destination systems via webhooks or API
• Monitor file processing and alert on errors

​

2. Clients

• Dedicated SFTP space: A secure environment where the client can upload files
• SFTP credentials: Username and password for connecting to the SFTP server
• SFTP host: The server address displayed on the client’s page for connection
• Client ID: A unique identifier used in API calls and data routing
• Associated pipelines: Workflows that process files uploaded by this client

// Example client object
{
  "clientId": "c4b3f495-5dfc-4a91-b604-a8e66ab4a220",
  "clientName": "acme_logistics",
  "displayName": "Acme Logistics Inc.",
  "status": "active",
  "created": "2023-05-15T10:30:00Z",
  "lastActive": "2023-06-01T08:45:22Z",
  "sftpUsername": "acme_sftp"
}

​

3. Schemas

• Fields: The columns or properties expected in the data
• Data types: The expected type for each field (string, number, date, etc.)
• Validation rules: Requirements for each field (required/optional, format, range, etc.)

// Example schema definition
{
  "schemaId": "ord-schema-v1",
  "name": "Order Schema v1",
  "fileType": "csv",
  "fields": [
    {
      "name": "order_id",
      "type": "string",
      "required": true,
      "validation": {
        "pattern": "^ORD-[0-9]{6}$"
      }
    },
    {
      "name": "customer_email",
      "type": "string",
      "required": true,
      "validation": {
        "format": "email"
      }
    },
    {
      "name": "order_date",
      "type": "date",
      "required": true,
      "sourceFormat": "MM/DD/YYYY",
      "targetFormat": "YYYY-MM-DD"
    },
    {
      "name": "total_amount",
      "type": "number",
      "required": true,
      "validation": {
        "min": 0
      }
    }
  ]
}

​

4. Pipelines

• Schema: The file structure we’re mapping to (defines expected data format)
• Webhook: Optional notification when new files are uploaded
• Starter file: Template that defines the expected header structure for Excel/CSV files
• Mappings: Column mappings (both automatic and manual) from source to target schema
• Transformations: Data manipulations that can be applied to specific columns

// Example pipeline configuration
{
  "options": {
    "delimiter": ",",
    "skipHeaderRow": true
  },
  "fieldMappings": [
    {
      "source": "customer_id",
      "target": "id"
    },
    {
      "source": "customer_name",
      "target": "name"
    },
    {
      "source": "customer_email",
      "target": "email",
      "transform": "toLowerCase"
    },
    {
      "source": "customer_phone",
      "target": "phone",
      "transform": "formatPhoneNumber"
    }
  ],
  "transformations": {
    "toLowerCase": "function(value) { return value.toLowerCase(); }",
    "formatPhoneNumber": "function(value) { return value.replace(/[^0-9]/g, ''); }"
  }
}

​

5. Typical Workflow

1. Client Setup: You create a new client in FileFeed, which generates SFTP credentials and a dedicated SFTP space.
2. Schema Definition: You define a schema that specifies the target data structure you want to receive after processing.
3. Pipeline Creation: You create a pipeline and associate it with the schema. This automatically creates a dedicated folder in the client’s SFTP space.
4. Starter File Setup: You upload a template file with the expected header structure for the files your client will upload.
5. Field Mapping: You define mappings between source columns in the uploaded files and target fields in your schema.
6. Transformation Setup: You create JavaScript transformation functions to apply to specific fields.
7. Webhook Configuration: You set up an optional webhook URL to receive notifications when files are processed.
8. File Upload: The client uploads a file to their dedicated pipeline folder in the SFTP space.
9. Automatic Processing: FileFeed detects the new file, applies the defined mappings and transformations.
10. Notification: If a webhook is configured, a notification is sent to the specified URL.

​

Webhook Listener

​

1. Understanding Webhooks

• GENERAL: Sent for all file processing events, including successful processing and error situations

​

2. Webhook Request Structure

{
  "event": "GENERAL",
  "timestamp": "2025-05-19T21:30:00.000Z",
  "data": {
    "fileId": "550e8400-e29b-41d4-a716-446655440000",
    "filename": "example_data.csv",
    "clientName": "acme_inc",
    "status": "completed",
    "processedFilename": "example_data_processed.csv",
    "jsonFilename": "example_data.json",
    "size": 24680,
    "processedAt": "2025-05-19T21:29:55.000Z"
  }
}

​

3. Webhook Security

• Requests contain an x-sftpsync-signature header
• The signature is an HMAC-SHA256 hash of the request body using your webhook secret
• The secret can be found in the webhook configuration section of your dashboard (Dashboard -> Webhooks -> Configuration -> View Configuration button -> Get secret)

​

4. Example (Node.js)

const express = require('express');
const crypto = require('crypto');
const app = express();

app.use(express.json());

const WEBHOOK_SECRET = 'your_webhook_secret';

function verifySignature(requestPayload, signature) {
  const hmac = crypto.createHmac('sha256', WEBHOOK_SECRET);
  const digest = hmac.update(JSON.stringify(requestPayload)).digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(digest),
    Buffer.from(signature)
  );
}

app.post('/webhooks/sftpsync', (req, res) => {
const signature = req.headers['x-sftpsync-signature'];
  const requestBody = req.body;

  if (!signature || !verifySignature(requestBody, signature)) {
    return res.status(401).send('Invalid signature');
  }

  const event = requestBody.event;

  switch (event) {
    case 'GENERAL': {
      const { filename, status } = requestBody.data;
      console.log(`File ${filename} event received with status: ${status}`);
      
      if (status === 'completed') {
        console.log(`File ${filename} processed successfully`);
      } else if (status === 'failed') {
        const errorMessage = requestBody.data.errorMessage || 'Unknown error';
        console.log(`File ${filename} processing failed: ${errorMessage}`);
      }
      break;
    }
    default:
      console.log(`Unknown event type: ${event}`);
  }

  res.status(200).send('Webhook received');
});

app.listen(3000, () => {
  console.log('Webhook listener running on port 3000');
});

​

5. Best Practices

• Respond quickly (within 5 seconds) to avoid webhook timeouts
• Implement idempotency to handle potential duplicate webhook deliveries
• Use a queue system for processing webhook data asynchronously
• Store your webhook secret securely
• Implement proper error handling

​

Retrieve processed data

import FileFeed from '@filefeed/sdk';

const filefeed = new FileFeed({ apiKey: process.env.FILEFEED_API_KEY! });

// Get recent completed runs, then paginate data for a run
const runs = await filefeed.pipelineRuns.list({ status: 'completed', limit: 25 });
for (const run of runs.data) {
  let offset: number | null = 0;
  do {
    const page = await filefeed.pipelineRuns.getData({ pipelineRunId: run.id, limit: 1000, offset });
    // process page.data
    offset = page.data.length === 1000 ? (offset ?? 0) + page.data.length : null;
  } while (offset !== null);
  // Acknowledge once persisted
  await filefeed.pipelineRuns.ack({ pipelineRunId: run.id });
}

• In the app: Dashboard → Pipeline Runs
• Via webhook: included in webhook payloads

​

Integration Checklist

​

Step-by-step

• Get API key (Dashboard → My Account → Security Settings)
• Create Client (SFTP credentials)
• Define Schema (fields, validation)
• Create Webhook (Dashboard → Webhooks)
• Create and activate Pipeline (link client + schema; mappings/transforms)
• Register Webhook (store secret; verify HMAC signature)
• Upload a sample file and confirm run “completed”
• Retrieve processed data and persist (SDK or REST)
• Acknowledge the pipeline run (OPTIONAL)
• Monitor runs and webhook deliveries; set alerts