Ingest All Files from File Storage

If you’re building an AI product, it’s become clear that business context is needed for your AI product to provide relevant and highly personalized responses. Your users’ business context lives in internal files and documents from across file storage providers like Google Drive, Dropbox, Sharepoint, OneDrive, and Box.

Paragon’s Managed Sync product enables you to easily ingest data from every file storage app including Google Drive, Sharepoint, Dropbox, and Box all with 3 APIs.

With Managed Sync,

• the Sync API provides users the ability to sync file data in a normalized schema from any file storage integration with their AI application backend

• the Permissions API allows users to check the native 3rd-party permissions policies of a file; for use cases like RAG, this API makes it easy to check the original permissions to ensure they are respected in your AI product, especially important for enterprise customers with admins that are syncing their companies files on their employees’ behalf

Here’s how it works:

1. Implement the file storage Connect Portal to allow your end-users to authenticate to Google Drive, Sharepoint, Box, etc.

2. Use the Sync API to ingest data from every file storage integrations

3. (Optional step) Use the Permissions API to enforce the native file storage permissions before returning data to users

End-User Authentication

The first step is enabling your users to authenticate to the third-party file storage application. The Connect Portal can be embedded natively (and completely white-listed) in your application to allow users to login with their credentials to their file storage integration provider of choice.

This is a an example of Paragon's connect portal embedded natively in a demo application

Ingesting File Data with the Sync API

On your frontend, your users will be able to authenticate with their file storage applications using Paragon’s embedded UI. On the backend, the Sync API makes it easy to spin up data pipelines that for your application to pull your users’ file data.

Here’s how the Sync API works with your application (YourApp.ai will represent your application for demonstration purposes):

Notice how your application has no need to interact with the Google Drive API (or any of the file storage integrations) - learning what APIs you need, what rate limits they have, navigating file trees, learning different webhook authentication types, API specific errors, the list goes on. Managed Sync abstracts all of the API discovery and custom logic needed per file storage integration.

Sync pipelines are incredibly efficient data pipelines that allow users to pull up-to-date data on demand or on a schedule. Spinning up a sync is as easy as calling the Sync API with the file storage name of choice, putting the power in the hands of your developers to spin up Sync pipelines that pull file data on a schedule (every day, every few hours, or even every few minutes) or on-demand as new users authenticate, specify folders and workspaces to sync, and new file storage integrations are added.

//Google Drive Sync Pipeline
curl --request POST https://managed-sync.useparagon.com/sync \\
  --header 'Authorization: Bearer <token>' \\
  --data '{
  "integration": "googledrive",
  "pipeline": "files",
  "configuration": {}
}'
//Sharepoint Sync Pipeline
curl --request POST https://managed-sync.useparagon.com/sync \\
  --header 'Authorization: Bearer <token>' \\
  --data '{
  "integration": "sharepoint",
  "pipeline": "files",
  "configuration": {}
}'
//Box Sync Pipeline
curl --request POST https://managed-sync.useparagon.com/sync \\
  --header 'Authorization: Bearer <token>' \\
  --data '{
  "integration": "box",
  "pipeline": "files",
  "configuration": {}
}'

Once a Sync is triggered for a file storage integration, webhooks will be sent to your application backend whenever a scheduled Sync is completed or when new files or updates are detected in the scheduled syncs. After receiving a webhook, you can then pull the file contents down for storage, transformation, indexing to a vector database, etc. with this API.

curl --request GET \\
  --url https://managed-sync.useparagon.com/sync/{sync_id}/records/{record_id}/content \\
  --header 'Authorization: Bearer <token>'

The data content responses from the Sync API are normalized across file storage integrations, meaning you can use the same file data pulling logic no matter what file storage you’re pulling from.

That’s how easy it is to pull your users’ file contents data for any file storage integration at scale and stay up-to-date on that data.

Native File Storage Permissions (if needed)

If your application is pulling down your users’ file data for a use case like RAG, you’ll most likely need to enforce the native Google Drive, Sharepoint, Box, etc. permissions from these files. This can become even more complex as you work with enterprise customers where an admin or IT member will enable a sync on behalf of their entire organization.

Rather than researching the 3rd-party permissions model, ingesting those permissions, and normalizing those permissions into your own access control list or fine-grain authorization graph database, all just to authorize the retrieval of file data in your application, Managed Sync’s Permissions API does this for you. All your application needs to do is call the Permissions API with the user and object IDs; the Permissions API will return whether the user is authorized.

curl --request POST \\
  --url https://managed-sync.useparagon.com/permissions/check \\
  --header 'Authorization: Bearer <token>' \\
  --data '{
  "object": {},
  "user": {},
  "role": "<string>"
}'

For more details on permissions, check out our Permissions Use Case page.

Wrapping Up

With Paragon’s Managed Sync, you can easily build data pipelines to ingest your customers' external knowledge from their third-party file storage systems. Paragon’s platform is invisible to your users as they stay on your application with no redirects, and makes it easier for your developer team to scale the number of integrations implemented without worrying about third-party API details.

If you’re interested in ingesting files from integration providers at scale and would like to get access to Managed Sync, book a call with our team.