Ingest Permissions for RAG and AI

Permissions are a necessary consideration for any use case that involves file/data ingestion. For use cases like RAG, your AI application will need to have an authentication and authorization strategy for data ingested from third-party providers like Google Drive, Dropbox, Salesforce, etc, so users can only get answers with data they are permitted to.

Permissions generally require 3 types of workflows:

1. An initial ingestion of all permissions for each file ingested

2. Webhook triggered jobs that listen for permission changes

3. Scheduled permission syncs when webhooks are not available

Initial Ingestion

Generally for file storage platforms like Google Drive or Dropbox, permissions are split up between readers, writers, and owners. You can imagine that the permissions data schema for a file generally look something like this:

{
	readers: [
		user1@example.com,
		user2@example.com
	],
	writers: [
		user3@example.com,
		user4@example.com
	],
	owner: [
		bigcheese@example.com
	]
}

With Paragon, you can ingest the permissions for every file a user has in their integration provider, like Dropbox for example.

Notice that you can define nuanced workflows where you can define different branching logic depending on if an object is a folder or file, or define javascript functions that map the third-party permissions schema to a schema your application expects.

Webhook Triggered for Permissions Changes

It’s common for permissions to change - give a new user read access, change a reader to a writer, revoke access. These changes can trigger Paragon workflows via third-party webhooks, allowing your application to ingest those permissions in real-time.

For example, Google Drive has a webhook for file changes, sending a webhook event whenever permissions change on a file.

Permissions Sync

The last category of workflow we have is scheduled permissions sync. This strategy involves getting permissions for a file storage system every day, every few hours, etc. This type of workflow can be useful for two types of situations:

1. No webhooks are available for permissions changes

2. Scheduled permissions checks in case webhook events are missed (maximize safety)

For example, Salesforce does not have a webhook for when users are given access to assets like the Contacts table. In this case, we may want to get permissions every day.

Wrapping Up

If permissions are available in a third-party API, Paragon can get the permissions data to your application. Paragon has helped AI companies ingest and handle permissions in their application and are happy to help you build out a robust permissions system for your application as well. Book a demo and reach out to our team.