Skip to content

Commit a7beeca

Browse files
authored
feat(plugin-import-export): adds new exportLimit, importLimit and per collection limit control (#15405)
This PR adds `exportLimit`, `importLimit` and per collection `limit` control and fixes a few reported bugs around preview behaviour and data exports to CSV and expands the test coverage to all fields except uploads as those will be expanded upon in another PR. Some of these bugs are reported directly, not on Github. ## New limits control Adds new exportLimit, importLimit and per collection limit control so you can force a limit per user if necessary for how many documents can be exported or imported in a single operation. This allows you to prevent certain users from causing excessive server/resource usage. The config can be added top level to affect all operations or you can override on a per collection and per operation level, defaults to 0 (unlimited). ```ts importExportPlugin({ exportLimit: ({ req }) => 500, // function with req importLimit: 100, // hardcoded number collections: [ { slug: 'posts', // per collection per operation overrides export: { limit: ({ req }) => 5, }, import: { limit: 5, }, }, ], }), ``` Bugs fixed: - Timezones being wrongly exported without being selected (but date is selected) - Timezones being duplicated when selected - `disableSave` and `format` having no effect on the collection's export configuration Chores: - Removed an internal utility in favour of a `payload/shared` one - Added new e2e tests for `format` - Int tests for `point`, `checkbox`, `select`, `radio`, `email`, `textarea` and `code`
1 parent 6158489 commit a7beeca

88 files changed

Lines changed: 5386 additions & 1522 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/plugins/import-export.mdx

Lines changed: 63 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -55,12 +55,14 @@ export default config
5555

5656
## Options
5757

58-
| Property | Type | Description |
59-
| -------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------- |
60-
| `collections` | array | Collections to include Import/Export controls in. Array of collection configs with per-collection options. Defaults to all. |
61-
| `debug` | boolean | If true, enables debug logging. |
62-
| `overrideExportCollection` | function | Function to override the default export collection. Receives `{ collection }` and returns modified collection config. |
63-
| `overrideImportCollection` | function | Function to override the default import collection. Receives `{ collection }` and returns modified collection config. |
58+
| Property | Type | Description |
59+
| -------------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------- |
60+
| `collections` | array | Collections to include Import/Export controls in. Array of collection configs with per-collection options. Defaults to all. |
61+
| `debug` | boolean | If true, enables debug logging. |
62+
| `exportLimit` | number\|function | Global maximum documents for export operations. Set to `0` for unlimited (default). Per-collection limits take precedence. |
63+
| `importLimit` | number\|function | Global maximum documents for import operations. Set to `0` for unlimited (default). Per-collection limits take precedence. |
64+
| `overrideExportCollection` | function | Function to override the default export collection. Receives `{ collection }` and returns modified collection config. |
65+
| `overrideImportCollection` | function | Function to override the default import collection. Receives `{ collection }` and returns modified collection config. |
6466

6567
### Per-Collection Configuration
6668

@@ -74,23 +76,25 @@ Each item in the `collections` array can have the following properties:
7476

7577
### ExportConfig Options
7678

77-
| Property | Type | Description |
78-
| -------------------- | -------- | --------------------------------------------------------------- |
79-
| `batchSize` | number | Documents per batch during export. Default: `100`. |
80-
| `disableDownload` | boolean | Disable download button for this collection. |
81-
| `disableJobsQueue` | boolean | Run exports synchronously for this collection. |
82-
| `disableSave` | boolean | Disable save button for this collection. |
83-
| `format` | string | Force format (`csv` or `json`) for this collection. |
84-
| `overrideCollection` | function | Override the export collection config for this specific target. |
79+
| Property | Type | Description |
80+
| -------------------- | ---------------- | ------------------------------------------------------------------------------------------------ |
81+
| `batchSize` | number | Documents per batch during export. Default: `100`. |
82+
| `disableDownload` | boolean | Disable download button for this collection. |
83+
| `disableJobsQueue` | boolean | Run exports synchronously for this collection. |
84+
| `disableSave` | boolean | Disable save button for this collection. |
85+
| `format` | string | Force format (`csv` or `json`) for this collection. |
86+
| `limit` | number\|function | Maximum documents to export. Set to `0` for unlimited (default). Overrides global `exportLimit`. |
87+
| `overrideCollection` | function | Override the export collection config for this specific target. |
8588

8689
### ImportConfig Options
8790

88-
| Property | Type | Description |
89-
| ---------------------- | -------- | --------------------------------------------------------------- |
90-
| `batchSize` | number | Documents per batch during import. Default: `100`. |
91-
| `defaultVersionStatus` | string | Default status for imported docs (`draft` or `published`). |
92-
| `disableJobsQueue` | boolean | Run imports synchronously for this collection. |
93-
| `overrideCollection` | function | Override the import collection config for this specific target. |
91+
| Property | Type | Description |
92+
| ---------------------- | ---------------- | ------------------------------------------------------------------------------------------------ |
93+
| `batchSize` | number | Documents per batch during import. Default: `100`. |
94+
| `defaultVersionStatus` | string | Default status for imported docs (`draft` or `published`). |
95+
| `disableJobsQueue` | boolean | Run imports synchronously for this collection. |
96+
| `limit` | number\|function | Maximum documents to import. Set to `0` for unlimited (default). Overrides global `importLimit`. |
97+
| `overrideCollection` | function | Override the import collection config for this specific target. |
9498

9599
### Example Configuration
96100

@@ -102,6 +106,10 @@ export default buildConfig({
102106
importExportPlugin({
103107
debug: true,
104108

109+
// Global limits (0 = unlimited, which is the default)
110+
exportLimit: 10000,
111+
importLimit: 5000,
112+
105113
// Override default export collection (e.g., add access control)
106114
// This will be used by all collections unless they further override the config
107115
overrideExportCollection: ({ collection }) => {
@@ -119,9 +127,11 @@ export default buildConfig({
119127
export: {
120128
format: 'csv',
121129
disableDownload: true,
130+
limit: 1000, // Override global exportLimit for this collection
122131
},
123132
import: {
124133
defaultVersionStatus: 'draft',
134+
limit: 500, // Override global importLimit for this collection
125135
},
126136
},
127137
{
@@ -134,6 +144,39 @@ export default buildConfig({
134144
})
135145
```
136146

147+
## Limiting Import and Export Size
148+
149+
You can limit the number of documents that can be imported or exported in a single operation. This helps prevent DDOS-style abuse and protects server resources during large data operations.
150+
151+
Limits can be set globally via `exportLimit` and `importLimit`, or per-collection using the `limit` option in export/import config. Per-collection limits take precedence over global limits. Set to `0` for unlimited (the default).
152+
153+
### Dynamic Limits
154+
155+
Limits can be a function that receives the request context, allowing dynamic limits based on user roles or other factors:
156+
157+
```ts
158+
importExportPlugin({
159+
// Dynamic global limit based on user role
160+
exportLimit: ({ req }) => {
161+
return req.user?.role === 'admin' ? 50000 : 1000
162+
},
163+
164+
collections: [
165+
{
166+
slug: 'sensitive-data',
167+
import: {
168+
// Per-collection dynamic limit
169+
limit: ({ req }) => {
170+
if (req.user?.subscription === 'enterprise') return 100000
171+
if (req.user?.subscription === 'pro') return 10000
172+
return 1000
173+
},
174+
},
175+
},
176+
],
177+
})
178+
```
179+
137180
## Collection-Specific Import and Export targets
138181

139182
By default, the plugin creates a single `exports` collection and a single `imports` collection that handle all import/export operations across your enabled collections. However, you can create separate import and export targets for specific collections by overriding the collection slug.

packages/plugin-import-export/src/components/ExportPreview/index.scss

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,23 @@
77
margin-bottom: 10px;
88
}
99

10+
&__export-info {
11+
display: flex;
12+
flex-direction: column;
13+
align-items: flex-end;
14+
gap: 2px;
15+
}
16+
1017
&__export-count {
1118
font-size: var(--font-size-small);
1219
color: var(--theme-elevation-500);
1320
}
1421

22+
&__limit-capped {
23+
font-size: var(--font-size-small);
24+
color: var(--theme-warning-500);
25+
}
26+
1527
&__pagination {
1628
display: flex;
1729
align-items: center;

packages/plugin-import-export/src/components/ExportPreview/index.tsx

Lines changed: 31 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ export const ExportPreview: React.FC = () => {
5050
})
5151
const [dataToRender, setDataToRender] = useState<any[]>([])
5252
const [exportTotalDocs, setExportTotalDocs] = useState<number>(0)
53+
const [maxLimit, setMaxLimit] = useState<number | undefined>(undefined)
5354
const [columns, setColumns] = useState<Column[]>([])
5455
const { i18n, t } = useTranslation<
5556
PluginImportExportTranslations,
@@ -128,6 +129,7 @@ export const ExportPreview: React.FC = () => {
128129
hasNextPage,
129130
hasPrevPage,
130131
limit: responseLimit,
132+
maxLimit: serverMaxLimit,
131133
page: responsePage,
132134
totalPages,
133135
}: ExportPreviewResponse = await res.json()
@@ -166,6 +168,7 @@ export const ExportPreview: React.FC = () => {
166168
}))
167169

168170
setExportTotalDocs(serverExportTotalDocs)
171+
setMaxLimit(serverMaxLimit)
169172
setPaginationData({
170173
hasNextPage,
171174
hasPrevPage,
@@ -224,17 +227,34 @@ export const ExportPreview: React.FC = () => {
224227
<Translation i18nKey="version:preview" t={t} />
225228
</h3>
226229
{exportTotalDocs > 0 && !isPending && (
227-
<span className={`${baseClass}__export-count`}>
228-
<Translation
229-
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
230-
// @ts-expect-error
231-
i18nKey="plugin-import-export:documentsToExport"
232-
t={t}
233-
variables={{
234-
count: exportTotalDocs,
235-
}}
236-
/>
237-
</span>
230+
<div className={`${baseClass}__export-info`}>
231+
<span className={`${baseClass}__export-count`}>
232+
<Translation
233+
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
234+
// @ts-expect-error
235+
i18nKey="plugin-import-export:documentsToExport"
236+
t={t}
237+
variables={{
238+
count: exportTotalDocs,
239+
}}
240+
/>
241+
</span>
242+
{typeof maxLimit === 'number' &&
243+
maxLimit > 0 &&
244+
typeof limit === 'number' &&
245+
limit > maxLimit && (
246+
<span className={`${baseClass}__limit-capped`}>
247+
<Translation
248+
// @ts-expect-error - plugin translations not typed
249+
i18nKey="plugin-import-export:limitCapped"
250+
t={t}
251+
variables={{
252+
limit: maxLimit,
253+
}}
254+
/>
255+
</span>
256+
)}
257+
</div>
238258
)}
239259
</div>
240260
{isPending && !dataToRender && (

packages/plugin-import-export/src/components/ImportPreview/index.tsx

Lines changed: 3 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ import {
1616
useTranslation,
1717
} from '@payloadcms/ui'
1818
import { formatDocTitle } from '@payloadcms/ui/shared'
19-
import { fieldAffectsData } from 'payload/shared'
19+
import { fieldAffectsData, getObjectDotNotation } from 'payload/shared'
2020
import React, { useState, useTransition } from 'react'
2121

2222
import type {
@@ -211,7 +211,7 @@ export const ImportPreview: React.FC = () => {
211211

212212
// Skip if this field doesn't exist in any document
213213
const hasData = docs.some((doc) => {
214-
const value = getValueAtPath(doc, fieldPath)
214+
const value = getObjectDotNotation(doc, fieldPath)
215215
return value !== undefined && value !== null
216216
})
217217

@@ -225,7 +225,7 @@ export const ImportPreview: React.FC = () => {
225225
field,
226226
Heading: label,
227227
renderedCells: docs.map((doc) => {
228-
const value = getValueAtPath(doc, fieldPath)
228+
const value = getObjectDotNotation(doc, fieldPath)
229229

230230
if (value === undefined || value === null) {
231231
return null
@@ -626,18 +626,3 @@ export const ImportPreview: React.FC = () => {
626626
</div>
627627
)
628628
}
629-
630-
// Helper function to get nested values
631-
const getValueAtPath = (obj: Record<string, unknown>, path: string): unknown => {
632-
const segments = path.split('.')
633-
let current: any = obj
634-
635-
for (const segment of segments) {
636-
if (current === null || current === undefined) {
637-
return undefined
638-
}
639-
current = current[segment]
640-
}
641-
642-
return current
643-
}

0 commit comments

Comments
 (0)