diff --git a/.github/workflows/dotnet-test.yml b/.github/workflows/dotnet-test.yml index 5566d1c..a581433 100644 --- a/.github/workflows/dotnet-test.yml +++ b/.github/workflows/dotnet-test.yml @@ -33,7 +33,7 @@ jobs: run: dotnet --version - name: Build - run: dotnet build PhenX.EntityFrameworkCore.BulkInsert.Net10.slnx --framework ${{ matrix.dotnet.tfm }} + run: dotnet build PhenX.EntityFrameworkCore.BulkInsert.slnx --framework ${{ matrix.dotnet.tfm }} - name: Test - run: dotnet test PhenX.EntityFrameworkCore.BulkInsert.Net10.slnx --no-build --verbosity normal --framework ${{ matrix.dotnet.tfm }} + run: dotnet test PhenX.EntityFrameworkCore.BulkInsert.slnx --no-build --verbosity normal --framework ${{ matrix.dotnet.tfm }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 0afc6e0..6a14f4c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -47,10 +47,10 @@ jobs: run: dotnet test PhenX.EntityFrameworkCore.BulkInsert.slnx --configuration Release --verbosity normal --framework net9.0 - name: Test net10.0 - run: dotnet test PhenX.EntityFrameworkCore.BulkInsert.Net10.slnx --configuration Release --verbosity normal --framework net10.0 + run: dotnet test PhenX.EntityFrameworkCore.BulkInsert.slnx --configuration Release --verbosity normal --framework net10.0 - name: Pack nuget packages - run: dotnet pack PhenX.EntityFrameworkCore.BulkInsert.Net10.slnx --configuration Release --no-build --output nupkgs /p:PackageVersion=$VERSION + run: dotnet pack PhenX.EntityFrameworkCore.BulkInsert.slnx --configuration Release --no-build --output nupkgs /p:PackageVersion=$VERSION - name: Upload nuget package if: github.ref_type == 'tag' && startsWith(github.ref, 'refs/tags/v') diff --git a/PhenX.EntityFrameworkCore.BulkInsert.Net10.slnx b/PhenX.EntityFrameworkCore.BulkInsert.Net10.slnx deleted file mode 100644 index b4f043f..0000000 --- a/PhenX.EntityFrameworkCore.BulkInsert.Net10.slnx +++ /dev/null @@ -1,21 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - diff --git a/README.md b/README.md index efefd14..7df4e90 100644 --- a/README.md +++ b/README.md @@ -1,196 +1,204 @@ -# PhenX.EntityFrameworkCore.BulkInsert - -A high-performance, provider-agnostic bulk insert extension for Entity Framework Core 8+. Supports SQL Server, PostgreSQL, SQLite, MySQL and Oracle. - -Its main purpose is to provide a fast way to perform simple bulk inserts in Entity Framework Core applications. - -## Why this library? - -- **Performance**: It is designed to be fast and memory efficient, making it suitable for high-performance applications. -- **Provider-agnostic**: It works with multiple database providers (SQL Server, PostgreSQL, SQLite and MySQL), allowing you to use it in different environments without changing your code. -- **Simplicity**: The API is simple and easy to use, making it accessible for developers of all skill levels. - -For now, it does not support navigation properties, complex types, owned types, shadow properties, or inheritance, -but they are in [the roadmap](#roadmap). - -## Packages - -| Package Name | Description | NuGet Link | -|---------------------------------------------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `PhenX.EntityFrameworkCore.BulkInsert.SqlServer` | For SQL Server | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.SqlServer.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.SqlServer) | -| `PhenX.EntityFrameworkCore.BulkInsert.PostgreSql` | For PostgreSQL | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.PostgreSql.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.PostgreSql) | -| `PhenX.EntityFrameworkCore.BulkInsert.Sqlite` | For SQLite | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.Sqlite.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.Sqlite) | -| `PhenX.EntityFrameworkCore.BulkInsert.MySql` | For MySql | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.Sqlite.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.MySql) | -| `PhenX.EntityFrameworkCore.BulkInsert.Oracle` | For Oracle | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.Oracle.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.Oracle) | -| `PhenX.EntityFrameworkCore.BulkInsert` | Common library | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert) | - -## Installation - -Install the NuGet package for your database provider: - -```shell -# For SQL Server -Install-Package PhenX.EntityFrameworkCore.BulkInsert.SqlServer - -# For PostgreSQL -Install-Package PhenX.EntityFrameworkCore.BulkInsert.PostgreSql - -# For SQLite -Install-Package PhenX.EntityFrameworkCore.BulkInsert.Sqlite - -# For MySql -Install-Package PhenX.EntityFrameworkCore.BulkInsert.MySql - -# For Oracle -Install-Package PhenX.EntityFrameworkCore.BulkInsert.Oracle -``` - -## Usage - -Register the bulk insert provider in your `DbContextOptions`: - -```csharp -services.AddDbContext(options => -{ - options - // .UseSqlServer(connectionString) // or UseNpgsql or UseSqlite, as appropriate - - .UseBulkInsertPostgreSql() - // OR - .UseBulkInsertSqlServer() - // OR - .UseBulkInsertSqlite() - // OR - .UseBulkInsertMySql() - // OR - .UseBulkInsertOracle() - ; -}); -``` - -### Very basic usage - -```csharp -// Asynchronously -await dbContext.ExecuteBulkInsertAsync(entities); - -// Or synchronously -dbContext.ExecuteBulkInsert(entities); -``` - -### Bulk insert with options - -```csharp -// Common options -await dbContext.ExecuteBulkInsertAsync(entities, options => -{ - options.BatchSize = 1000; // Set the batch size for the insert operation, the default value is different for each provider -}); - -// Provider specific options, when available, example for SQL Server -await dbContext.ExecuteBulkInsertAsync(entities, (SqlServerBulkInsertOptions o) => // <<< here specify the SQL Server options class -{ - options.EnableStreaming = true; // Enable streaming for SQL Server -}); - -// Provider specific options, supporting multiple providers -await dbContext.ExecuteBulkInsertAsync(entities, o => -{ - o.MoveRows = true; - - if (o is SqlServerBulkInsertOptions sqlServerOptions) - { - sqlServerOptions.EnableStreaming = true; - } - else if (o is MySqlBulkInsertOptions mysqlOptions) - { - mysqlOptions.BatchSize = 1000; - } -}); -``` - -### Returning inserted entities - -```csharp -await dbContext.ExecuteBulkInsertReturnEntitiesAsync(entities); -``` - -### Conflict resolution / merge / upsert - -Conflict resolution works by specifying columns that should be used to detect conflicts and the action to take when -a conflict is detected (e.g., update existing rows), using the `onConflict` parameter. - - * The conflicting columns are specified with the `Match` property and must have a unique constraint in the database. - * The action to take when a conflict is detected is specified with the `Update` property. If not specified, the default action is to do nothing (i.e., skip the conflicting rows). - * You can also specify the condition for the update action using either the `Where` or the `RawWhere` property. If not specified, the update action will be applied to all conflicting rows. - -```csharp -await dbContext.ExecuteBulkInsertAsync(entities, onConflict: new OnConflictOptions -{ - Match = e => new - { - e.Name, - // ...other columns to match on - }, - - // Optional: specify the update action, if not specified, the default action is to do nothing - // Excluded is the row being inserted which is in conflict, and Inserted is the row already in the database. - Update = (inserted, excluded) => new TestEntity - { - Price = inserted.Price // Update the Price column with the new value - }, - - // Optional: specify the condition for the update action - // Excluded is the row being inserted which is in conflict, and Inserted is the row already in the database. - // Using raw SQL condition - RawWhere = (insertedTable, excludedTable) => $"{excludedTable}.some_price > {insertedTable}.some_price", - - // OR using a lambda expression - Where = (inserted, excluded) => excluded.Price > inserted.Price, -}); -``` - -## Roadmap - -- [ ] [Add support for navigation properties](https://github.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/issues/2) -- [x] [Add support for complex types](https://github.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/issues/3) -- [x] Add support for owned types -- [ ] Add support for shadow properties -- [ ] Add support for TPT (Table Per Type) inheritance -- [ ] Add support for TPC (Table Per Concrete Type) inheritance -- [ ] Add support for TPH (Table Per Hierarchy) inheritance - -## Benchmarks - -Benchmark projects are available in the [`tests/PhenX.EntityFrameworkCore.BulkInsert.Benchmark`](tests/PhenX.EntityFrameworkCore.BulkInsert.Benchmark/LibComparator.cs) directory. -Run them to compare performance with raw bulk insert methods and other libraries (https://github.com/borisdj/EFCore.BulkExtensions -and https://entityframework-extensions.net/bulk-extensions), using optimized configuration (local Docker is required). - -Legend : - * `PhenX_EntityFrameworkCore_BulkInsert`: this library - * `RawInsert`: naive implementation without any library, using the native provider API (SqlBulkCopy for SQL Server, BeginBinaryImport for PostgreSQL, raw inserts for SQLite) - * `Z_EntityFramework_Extensions_EFCore`: https://entityframework-extensions.net/bulk-extensions - * `EFCore_BulkExtensions`: https://github.com/borisdj/EFCore.BulkExtensions - * `Linq2Db`: https://github.com/linq2db/linq2db - -PostgreSQL results with 500 000 rows : - -![bench-postgresql.png](https://raw.githubusercontent.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/refs/heads/main/images/bench-postgresql.png) - -SQLite results with 500 000 rows : - -![bench-sqlite.png](https://raw.githubusercontent.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/refs/heads/main/images/bench-sqlite.png) - -MySQL results with 500 000 rows : - -![bench-mysql.png](https://raw.githubusercontent.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/refs/heads/main/images/bench-mysql.png) - -Where are the SQL Server and Oracle benchmarks? [You can run them yourself](https://www.google.com/search?q=dewitt+clause). - -## Contributing - -Contributions are welcome! Please open issues or submit pull requests for bug fixes, features, or documentation improvements. - -## License - -MIT License. See [LICENSE](LICENSE) for details. +# PhenX.EntityFrameworkCore.BulkInsert + +A high-performance, provider-agnostic bulk insert extension for Entity Framework Core 8+. Supports SQL Server, PostgreSQL, SQLite, MySQL and Oracle. + +Its main purpose is to provide a fast way to perform simple bulk inserts in Entity Framework Core applications. + +## Why this library? + +- **Performance**: It is designed to be fast and memory efficient, making it suitable for high-performance applications. +- **Provider-agnostic**: It works with multiple database providers (SQL Server, PostgreSQL, SQLite and MySQL), allowing you to use it in different environments without changing your code. +- **Simplicity**: The API is simple and easy to use, making it accessible for developers of all skill levels. + +For now, it does not support navigation properties, complex types, owned types, shadow properties, or inheritance, +but they are in [the roadmap](#roadmap). + +## Packages + +| Package Name | Description | NuGet Link | +|---------------------------------------------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `PhenX.EntityFrameworkCore.BulkInsert.SqlServer` | For SQL Server | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.SqlServer.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.SqlServer) | +| `PhenX.EntityFrameworkCore.BulkInsert.PostgreSql` | For PostgreSQL | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.PostgreSql.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.PostgreSql) | +| `PhenX.EntityFrameworkCore.BulkInsert.Sqlite` | For SQLite | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.Sqlite.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.Sqlite) | +| `PhenX.EntityFrameworkCore.BulkInsert.MySql` | For MySql | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.Sqlite.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.MySql) | +| `PhenX.EntityFrameworkCore.BulkInsert.Oracle` | For Oracle | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.Oracle.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert.Oracle) | +| `PhenX.EntityFrameworkCore.BulkInsert` | Common library | [![NuGet](https://img.shields.io/nuget/v/PhenX.EntityFrameworkCore.BulkInsert.svg)](https://www.nuget.org/packages/PhenX.EntityFrameworkCore.BulkInsert) | + +### Dependencies + +This library depends on the official Entity Framework Core packages provided by each database vendor. + +MySQL is a special case: the [Pomelo MySQL](https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql) package is widely preferred for its superior MySQL feature support. However, the package relies heavily on a single maintainer and does not yet support .NET 10. See the relevant [issue](https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql). + +A community fork is available at https://github.com/microting/Pomelo.EntityFrameworkCore.MySql that addresses this gap. **For .NET 10 only**, this library takes a dependency on that fork. + +## Installation + +Install the NuGet package for your database provider: + +```shell +# For SQL Server +Install-Package PhenX.EntityFrameworkCore.BulkInsert.SqlServer + +# For PostgreSQL +Install-Package PhenX.EntityFrameworkCore.BulkInsert.PostgreSql + +# For SQLite +Install-Package PhenX.EntityFrameworkCore.BulkInsert.Sqlite + +# For MySql +Install-Package PhenX.EntityFrameworkCore.BulkInsert.MySql + +# For Oracle +Install-Package PhenX.EntityFrameworkCore.BulkInsert.Oracle +``` + +## Usage + +Register the bulk insert provider in your `DbContextOptions`: + +```csharp +services.AddDbContext(options => +{ + options + // .UseSqlServer(connectionString) // or UseNpgsql or UseSqlite, as appropriate + + .UseBulkInsertPostgreSql() + // OR + .UseBulkInsertSqlServer() + // OR + .UseBulkInsertSqlite() + // OR + .UseBulkInsertMySql() + // OR + .UseBulkInsertOracle() + ; +}); +``` + +### Very basic usage + +```csharp +// Asynchronously +await dbContext.ExecuteBulkInsertAsync(entities); + +// Or synchronously +dbContext.ExecuteBulkInsert(entities); +``` + +### Bulk insert with options + +```csharp +// Common options +await dbContext.ExecuteBulkInsertAsync(entities, options => +{ + options.BatchSize = 1000; // Set the batch size for the insert operation, the default value is different for each provider +}); + +// Provider specific options, when available, example for SQL Server +await dbContext.ExecuteBulkInsertAsync(entities, (SqlServerBulkInsertOptions o) => // <<< here specify the SQL Server options class +{ + options.EnableStreaming = true; // Enable streaming for SQL Server +}); + +// Provider specific options, supporting multiple providers +await dbContext.ExecuteBulkInsertAsync(entities, o => +{ + o.MoveRows = true; + + if (o is SqlServerBulkInsertOptions sqlServerOptions) + { + sqlServerOptions.EnableStreaming = true; + } + else if (o is MySqlBulkInsertOptions mysqlOptions) + { + mysqlOptions.BatchSize = 1000; + } +}); +``` + +### Returning inserted entities + +```csharp +await dbContext.ExecuteBulkInsertReturnEntitiesAsync(entities); +``` + +### Conflict resolution / merge / upsert + +Conflict resolution works by specifying columns that should be used to detect conflicts and the action to take when +a conflict is detected (e.g., update existing rows), using the `onConflict` parameter. + + * The conflicting columns are specified with the `Match` property and must have a unique constraint in the database. + * The action to take when a conflict is detected is specified with the `Update` property. If not specified, the default action is to do nothing (i.e., skip the conflicting rows). + * You can also specify the condition for the update action using either the `Where` or the `RawWhere` property. If not specified, the update action will be applied to all conflicting rows. + +```csharp +await dbContext.ExecuteBulkInsertAsync(entities, onConflict: new OnConflictOptions +{ + Match = e => new + { + e.Name, + // ...other columns to match on + }, + + // Optional: specify the update action, if not specified, the default action is to do nothing + // Excluded is the row being inserted which is in conflict, and Inserted is the row already in the database. + Update = (inserted, excluded) => new TestEntity + { + Price = inserted.Price // Update the Price column with the new value + }, + + // Optional: specify the condition for the update action + // Excluded is the row being inserted which is in conflict, and Inserted is the row already in the database. + // Using raw SQL condition + RawWhere = (insertedTable, excludedTable) => $"{excludedTable}.some_price > {insertedTable}.some_price", + + // OR using a lambda expression + Where = (inserted, excluded) => excluded.Price > inserted.Price, +}); +``` + +## Roadmap + +- [ ] [Add support for navigation properties](https://github.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/issues/2) +- [x] [Add support for complex types](https://github.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/issues/3) +- [x] Add support for owned types +- [ ] Add support for shadow properties +- [ ] Add support for TPT (Table Per Type) inheritance +- [ ] Add support for TPC (Table Per Concrete Type) inheritance +- [ ] Add support for TPH (Table Per Hierarchy) inheritance + +## Benchmarks + +Benchmark projects are available in the [`tests/PhenX.EntityFrameworkCore.BulkInsert.Benchmark`](tests/PhenX.EntityFrameworkCore.BulkInsert.Benchmark/LibComparator.cs) directory. +Run them to compare performance with raw bulk insert methods and other libraries (https://github.com/borisdj/EFCore.BulkExtensions +and https://entityframework-extensions.net/bulk-extensions), using optimized configuration (local Docker is required). + +Legend : + * `PhenX_EntityFrameworkCore_BulkInsert`: this library + * `RawInsert`: naive implementation without any library, using the native provider API (SqlBulkCopy for SQL Server, BeginBinaryImport for PostgreSQL, raw inserts for SQLite) + * `Z_EntityFramework_Extensions_EFCore`: https://entityframework-extensions.net/bulk-extensions + * `EFCore_BulkExtensions`: https://github.com/borisdj/EFCore.BulkExtensions + * `Linq2Db`: https://github.com/linq2db/linq2db + +PostgreSQL results with 500 000 rows : + +![bench-postgresql.png](https://raw.githubusercontent.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/refs/heads/main/images/bench-postgresql.png) + +SQLite results with 500 000 rows : + +![bench-sqlite.png](https://raw.githubusercontent.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/refs/heads/main/images/bench-sqlite.png) + +MySQL results with 500 000 rows : + +![bench-mysql.png](https://raw.githubusercontent.com/PhenX/PhenX.EntityFrameworkCore.BulkInsert/refs/heads/main/images/bench-mysql.png) + +Where are the SQL Server and Oracle benchmarks? [You can run them yourself](https://www.google.com/search?q=dewitt+clause). + +## Contributing + +Contributions are welcome! Please open issues or submit pull requests for bug fixes, features, or documentation improvements. + +## License + +MIT License. See [LICENSE](LICENSE) for details. diff --git a/src/PhenX.EntityFrameworkCore.BulkInsert.MySql/PhenX.EntityFrameworkCore.BulkInsert.MySql.csproj b/src/PhenX.EntityFrameworkCore.BulkInsert.MySql/PhenX.EntityFrameworkCore.BulkInsert.MySql.csproj index bd38897..b2b6287 100644 --- a/src/PhenX.EntityFrameworkCore.BulkInsert.MySql/PhenX.EntityFrameworkCore.BulkInsert.MySql.csproj +++ b/src/PhenX.EntityFrameworkCore.BulkInsert.MySql/PhenX.EntityFrameworkCore.BulkInsert.MySql.csproj @@ -1,9 +1,5 @@  - - net8.0;net9.0 - - @@ -12,7 +8,7 @@ - +