diff --git a/documentation/en/user/source/configuration/storage.rst b/documentation/en/user/source/configuration/storage.rst index 1143e38480..f47ab11c2d 100644 --- a/documentation/en/user/source/configuration/storage.rst +++ b/documentation/en/user/source/configuration/storage.rst @@ -17,7 +17,7 @@ Starting with version 1.8.0, GeoWebCache supports multiple persistent storage me * S3 blob store: stores tiles in an `Amazon Simple Storage Service `_ bucket, as individual "objects" following a `TMS `_-like key structure. * Google Cloud Storage blob store: stores tiles in a GCS bucket using the same TMS-like structure as S3. -* Azure blob store, MBTiles blob store, Swift blob store: additional storage backends described below. +* Azure blob store, Swift blob store: additional storage backends described below. Zero or more blobstores can be configured in the configuration file to store tiles at different locations and on different storage back-ends. One of the configured blobstores will be the **default** one. Meaning that it will be used to store the tiles of every layer whose configuration @@ -450,189 +450,6 @@ if using GWC inside GeoServer it will be the internal layer identifier, e.g., something like ``LayerInfoImpl--5f036b28:16bbda57c0e:-7ffc`` which can be retrieved by checking the GeoServer configuration files for the layer in question. -MBTiles Blob Store -++++++++++++++++++ - -This blob store allow us to store tiles using the `MBTiles `_ specification (version 1.1) which defines a schema for storing tiles in an `SQLite `_ database with some restrictions regarding tiles formats and projections. - -MBTiles specification only supports JPEG and PNG formats and projection EPSG:3857 is assumed. The implemented blob store will read and write MBTiles files compliant with the specification but will also be able to write and read MBTiles files that use others formats and projections. - -Using the MBTiles blob store will bring several benefits at the cost of some performance loss. The MBTiles storage uses a significantly smaller number of files, which results in easier data handling (e.g., backups, moving tiles between environments). In some cases the stored data will be more compact reducing the size of the data on disk. - -When compared to the file blob store this store has two limitations: - -* This store does not integrate with disk quota, this is a consequence of using database files. -* **This store cannot be shared among several GeoWebCache instances.** - -.. note:: If disk quota is activated the stored stats will not make much sense and will not reflect the actual disk usage, the size of the database files cannot be really controlled. - -Database files cannot be managed as simple files. When connections to a database are open the associated file should not be deleted, moved or switched or the database file may become corrupted. Databases files can also become fragmented after deleting an huge amount of data or after frequent inserts, updates or delete operations. - -File Path Templates -```````````````````` - -An MBTiles file will correspond to an SQLite database file. In order to limit the amount of contention on each single database file users will be allowed to decide the granularity of the databases files. When GeoWebCache needs to map a tile to a database file it will only look at the databases files paths, it will not take in account the MBTiles metadata (this is why this store is able to handle others formats and projections). - -To configure the databases files granularity the user needs to provide a file path template. The default file path template for the MBTiles blob store is this one: - -.. code-block:: none - - {layer}/{grid}{format}{params}/{z}-{x}-{y}.sqlite - -This file template will stores all the tiles belonging to a certain layer in a single folder that will contain sub folders for each given format, projection and set of parameters and will group tiles with the same zoom level, column range and row range in a SQLite file. The column and row range values are passed by configuration, by default those values are equal to 250. The provided files paths templates will always be considered relative to the root directory provided as a configuration option. - -Follows an example of what the blob store root directory structure may look like when using the default path template: - -.. code-block:: none - - . - |-- nurc_Pk50095 - | `-- EPSG_4326image_pngnull - | |-- 11_2000_1500.sqlite - | `-- 12_4250_3000.sqlite - `-- topp_states - |-- EPSG_900913image_jpeg7510004a12f49fdd49a2ba366e9c4594be7e4358 - | |-- 6_250_500.sqlite - | `-- 7_0_0.sqlite - `-- EPSG_900913image_jpegnull - |-- 3_500_0.sqlite - |-- 4_0_250.sqlite - `-- 8_750_500.sqlite - -If no parameters were provided *null* string will be used. Is the responsibility of the user to define a file path template that will avoid collisions. - -The terms that can be used in the file path template are: - -* **grid**: the grid set id -* **layer**: the name of the layer -* **format**: the image format of the tiles -* **params**: parameters unique hash -* **x**: column range, computed based on the column range count configuration property -* **y**: row range, computed based on the row range count configuration property -* **z**: the zoom level - -It is also possible to use parameters values, like *style* for example. If the parameter is not present *null* will be used. - -.. note:: Characters ``\`` and ``/`` can be used as path separator, they will be translated to the operating system specific one (``\`` for Linux and ``/`` for Windows). Any special char like ``\``, ``/``, ``:`` or empty space used in a term value will be substituted with an underscore. - -MBTiles Metadata -````````````````` - -A valid MBTiles file will need some metadata, the image format and layer name will be automatically added when an MBTiles file is created. The user can provide the remaining metadata using a properties file whose name must follow this pattern: - -.. code-block:: none - - .metadata - -As an example, to add metadata ``description`` and ``attribution`` entries to layer ``tiger_roads`` a file named ``tiger_roads.properties`` with the following content should be present in the metadata directory: - -.. code-block:: none - - description=ny_roads - attribution=geoserver - -The directory that contains this metadata files is defined by a configuration property. - -Vector Tile Compression -``````````````````````` - -Some non-standard MBTiles files contain vector tiles, and these are sometimes compressed using gzip. A ``gzipVector`` entry to the the store configuration with a value of ``true`` will enable this behaviour. Raster tiles will not be affected. - -Expiration Rules -````````````````` - -The MBTiles specification don't give information about when a tile was created. To allow expire rules, an auxiliary table is used to store tile creation time. In the presence of an MBTiles file generated by a third party tool it is assumed that the creation time of a tile was the first time it was accessed. This feature can be activated or deactivated by configuration. Note that this will not break the MBTiles specification compliance. - -Eager Truncate -``````````````` - -When performing a truncate of the cache the store will try to remove the whole database file avoiding to create fragmented space. This is not suitable for all the situations and is highly dependent on the database files granularity. The configuration property ``eagerDelete`` allows the user to disable or deactivate this feature which is disabled by default. - -When a truncate request by tile range is received all the the databases files that contains tiles that belong to the tile range are identified. If eager delete is set to true those databases files are deleted otherwise a single delete query for each file is performed. - -Configuration Example -`````````````````````` - -Follows as an example the default configuration of the MBTiles store: - -.. code-block:: xml - - - mbtiles-store - true - /tmp/gwc-mbtiles - {grid}/{layer}/{format}/{params}/{z}/tiles_{x}_{y}.sqlite - 250 - 250 - 1000 - 500 - false - true - 5 - /tmp/gwc-mbtiles/layersMetadata - - -The *rootDirectory* property defines the location where all the files produced by this store will be created. The *templatePath* property is used to control the granularity of the database files (see section above). Properties *rowRangeCount* and *columnRangeCount* will be used by the path template to compute tile ranges. - -The *poolSize* property allows to control the max number of open database files, when defining this property the user should take in account the number open files allowed by the operating system. The *poolReaperIntervalMs* property controls how often the pool size will be checked to see if some database files connections need to be closed. - -Property *eagerDelete* controls how the truncate operation is performed (see section above). The property *useCreateTime* can be used to activate or deactivate the insertion of the tile creation time (see section above). Property *executorConcurrency* controls the parallelism used to perform certain operations, like the truncate operation for example. Property *mbtilesMetadataDirectory* defines the directory where the store will look for user provided MBTiles metadata. - -.. note:: Since the connection pool eviction happens at a certain interval, it means that the number of files open concurrently can go above the threshold limit for a certain amount of time. - -Replace Operation -`````````````````` - -As said before, if the cache is running SQLite files cannot be simply switched, first all connections need to be closed. The replace operation was created for this propose. The replace operation will first copy the new file side by side the old one, then block the requests to the old file, close the connections tot he old file, delete the old one, rename the new file to current one, reopen the new db file and start serving requests again. Should be almost instant. - -A REST entry point for this operation is available, it will be possible to submit a ZIP file or a single file along with the request. The replace operation can also use an already present file or directory. When using a directory the directory structure will be used to find the destination of each file, all the paths will be assumed to be relative to the store root directory. This means that is possible to replace a store content with another store content (a seeded one for example) by zipping the second store root directory and send it as a replacement. - -.. note:: When using a local directory or submitting a zip file all the file present in the directory will be considered. - -There is four ways to invoke this operation. Follows an example of all those variants invocations using CURL. - -Replace a single file uploading the replacement file: - -.. code-block:: none - - curl -u geowebcache:secured -H 'Content-Type: multipart/form-data' - -F "file=@tiles_0_0.sqlite" - -F "destination=EPSG_4326/sf_restricted/image_png/null/10/tiles_0_0.sqlite" - -F "layer=sf:restricted" - -XPOST 'http://localhost:8080/geowebcache/rest/sqlite/replace' - -Replace a single file using a file already present on the system: - -.. code-block:: none - - curl -u geowebcache:secured -H 'Content-Type: multipart/form-data' - -F "source=/tmp/tiles_0_0.sqlite" - -F "destination=EPSG_4326/sf_restricted/image_png/null/10/tiles_0_0.sqlite" - -F "layer=sf:restricted" - -XPOST 'http://localhost:8080/geowebcache/rest/sqlite/replace' - -Replace multiple files uploading a ZIP file: - -.. code-block:: none - - curl -u geowebcache:secured -H 'Content-Type: multipart/form-data' - -F "file=@tiles.zip" - -F "layer=sf:restricted" - -XPOST 'http://localhost:8080/geowebcache/rest/sqlite/replace' - -Replace multiple files using a directory already present on the system: - -.. code-block:: none - - curl -u geowebcache:secured -H 'Content-Type: multipart/form-data' - -F "source=/tmp/tiles" - -F "layer=sf:restricted" - -XPOST 'http://localhost:8080/geowebcache/rest/sqlite/replace' - -The *layer* parameter identifies the layer whose associated blob store content should be replaced. The *file* parameter is used to upload a single file or a ZIP file. The *source* parameter is used to reference an already present file or directory. The *destination* parameter is used to define the file that should be replaced with the provided file. - -This are the only valid combinations of this parameters other combinations will ignore some of the provided parameters or will throw an exception. - OpenStack Swift (Swift) Blob Store +++++++++++++++++++++++++++++++++++++++++++++ diff --git a/documentation/en/user/source/installation/upgrading.rst b/documentation/en/user/source/installation/upgrading.rst index 339d2f33db..e67accaf99 100644 --- a/documentation/en/user/source/installation/upgrading.rst +++ b/documentation/en/user/source/installation/upgrading.rst @@ -33,6 +33,18 @@ Java 11 LTS GeoWebCache 1.15 GeoWebCache 1.22 GeoWebCache 1.27 OpenJDK Java 8 LTS GeoWebCache 1.9 GeoWebCache 1.9 GeoWebCache 1.22 Oracle and OpenJDK ============ ================= ================ ================ ================== +GeoWebCache 1.28.6 Update +------------------------- + +SQLite and MBTiles blob stores removed +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``gwc-sqlite`` module has been removed. It provided the SQLite and MBTiles blob stores, +configured with the ```` element. Caches stored this way are no longer +readable. If you used one of these stores, move the cache to another blob store (file, S3, +Azure, Google Cloud Storage or Swift) before upgrading. The MBTiles tile layer +(``gwc-mbtiles`` module), which serves tiles from an MBTiles file as a layer, is not affected. + GeoWebCache 1.18 Update ----------------------- diff --git a/geowebcache/core/src/main/resources/geowebcache.xml b/geowebcache/core/src/main/resources/geowebcache.xml index 8b464be086..4120ea38dc 100644 --- a/geowebcache/core/src/main/resources/geowebcache.xml +++ b/geowebcache/core/src/main/resources/geowebcache.xml @@ -83,23 +83,6 @@ true --> - - diff --git a/geowebcache/pom.xml b/geowebcache/pom.xml index 4862143611..0680fae51a 100644 --- a/geowebcache/pom.xml +++ b/geowebcache/pom.xml @@ -24,7 +24,6 @@ diskquota arcgiscache s3storage - sqlite azureblob swiftblob mbtiles diff --git a/geowebcache/sqlite/pom.xml b/geowebcache/sqlite/pom.xml deleted file mode 100644 index ce0a1febf9..0000000000 --- a/geowebcache/sqlite/pom.xml +++ /dev/null @@ -1,70 +0,0 @@ - - - 4.0.0 - - org.geowebcache - geowebcache - 1.28-SNAPSHOT - - - org.geowebcache - gwc-sqlite - jar - SQLite Storage - - - org.geowebcache - gwc-core - ${project.version} - - - commons-fileupload - commons-fileupload - - - org.geotools - gt-mbtiles - - - org.geotools - gt-mbtiles - tests - test - - - org.geowebcache - gwc-core - tests - test - - - commons-io - commons-io - - - org.springframework - spring-test - test - - - javax.servlet - javax.servlet-api - test - - - org.mockito - mockito-core - test - - - org.easymock - easymock - test - - - org.awaitility - awaitility - test - - - diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/ExtendedConnection.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/ExtendedConnection.java deleted file mode 100644 index 858bc4ed0f..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/ExtendedConnection.java +++ /dev/null @@ -1,323 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.sql.Array; -import java.sql.Blob; -import java.sql.CallableStatement; -import java.sql.Clob; -import java.sql.Connection; -import java.sql.DatabaseMetaData; -import java.sql.NClob; -import java.sql.PreparedStatement; -import java.sql.SQLClientInfoException; -import java.sql.SQLException; -import java.sql.SQLWarning; -import java.sql.SQLXML; -import java.sql.Savepoint; -import java.sql.Statement; -import java.sql.Struct; -import java.util.Map; -import java.util.Properties; -import java.util.concurrent.Executor; - -/** Extended connection that allow us to keep track if a connection was closed or not. */ -final class ExtendedConnection implements Connection { - - private final Connection connection; - - private boolean closed = false; - - public ExtendedConnection(Connection connection) { - this.connection = connection; - } - - boolean closeInvoked() { - return closed; - } - - @Override - public void close() throws SQLException { - closed = true; - } - - @Override - public Statement createStatement() throws SQLException { - return connection.createStatement(); - } - - @Override - public PreparedStatement prepareStatement(String sql) throws SQLException { - return connection.prepareStatement(sql); - } - - @Override - public CallableStatement prepareCall(String sql) throws SQLException { - return connection.prepareCall(sql); - } - - @Override - public String nativeSQL(String sql) throws SQLException { - return connection.nativeSQL(sql); - } - - @Override - public void setAutoCommit(boolean autoCommit) throws SQLException { - connection.setAutoCommit(autoCommit); - } - - @Override - public boolean getAutoCommit() throws SQLException { - return connection.getAutoCommit(); - } - - @Override - public void commit() throws SQLException { - connection.commit(); - } - - @Override - public void rollback() throws SQLException { - connection.rollback(); - } - - @Override - public boolean isClosed() throws SQLException { - return connection.isClosed(); - } - - @Override - public DatabaseMetaData getMetaData() throws SQLException { - return connection.getMetaData(); - } - - @Override - public void setReadOnly(boolean readOnly) throws SQLException { - connection.setReadOnly(readOnly); - } - - @Override - public boolean isReadOnly() throws SQLException { - return connection.isReadOnly(); - } - - @Override - public void setCatalog(String catalog) throws SQLException { - connection.setCatalog(catalog); - } - - @Override - public String getCatalog() throws SQLException { - return connection.getCatalog(); - } - - @Override - public void setTransactionIsolation(int level) throws SQLException { - connection.setTransactionIsolation(level); - } - - @Override - public int getTransactionIsolation() throws SQLException { - return connection.getTransactionIsolation(); - } - - @Override - public SQLWarning getWarnings() throws SQLException { - return connection.getWarnings(); - } - - @Override - public void clearWarnings() throws SQLException { - connection.clearWarnings(); - } - - @Override - public Statement createStatement(int resultSetType, int resultSetConcurrency) throws SQLException { - return connection.createStatement(resultSetType, resultSetConcurrency); - } - - @Override - public PreparedStatement prepareStatement(String sql, int resultSetType, int resultSetConcurrency) - throws SQLException { - return connection.prepareStatement(sql, resultSetType, resultSetConcurrency); - } - - @Override - public CallableStatement prepareCall(String sql, int resultSetType, int resultSetConcurrency) throws SQLException { - return connection.prepareCall(sql, resultSetType, resultSetConcurrency); - } - - @Override - public Map> getTypeMap() throws SQLException { - return connection.getTypeMap(); - } - - @Override - public void setTypeMap(Map> map) throws SQLException { - connection.setTypeMap(map); - } - - @Override - public void setHoldability(int holdability) throws SQLException { - connection.setHoldability(holdability); - } - - @Override - public int getHoldability() throws SQLException { - return connection.getHoldability(); - } - - @Override - public Savepoint setSavepoint() throws SQLException { - return connection.setSavepoint(); - } - - @Override - public Savepoint setSavepoint(String name) throws SQLException { - return connection.setSavepoint(name); - } - - @Override - public void rollback(Savepoint savepoint) throws SQLException { - connection.rollback(savepoint); - } - - @Override - public void releaseSavepoint(Savepoint savepoint) throws SQLException { - connection.releaseSavepoint(savepoint); - } - - @Override - public Statement createStatement(int resultSetType, int resultSetConcurrency, int resultSetHoldability) - throws SQLException { - return connection.createStatement(resultSetType, resultSetConcurrency, resultSetHoldability); - } - - @Override - public PreparedStatement prepareStatement( - String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability) throws SQLException { - return connection.prepareStatement(sql, resultSetType, resultSetConcurrency, resultSetHoldability); - } - - @Override - public CallableStatement prepareCall( - String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability) throws SQLException { - return connection.prepareCall(sql, resultSetType, resultSetConcurrency, resultSetHoldability); - } - - @Override - public PreparedStatement prepareStatement(String sql, int autoGeneratedKeys) throws SQLException { - return connection.prepareStatement(sql, autoGeneratedKeys); - } - - @Override - public PreparedStatement prepareStatement(String sql, int[] columnIndexes) throws SQLException { - return connection.prepareStatement(sql, columnIndexes); - } - - @Override - public PreparedStatement prepareStatement(String sql, String[] columnNames) throws SQLException { - return connection.prepareStatement(sql, columnNames); - } - - @Override - public Clob createClob() throws SQLException { - return connection.createClob(); - } - - @Override - public Blob createBlob() throws SQLException { - return connection.createBlob(); - } - - @Override - public NClob createNClob() throws SQLException { - return connection.createNClob(); - } - - @Override - public SQLXML createSQLXML() throws SQLException { - return connection.createSQLXML(); - } - - @Override - public boolean isValid(int timeout) throws SQLException { - return connection.isValid(timeout); - } - - @Override - public void setClientInfo(String name, String value) throws SQLClientInfoException { - connection.setClientInfo(name, value); - } - - @Override - public void setClientInfo(Properties properties) throws SQLClientInfoException { - connection.setClientInfo(properties); - } - - @Override - public String getClientInfo(String name) throws SQLException { - return connection.getClientInfo(name); - } - - @Override - public Properties getClientInfo() throws SQLException { - return connection.getClientInfo(); - } - - @Override - public Array createArrayOf(String typeName, Object[] elements) throws SQLException { - return connection.createArrayOf(typeName, elements); - } - - @Override - public Struct createStruct(String typeName, Object[] attributes) throws SQLException { - return connection.createStruct(typeName, attributes); - } - - @Override - public void setSchema(String schema) throws SQLException { - connection.setSchema(schema); - } - - @Override - public String getSchema() throws SQLException { - return connection.getSchema(); - } - - @Override - public void abort(Executor executor) throws SQLException { - connection.abort(executor); - } - - @Override - public void setNetworkTimeout(Executor executor, int milliseconds) throws SQLException { - connection.setNetworkTimeout(executor, milliseconds); - } - - @Override - public int getNetworkTimeout() throws SQLException { - return connection.getNetworkTimeout(); - } - - @Override - public T unwrap(Class iface) throws SQLException { - return connection.unwrap(iface); - } - - @Override - public boolean isWrapperFor(Class iface) throws SQLException { - return connection.isWrapperFor(iface); - } -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/FileManager.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/FileManager.java deleted file mode 100644 index d2d4d23c21..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/FileManager.java +++ /dev/null @@ -1,417 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import static org.geowebcache.sqlite.Utils.Tuple; - -import java.io.File; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Objects; -import java.util.Set; -import java.util.logging.Level; -import java.util.logging.Logger; -import java.util.regex.Matcher; -import java.util.regex.Pattern; -import org.geotools.util.logging.Logging; -import org.geowebcache.filter.parameters.ParametersUtils; -import org.geowebcache.storage.TileObject; -import org.geowebcache.storage.TileRange; - -/** - * Class responsible to map GWC concepts (layer, tile, tile range, etc ...) to a filesystem file. The mapping is defined - * by a template that can use the information associated with a tile. - * - *

- * - *

The template supported terms are: - * - *

    - *
  • params - *
  • x - *
  • y - *
  • z - *
  • layer - *
  • grid - *
  • format - *
- * - * It is also possible to use parameters referencing them by their name. - * - *

- * - *

For example a template like the following: - * - *

- * - * 
- * {grid}/{layer}/{format}/{params}/{z}/tiles_{x}_{y}.sqlite
- *
- * - *
- * - * will produce paths similar to this one: - * - *
- * - * 
- * EPSG_4326/img_states/image_png/10/tiles_350_625.sqlite
- *
- * - *
- * - *

- * - *

Is possible to map all tiles to a single file by defining a static template (no terms). Although, if a term is - * used it cannot be NULL otherwise an exception will be throw. If a referenced parameter doesn't exists the string - * 'null' will be used. - */ -final class FileManager { - - private static Logger LOGGER = Logging.getLogger(FileManager.class.getName()); - - private static final Pattern PATH_TEMPLATE_ATTRIBUTE_PATTERN = Pattern.compile("\\{(.+?)\\}"); - - private final File rootPath; - - private final long rowRangeCount; - private final long columnRangeCount; - - // path builder extracted from the path template - private final String[] pathBuilderOriginal; - - // keep track of which terms are used in the path template - // the boolean tell us if the term was used in the path template - // and the integer define the position of the term in the path builder - private final Tuple replaceParametersId; - private final Tuple replaceZoom; - private final Tuple replaceRow; - private final Tuple replaceColumn; - private final Tuple replaceLayerName; - private final Tuple replaceGridSetId; - private final Tuple replaceFormat; - - // parameters used in the path template - private final Set> replaceParameters; - - FileManager(File rootDirectory, String pathTemplate, long rowRangeCount, long columnRangeCount) { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info(String.format( - "Initiating file manager: [rootDirectory='%s', pathTemplate='%s', " - + "rowRangeCount='%d', columnRangeCount='%d'].", - rootDirectory, pathTemplate, rowRangeCount, columnRangeCount)); - } - this.rootPath = rootDirectory; - this.rowRangeCount = rowRangeCount; - this.columnRangeCount = columnRangeCount; - // parsing the path template and extracting the terms that are used - Tuple>> parserResult = - parsePathTemplate(rootDirectory.getPath(), pathTemplate); - pathBuilderOriginal = parserResult.first; - replaceParametersId = findAndRemove(parserResult.second, "params"); - replaceZoom = findAndRemove(parserResult.second, "z"); - replaceRow = findAndRemove(parserResult.second, "x"); - replaceColumn = findAndRemove(parserResult.second, "y"); - replaceLayerName = findAndRemove(parserResult.second, "layer"); - replaceGridSetId = findAndRemove(parserResult.second, "grid"); - replaceFormat = findAndRemove(parserResult.second, "format"); - replaceParameters = parserResult.second; - } - - /** Builds the complete file path associated to the provided tile. */ - File getFile(TileObject tile) { - if (tile.getParametersId() == null && tile.getParameters() != null) { - tile.setParametersId(ParametersUtils.getId(tile.getParameters())); - } - return getFile( - tile.getParametersId(), - tile.getXYZ(), - tile.getLayerName(), - tile.getGridSetId(), - tile.getBlobFormat(), - tile.getParameters()); - } - - /** Build a complete file path using the provided terms. */ - File getFile( - String parametersId, - long[] xyz, - String layerName, - String gridSetId, - String format, - Map parameters) { - // init this local thread path builder - String[] pathBuilderCopy = getPathBuilderCopy(); - // replace the terms used in the path template with the respective values - if (replaceParametersId.first) { - pathBuilderCopy[replaceParametersId.second] = - normalizeAttributeValue("params", handleParametersId(parametersId, parameters)); - } - if (replaceZoom.first) { - pathBuilderCopy[replaceZoom.second] = String.valueOf(getLongValue(xyz, 2)); - } - if (replaceRow.first) { - pathBuilderCopy[replaceRow.second] = String.valueOf(computeColumnRange(getLongValue(xyz, 0))); - } - if (replaceColumn.first) { - pathBuilderCopy[replaceColumn.second] = String.valueOf(computeRowRange(getLongValue(xyz, 1))); - } - if (replaceLayerName.first) { - pathBuilderCopy[replaceLayerName.second] = normalizeAttributeValue("layer", layerName); - } - if (replaceGridSetId.first) { - pathBuilderCopy[replaceGridSetId.second] = normalizeAttributeValue("grid", gridSetId); - } - if (replaceFormat.first) { - pathBuilderCopy[replaceFormat.second] = normalizeAttributeValue("format", format); - } - // replace the parameters used in the path template with the respective values - for (Tuple replaceParameter : replaceParameters) { - // searching for the parameter value in a non case sensitive way - String value = parameters.get(replaceParameter.first.toUpperCase()); - value = value == null ? parameters.get(replaceParameter.first.toLowerCase()) : value; - // if the parameter doesn't exits we use string 'null' as value - value = value == null ? "null" : normalizeAttributeValue(replaceParameter.first, value); - pathBuilderCopy[replaceParameter.second] = value; - } - return new File(concatStringArray(pathBuilderCopy, 0)); - } - - /** Return the files present in the root directory that correspond to a certain layer. */ - List getFiles(String layerName) { - // init the thread local path builder - String[] pathBuilderCopy = getPathBuilderCopy(); - // we only need to replace the layer term - if (replaceLayerName.first) pathBuilderCopy[replaceLayerName.second] = layerName; - return getFiles(pathBuilderCopy); - } - - /** Return the files present in the root directory that correspond to a certain layer and certain grid set. */ - List getFiles(String layerName, String gridSetId) { - // init the thread local path builder - String[] pathBuilderCopy = getPathBuilderCopy(); - // we replace the layer and grid set terms - if (replaceLayerName.first) pathBuilderCopy[replaceLayerName.second] = layerName; - if (replaceGridSetId.first) pathBuilderCopy[replaceGridSetId.second] = gridSetId; - return getFiles(pathBuilderCopy); - } - - /** Return the files present in the root directory that correspond to a certain layer and certain grid set. */ - List getParametersFiles(String layerName, String parametersId) { - // init the thread local path builder - String[] pathBuilderCopy = getPathBuilderCopy(); - // we replace the layer and grid set terms - if (replaceLayerName.first) pathBuilderCopy[replaceLayerName.second] = layerName; - if (replaceParametersId.first) pathBuilderCopy[replaceParametersId.second] = parametersId; - return getFiles(pathBuilderCopy); - } - - /** Build the paths correspondent to a tile range. For each file we return the associated tiles range by zoom. */ - Map> getFiles(TileRange tileRange) { - Map> files = new HashMap<>(); - // let's iterate of all the available zoom levels - for (int z = tileRange.getZoomStart(); z <= tileRange.getZoomStop(); z++) { - long[] range = tileRange.rangeBounds(z); - if (range == null) { - // this zoom level doesn't have any tiles associated - continue; - } - // get the files and associated tiles for the current zoom level - getFiles( - files, - tileRange.getParametersId(), - tileRange.getLayerName(), - tileRange.getGridSetId(), - tileRange.getMimeType().getFormat(), - tileRange.getParameters(), - z, - range); - } - return files; - } - - /** This method will substitute any char that cannot be used in a file path with an underscore. */ - public static String normalizePathValue(String value) { - return value.replaceAll("\\\\|/|:|(?:\\s+)", "_"); - } - - /** - * Helper method that for a specific zoom level and a range of tiles will build all the files paths need to contains - * those tiles. - */ - private void getFiles( - Map> files, - String parametersId, - String layerName, - String gridSetId, - String format, - Map parameters, - long z, - long[] range) { - long minRangeX = (range[0] / columnRangeCount) * columnRangeCount; - long maxRangeX = (range[2] / columnRangeCount) * rowRangeCount; - long minRangeY = (range[1] / rowRangeCount) * rowRangeCount; - long maxRangeY = (range[3] / rowRangeCount) * rowRangeCount; - for (long x = minRangeX; x <= maxRangeX; x += columnRangeCount) { - long minx = Math.max(x, range[0]); - long maxx = Math.min(x + columnRangeCount - 1, range[2]); - for (long y = minRangeY; y <= maxRangeY; y += rowRangeCount) { - long[] tile = {x, y, z}; - File file = getFile(parametersId, tile, layerName, gridSetId, format, parameters); - long miny = Math.max(y, range[1]); - long maxy = Math.min(y + rowRangeCount - 1, range[3]); - List ranges = files.get(file); - if (ranges == null) { - ranges = new ArrayList<>(); - files.put(file, ranges); - } - ranges.add(new long[] {minx, miny, maxx, maxy, z}); - } - } - } - - /** If the provided parameters id is null a new one will be build based on the provided parameters. */ - private static String handleParametersId(String parametersId, Map parameters) { - if (parametersId != null) { - // the provided parameters id is ok - return parametersId; - } - // computing a new parameters id based on the provided parameters - String computedParametersId = ParametersUtils.getId(parameters); - if (computedParametersId == null) { - // the provided parameter are null or empty let's use the string 'null' as parameter id - return "null"; - } - return computedParametersId; - } - - private static String normalizeAttributeValue(String attribute, String value) { - Utils.check(value != null, "Path template attribute '%s' value is NULL.", attribute); - return normalizePathValue(value); - } - - private static long getLongValue(long[] xyz, int index) { - Utils.check(xyz != null, "Path template attribute 'xyz' is NULL."); - Utils.check(xyz.length == 3, "Path template attribute 'xyz' doesn't have the correct length."); - return xyz[index]; - } - - /** Helper method that will find in the root directory the files that match the provided path builder. */ - private List getFiles(String[] pathBuilderCopy) { - // build the concrete path with the embedded regex values (.*?) - String pathRegex = concatStringArray(pathBuilderCopy, 1); - // separate all the path parts, useful to walk in the path hierarchy - String[] pathRegexParts = pathRegex.split(Utils.REGEX_FILE_SEPARATOR); - // walk the directory tree to find all the files that match the builder path - return walkFileTreeWithRegex(rootPath, 0, pathRegexParts); - } - - /** Helper method that will walk recursively the directory hierarchy based on the provided path parts. */ - private static List walkFileTreeWithRegex(File path, int level, String[] pathParts) { - // filter the current directory files that match the current path part - File[] files = path.listFiles((directory, name) -> { - String pathPart = pathParts[level]; - // if need the current path will be interpreted as a regex (.*?) - return pathPart.equals(name) || name.matches(pathPart); - }); - if (Objects.isNull(files)) { - return Collections.emptyList(); - } - if (level != pathParts.length - 1) { - // let's walk recursively in the matched files - List matchedFiles = new ArrayList<>(); - for (File file : files) { - matchedFiles.addAll(walkFileTreeWithRegex(file, level + 1, pathParts)); - } - return matchedFiles; - } - // we are in the last directory before the path end so we simply return the matched files - return Arrays.asList(files); - } - - private static String concatStringArray(String[] array, int startIndex) { - StringBuilder result = new StringBuilder(); - for (int i = startIndex; i < array.length; i++) { - result.append(array[i]); - } - return result.toString(); - } - - private String[] getPathBuilderCopy() { - String[] pathBuilderCopy = new String[pathBuilderOriginal.length]; - System.arraycopy(pathBuilderOriginal, 0, pathBuilderCopy, 0, pathBuilderOriginal.length); - return pathBuilderCopy; - } - - private static Tuple findAndRemove(Set> attributes, String attribute) { - Tuple found = null; - for (Tuple candidateAttribute : attributes) { - if (candidateAttribute.first.equals(attribute)) { - if (found != null) { - throw Utils.exception("Term '%s' appears multiple times in the path template.", attribute); - } - found = candidateAttribute; - } - } - if (found != null) { - attributes.remove(found); - return Tuple.tuple(true, found.second); - } - return Tuple.tuple(false, -1); - } - - /** - * Helper method that will parse a path template and return a path builder and the found terms and the used - * parameters. - */ - private static Tuple>> parsePathTemplate( - String rootPath, String pathTemplate) { - // replacing chars '\' and '/' with the current os path separator - pathTemplate = pathTemplate.replaceAll("(\\\\)|/", Utils.REGEX_FILE_SEPARATOR); - List pathBuilder = new ArrayList<>(); - // the first element of the path builder is the root directory - pathBuilder.add(rootPath + File.separator); - Set> attributes = new HashSet<>(); - // matching all the available terms in the path template - Matcher matcher = PATH_TEMPLATE_ATTRIBUTE_PATTERN.matcher(pathTemplate); - int lastMatchIndex = 0; - int pathBuilderIndex = 1; - while (matcher.find()) { - // keeping track of the found term and is position on the path builder - pathBuilder.add(pathTemplate.substring(lastMatchIndex, matcher.start())); - // adding the match all regex expression to the path builder (match all files at that - // level) - pathBuilder.add(".*?"); - String attribute = matcher.group(1).toLowerCase(); - attributes.add(Tuple.tuple(attribute, pathBuilderIndex + 1)); - pathBuilderIndex += 2; - lastMatchIndex = matcher.end(); - } - pathBuilder.add(pathTemplate.substring(lastMatchIndex, pathTemplate.length())); - return Tuple.tuple(pathBuilder.toArray(new String[pathBuilder.size()]), attributes); - } - - private long computeRowRange(long tileRow) { - return (tileRow / rowRangeCount) * rowRangeCount; - } - - private long computeColumnRange(long tileRow) { - return (tileRow / columnRangeCount) * columnRangeCount; - } -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/GeoToolsMbtilesUtils.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/GeoToolsMbtilesUtils.java deleted file mode 100644 index 4790018a23..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/GeoToolsMbtilesUtils.java +++ /dev/null @@ -1,47 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.io.File; -import java.sql.Connection; -import org.geotools.mbtiles.MBTilesFile; -import org.springframework.jdbc.datasource.SingleConnectionDataSource; - -/** Utils objects used to interact with GeoTools. */ -public final class GeoToolsMbtilesUtils { - - private GeoToolsMbtilesUtils() {} - - public static MBTilesFile getMBTilesFile(Connection connection, File file) { - try { - return new MbtilesFileExtended(connection); - } catch (Exception exception) { - throw new RuntimeException("Error creating MBTiles file for '%s'.".formatted(file), exception); - } - } - - /** Extended version of GeoTools Mbtiles that allow us to pass the connection to be used. */ - private static final class MbtilesFileExtended extends MBTilesFile { - - public MbtilesFileExtended(Connection connection) throws Exception { - super(new SingleConnectionDataSource(connection, false)); - } - - @Override - public void close() { - super.close(); - ((SingleConnectionDataSource) connPool).destroy(); - } - } -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/MbtilesBlobStore.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/MbtilesBlobStore.java deleted file mode 100644 index 6ffb547a79..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/MbtilesBlobStore.java +++ /dev/null @@ -1,739 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; -import java.io.File; -import java.io.FileInputStream; -import java.io.IOException; -import java.io.InputStream; -import java.nio.file.DirectoryStream; -import java.nio.file.Files; -import java.nio.file.Path; -import java.sql.Connection; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Objects; -import java.util.Optional; -import java.util.Properties; -import java.util.concurrent.CompletionService; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.ExecutorCompletionService; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.TimeUnit; -import java.util.logging.Level; -import java.util.logging.Logger; -import java.util.regex.Matcher; -import java.util.regex.Pattern; -import java.util.zip.GZIPInputStream; -import java.util.zip.GZIPOutputStream; -import org.apache.commons.io.IOUtils; -import org.geotools.jdbc.util.SqlUtil; -import org.geotools.mbtiles.MBTilesFile; -import org.geotools.mbtiles.MBTilesMetadata; -import org.geotools.mbtiles.MBTilesTile; -import org.geotools.util.logging.Logging; -import org.geowebcache.filter.parameters.ParametersUtils; -import org.geowebcache.mime.MimeException; -import org.geowebcache.mime.MimeType; -import org.geowebcache.storage.BlobStoreListener; -import org.geowebcache.storage.BlobStoreListenerList; -import org.geowebcache.storage.CompositeBlobStore; -import org.geowebcache.storage.StorageException; -import org.geowebcache.storage.TileObject; -import org.geowebcache.storage.TileRange; - -/** Blobstore that store the tiles in a sqlite database using the mbtiles specification. */ -public final class MbtilesBlobStore extends SqliteBlobStore { - - private static Logger LOGGER = Logging.getLogger(MbtilesBlobStore.class.getName()); - - // pattern for matching the name of a file that contains mbtiles metadata (layerName.properties) - private static final Pattern MBTILES_METADATA_FILE_NAME_PATTERN = Pattern.compile("(.*?)\\.properties"); - - // sqlite database that will contain layers metadata - private final File metadataFile; - - // if true we will prefer delete a file rather than delete only a few tiles (avoiding vacuum) - private final boolean eagerDelete; - - // if false we will not care about tile create time (expiration rules will not work) - private final boolean useCreateTime; - - private final BlobStoreListenerList listeners; - - // parsed mbtiles metadata indexed per layer - private final Map layersMetadata = new ConcurrentHashMap<>(); - - // Executor that can be used to perform parallel operations - private final ExecutorService executorService; - - // Apply GZIP compression to uncompressed vector tile formats. - private final boolean gzipVector; - - MbtilesBlobStore(MbtilesInfo configuration) throws StorageException { - // caution this constructor will create a new connection pool - this( - configuration, - new SqliteConnectionManager(configuration.getPoolSize(), configuration.getPoolReaperIntervalMs())); - } - - public MbtilesBlobStore(MbtilesInfo configuration, SqliteConnectionManager connectionManager) - throws StorageException { - super(configuration, connectionManager); - metadataFile = new File(configuration.getRootDirectoryFile(), "metadata.sqlite"); - - boolean exists = metadataFile.exists(); - boolean empty = true; - - if (configuration.getRootDirectoryFile().exists()) { - try (DirectoryStream ds = Files.newDirectoryStream( - configuration.getRootDirectoryFile().toPath())) { - for (@SuppressWarnings("unused") Path p : ds) { - LOGGER.severe("Found file: " + p); - empty = false; - break; - } - } catch (StorageException e) { - throw e; - } catch (IOException e) { - throw new StorageException( - "Error while checking that " + configuration.getRootDirectory() + " is empty", e); - } - } else { - throw new StorageException("Root directory file does not exist: " + configuration.getRootDirectoryFile()); - } - - CompositeBlobStore.checkSuitability(configuration.getRootDirectory(), exists, empty); - - eagerDelete = configuration.eagerDelete(); - useCreateTime = configuration.useCreateTime(); - executorService = Executors.newFixedThreadPool(configuration.getExecutorConcurrency()); - listeners = new BlobStoreListenerList(); - gzipVector = configuration.isGzipVector(); - - initMbtilesLayersMetadata(configuration.getMbtilesMetadataDirectory()); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("MBTiles blob store initiated: [eagerDelete='%b', useCreateTime='%b'." - .formatted(eagerDelete, useCreateTime)); - } - } - - private boolean tileIsGzipped(TileObject tile) throws MimeException { - return gzipVector && MimeType.createFromFormat(tile.getBlobFormat()).isVector(); - } - - @Override - public void put(TileObject tile) throws StorageException { - File file = fileManager.getFile(tile); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Tile '%s' mapped to file '%s'.".formatted(tile, file)); - } - initDatabaseFileIfNeeded(file, tile.getLayerName(), tile.getBlobFormat()); - // do work in write mode - connectionManager.doWork(file, false, connection -> { - // instantiating geotools needed objects - MBTilesFile mbtiles = GeoToolsMbtilesUtils.getMBTilesFile(connection, file); - MBTilesTile gtTile = new MBTilesTile(tile.getXYZ()[2], tile.getXYZ()[0], tile.getXYZ()[1]); - try { - final boolean gzipped = tileIsGzipped(tile); - - byte[] bytes; - if (gzipped) { - try (ByteArrayOutputStream byteStream = new ByteArrayOutputStream(); - GZIPOutputStream gzOut = new GZIPOutputStream(byteStream)) { - gzOut.write(Utils.resourceToByteArray(tile.getBlob())); - gzOut.flush(); - bytes = byteStream.toByteArray(); - } - } else { - bytes = Utils.resourceToByteArray(tile.getBlob()); - } - gtTile.setData(bytes); - - // if necessary getting old data size for listeners - byte[] olData = null; - if (!listeners.isEmpty()) { - olData = mbtiles.loadTile(tile.getXYZ()[2], tile.getXYZ()[0], tile.getXYZ()[1]) - .getData(); - } - // saving the tile - mbtiles.saveTile(gtTile); - if (useCreateTime) { - // we need to store this tile create time - putTileCreateTime( - connection, - tile.getXYZ()[2], - tile.getXYZ()[0], - tile.getXYZ()[1], - System.currentTimeMillis()); - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Tile '%s' saved in file '%s'.".formatted(tile, file)); - } - if (listeners.isEmpty()) { - // no listeners to update we are done - return; - } - if (olData == null) { - // this was new tile - listeners.sendTileStored(tile); - } else { - // this an update - listeners.sendTileUpdated(tile, olData.length); - } - } catch (Exception exception) { - throw Utils.exception(exception, "Error saving tile '%s' in file '%s'.", tile, file); - } - }); - - persistParameterMap(tile); - } - - @Override - public boolean get(final TileObject tile) throws StorageException { - File file = fileManager.getFile(tile); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Tile '%s' mapped to file '%s'.".formatted(tile, file)); - } - initDatabaseFileIfNeeded(file, tile.getLayerName(), tile.getBlobFormat()); - // do work in readonly mode - boolean exists = connectionManager.doWork(file, true, connection -> { - // instantiating geotools mbtiles reader - try (MBTilesFile mbtiles = GeoToolsMbtilesUtils.getMBTilesFile(connection, file)) { - - final boolean gzipped = tileIsGzipped(tile); - - // loading the tile using geotools reader - MBTilesTile gtTile = mbtiles.loadTile(tile.getXYZ()[2], tile.getXYZ()[0], tile.getXYZ()[1]); - - byte[] bytes = gtTile.getData(); - if (gtTile.getData() != null) { - if (gzipped) { - try (ByteArrayOutputStream byteOut = new ByteArrayOutputStream(); - ByteArrayInputStream byteIn = new ByteArrayInputStream(gtTile.getData()); - GZIPInputStream gzIn = new GZIPInputStream(byteIn)) { - IOUtils.copy(gzIn, byteOut); - bytes = byteOut.toByteArray(); - } - } else { - bytes = gtTile.getData(); - } - tile.setBlob(Utils.byteArrayToResource(bytes)); - - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Tile '%s' found on file '%s'.".formatted(tile, file)); - } - return true; - } - } catch (Exception exception) { - throw Utils.exception(exception, "Error loading tile '%s' from MBTiles file '%s'.", tile, file); - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Tile '%s' not found on file '%s'.".formatted(tile, file)); - } - return false; - }); - if (exists && useCreateTime) { - // the tile exists and we need to set the create time in the tile object - Long createdTime = getTileCreateTime(file, tile.getXYZ()[2], tile.getXYZ()[0], tile.getXYZ()[1]); - if (createdTime == null) { - // no create time associated with this tile let's assume the last modified time - createdTime = file.lastModified(); - // update the create time - putTileCreateTime(file, tile.getXYZ()[2], tile.getXYZ()[0], tile.getXYZ()[1], createdTime); - } - tile.setCreated(createdTime); - } else if (exists) { - // we don't care about the create time, tile will never expire - tile.setCreated(System.currentTimeMillis()); - } - return exists; - } - - @Override - public boolean delete(TileObject tile) throws StorageException { - File file = fileManager.getFile(tile); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Tile '%s' mapped to file '%s'.".formatted(tile, file)); - } - if (!file.exists()) { - // database file doesn't exists so nothing to do - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Containing file '%s' for tile '%s' doesn't exists.".formatted(file, tile)); - } - return false; - } - // do work on write mode - return connectionManager.doWork(file, false, connection -> { - // instantiating geotools objects without setting the tile data (this way - // geotools will try to remove the tile) - MBTilesFile mbtiles = GeoToolsMbtilesUtils.getMBTilesFile(connection, file); - MBTilesTile gtTile = new MBTilesTile(tile.getXYZ()[2], tile.getXYZ()[0], tile.getXYZ()[1]); - try { - // getting tile old data and checking if the tile exists - byte[] olData = mbtiles.loadTile(tile.getXYZ()[2], tile.getXYZ()[0], tile.getXYZ()[1]) - .getData(); - if (olData != null) { - // tile exists so let's remove the tile - tile.setBlobSize(olData.length); - mbtiles.saveTile(gtTile); - // updating the listener if any - listeners.sendTileDeleted(tile); - if (useCreateTime) { - // we care about the create time so let's remove it - deleteTileCreateTime(connection, tile.getXYZ()[2], tile.getXYZ()[0], tile.getXYZ()[1]); - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Tile '%s' deleted from file '%s'.".formatted(tile, file)); - } - return true; - } - } catch (Exception exception) { - throw Utils.exception(exception, "Error deleting tile '%s' from MBTiles file '%s'.", tile, file); - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Tile '%s' not found on file '%s'.".formatted(tile, file)); - } - return false; - }); - } - - @Override - public synchronized void putLayerMetadata(String layerName, String key, String value) { - // storing metadata associated with a layer in the metadata file - connectionManager.executeSql( - metadataFile, - "CREATE TABLE IF NOT EXISTS metadata (layerName text, key text, value text, PRIMARY KEY(layerName, key));"); - connectionManager.executeSql( - metadataFile, "INSERT OR REPLACE INTO metadata VALUES (?, ?, ?);", layerName, key, value); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Metadata for layer '%s' for key '%s' inserted or updated on file '%s'." - .formatted(layerName, key, metadataFile)); - } - } - - @Override - public String getLayerMetadata(String layerName, String key) { - try { - return connectionManager.executeQuery( - metadataFile, - resultSet -> { - try { - if (resultSet.next()) { - // metadata value is available - String value = resultSet.getString(1); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Metadata for layer '%s' with key '%s' found '%s'." - .formatted(layerName, key, value)); - } - return value; - } - // metadata value not found - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine( - "Metadata for layer '%s' with key '%s' not found.".formatted(layerName, key)); - } - return null; - } catch (Exception exception) { - throw Utils.exception(exception, "Error reading result set."); - } - }, - "SELECT value FROM metadata WHERE layerName = ? AND key = ?;", - layerName, - key); - } catch (Exception exception) { - // probably because the metadata table doesn't exists - if (LOGGER.isLoggable(Level.SEVERE)) { - LOGGER.log(Level.SEVERE, "Error getting metadata from file '%s'.".formatted(metadataFile), exception); - } - return null; - } - } - - @Override - public boolean layerExists(String layerName) { - // a layer exists if there is at least one file associated to it - return !fileManager.getFiles(layerName).isEmpty(); - } - - @Override - public boolean delete(String layerName) throws StorageException { - boolean deleted = deleteFiles(fileManager.getFiles(layerName)); - listeners.sendLayerDeleted(layerName); - return deleted; - } - - @Override - public boolean deleteByGridsetId(String layerName, String gridSetId) throws StorageException { - boolean deleted = deleteFiles(fileManager.getFiles(layerName, gridSetId)); - listeners.sendGridSubsetDeleted(layerName, gridSetId); - return deleted; - } - - @Override - public boolean deleteByParametersId(String layerName, String parametersId) throws StorageException { - boolean deleted = deleteFiles(fileManager.getParametersFiles(layerName, parametersId)); - listeners.sendParametersDeleted(layerName, parametersId); - return deleted; - } - - @Override - public boolean delete(TileRange tileRange) throws StorageException { - // getting the files associated with this tile range - Map> files = fileManager.getFiles(tileRange); - if (files.isEmpty()) { - // no files so nothing to do - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Nothing to do."); - } - return false; - } - // let's delete the tiles - CompletionService completionService = new ExecutorCompletionService<>(executorService); - int tasks = 0; - for (Map.Entry> entry : files.entrySet()) { - // FIXME: should we tell something to the listeners ? - File file = entry.getKey(); - if (!file.exists()) { - // this database file doesn't exists, so nothing to do - continue; - } - if (eagerDelete) { - // we delete the whole file avoiding fragmentation on the database - completionService.submit(() -> connectionManager.delete(file), true); - } else { - // we need to delete all tiles that belong to the tiles range and are stored in the - // current file - for (long[] range : entry.getValue()) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Deleting tiles range [minx=%d, miny=%d, maxx=%d, maxxy=%d, zoom=%d] in file '%s'." - .formatted(range[0], range[1], range[2], range[3], range[4], file)); - } - completionService.submit( - () -> connectionManager.executeSql( - file, - "DELETE FROM tiles WHERE zoom_level = ? AND tile_column BETWEEN ? AND ? AND tile_row BETWEEN ? AND ?;", - range[4], - range[0], - range[2], - range[1], - range[3]), - true); - } - } - tasks++; - } - // let's wait for the tasks to finish - for (int i = 0; i < tasks; i++) { - try { - completionService.take().get(); - } catch (Exception exception) { - throw Utils.exception(exception, "Something bad happen when deleting tile range."); - } - } - return true; - } - - @Override - public boolean rename(String oldLayerName, String newLayerName) throws StorageException { - List files = fileManager.getFiles(oldLayerName); - if (files.isEmpty()) { - return false; - } - for (File currentFile : files) { - String normalizedLayerName = FileManager.normalizePathValue(newLayerName); - File newFile = new File(currentFile.getPath().replace(oldLayerName, normalizedLayerName)); - connectionManager.rename(currentFile, newFile); - } - listeners.sendLayerRenamed(oldLayerName, newLayerName); - return true; - } - - @Override - public void addListener(BlobStoreListener listener) { - listeners.addListener(listener); - } - - @Override - public boolean removeListener(BlobStoreListener listener) { - return listeners.removeListener(listener); - } - - @Override - public void clear() throws StorageException { - connectionManager.reapAllConnections(); - } - - @Override - public void destroy() { - connectionManager.reapAllConnections(); - connectionManager.stopPoolReaper(); - executorService.shutdown(); - try { - executorService.awaitTermination(5, TimeUnit.SECONDS); - } catch (Exception exception) { - if (LOGGER.isLoggable(Level.SEVERE)) { - LOGGER.log(Level.SEVERE, "Error when waiting for executor task to finish.", exception); - } - } - } - - /** Helper method that delete the provided files. */ - private boolean deleteFiles(List files) throws StorageException { - if (files.isEmpty()) { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("No files to delete."); - } - return false; - } - // asking the connection manager to remove the database files - CompletionService completionService = new ExecutorCompletionService<>(executorService); - int tasks = 0; - for (File file : files) { - completionService.submit(() -> connectionManager.delete(file), true); - tasks++; - } - // let's wait for the tasks to finish - for (int i = 0; i < tasks; i++) { - try { - completionService.take().get(); - } catch (Exception exception) { - throw Utils.exception(exception, "Something bad happen when deleting files."); - } - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Files deleted."); - } - return true; - } - - /** Helper method that deletes the create time of a tile. */ - private void deleteTileCreateTime(Connection connection, long z, long x, long y) throws StorageException { - try { - connectionManager.executeSql( - connection, - "DELETE FROM tiles_metadata " + "WHERE zoom_level = ? AND tile_column = ? AND tile_row = ?", - z, - x, - y); - } catch (Exception exception) { - // probably the table doesn't exists - if (LOGGER.isLoggable(Level.SEVERE)) { - LOGGER.log( - Level.SEVERE, - "Something bad happen when deleting create time for tile '%d-%d-%d'.".formatted(x, y, z), - exception); - } - } - } - - /** Helper method that retrieves the create time of a tile. */ - private Long getTileCreateTime(File file, long z, long x, long y) throws StorageException { - String query = - "SELECT create_time FROM tiles_metadata WHERE zoom_level = ? " + "AND tile_column = ? AND tile_row = ?"; - try { - return connectionManager.executeQuery( - file, - resultSet -> { - if (resultSet.next()) { - return resultSet.getLong(1); - } - return null; - }, - query, - z, - x, - y); - } catch (Exception exception) { - // probably the table doesn't exists - if (LOGGER.isLoggable(Level.SEVERE)) { - LOGGER.log( - Level.SEVERE, - "Something bad happen when querying create time for tile '%d-%d-%d'.".formatted(x, y, z), - exception); - } - } - return null; - } - - /** Helper method that puts the create time of a tile opening a connection for it. */ - private void putTileCreateTime(File file, long z, long x, long y, long createTime) { - connectionManager.doWork(file, false, connection -> { - putTileCreateTime(connection, z, x, y, createTime); - }); - } - - /** Helper method that puts the create time of a tile using the provided connection. */ - private void putTileCreateTime(Connection connection, long z, long x, long y, long createTime) { - createTilesMetadataTable(connection); - connectionManager.executeSql( - connection, "INSERT OR REPLACE INTO tiles_metadata VALUES (?, ?, ?, ?);", z, x, y, createTime); - } - - private void createTilesMetadataTable(Connection connection) { - connectionManager.executeSql( - connection, - "CREATE TABLE IF NOT EXISTS tiles_metadata (zoom_level integer, tile_column integer, " - + "tile_row integer, create_time integer, " - + "CONSTRAINT pk_tiles PRIMARY KEY(zoom_level, tile_column,tile_row));"); - } - - /** Init database file if it doesn't exists. */ - void initDatabaseFileIfNeeded(File file, String layerName, String format) { - if (file.exists()) { - // database file exists - return; - } - // initiating the database file - connectionManager.doWork(file, false, (connection) -> { - try { - // creating mbtiles tables - SqlUtil.runScript(getClass().getResourceAsStream("/org/geotools/mbtiles/mbtiles.sql"), connection); - // create tiles metadata table for storing the create time if needed - createTilesMetadataTable(connection); - // insert mbtiles metadata for this layer - insertMbtilesLayerMetadata(file, connection, layerName, format); - } catch (Exception exception) { - throw Utils.exception(exception, "Error running geotools mbtiles sql script."); - } - }); - } - - /** Store the mbtiles metadata associated with a file. */ - private void insertMbtilesLayerMetadata(File file, Connection connection, String layerName, String format) { - MBTilesMetadata gtMetadata = new MBTilesMetadata(); - gtMetadata.setName(layerName); - // checking if we have a mbtiles supported format, otherwise we don't insert anything - if (format.contains("png")) { - gtMetadata.setFormat(MBTilesMetadata.t_format.PNG); - } else if (format.contains("jpeg")) { - gtMetadata.setFormat(MBTilesMetadata.t_format.JPEG); - } else if (format.contains("protobuf")) { - gtMetadata.setFormat(MBTilesMetadata.t_format.PBF); - } - MBTilesMetadata existingMetadata = layersMetadata.get(FileManager.normalizePathValue(layerName)); - if (existingMetadata != null) { - // we have some user provided metadata let's use it - gtMetadata.setName(layerName); - gtMetadata.setAttribution(existingMetadata.getAttribution()); - gtMetadata.setBounds(existingMetadata.getBounds()); - gtMetadata.setDescription(existingMetadata.getDescription()); - gtMetadata.setMaxZoom(existingMetadata.getMaxZoom()); - gtMetadata.setMinZoom(existingMetadata.getMinZoom()); - gtMetadata.setType(existingMetadata.getType()); - gtMetadata.setVersion(existingMetadata.getVersion()); - } - try (MBTilesFile mbtiles = GeoToolsMbtilesUtils.getMBTilesFile(connection, file)) { - mbtiles.saveMetaData(gtMetadata); - } catch (Exception exception) { - throw Utils.exception(exception, "Error storing metadata on file '%s'.", file); - } - } - - /** Reads user provided mbtiles metadata for a layer. */ - private void initMbtilesLayersMetadata(String mbtilesMetadataDirectoryPath) { - if (mbtilesMetadataDirectoryPath == null) { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Mbtiles metadata directory path is NULL, no mbtiles metadata will be parsed."); - } - return; - } - File mbtilesMetadataDirectory = new File(mbtilesMetadataDirectoryPath); - if (!mbtilesMetadataDirectory.exists()) { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Mbtiles metadata directory '%s' doesn't exists, no mbtiles metadata will be parsed." - .formatted(mbtilesMetadataDirectoryPath)); - } - return; - } - File[] files = mbtilesMetadataDirectory.listFiles(); - if (files == null) { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("No files present in mbtiles metadata directory '%s', no mbtiles metadata will be parsed." - .formatted(mbtilesMetadataDirectoryPath)); - } - return; - } - for (File file : files) { - Matcher matcher = MBTILES_METADATA_FILE_NAME_PATTERN.matcher(file.getName()); - if (matcher.matches()) { - // extracting the layer name - String layerName = matcher.group(1); - // parsing mbtiles metadata properties - Properties metadata = new Properties(); - try (InputStream input = new FileInputStream(file)) { - metadata.load(input); - } catch (Exception exception) { - throw Utils.exception(exception, "Error reading mbtiles metadata file '%s'.", file); - } - // creating geotools mbtiles metadata file - MBTilesMetadata gtMetadata = new MBTilesMetadata(); - gtMetadata.setAttribution(metadata.getProperty("attribution")); - gtMetadata.setBoundsStr(metadata.getProperty("bounds")); - gtMetadata.setDescription(metadata.getProperty("description")); - gtMetadata.setMaxZoomStr(metadata.getProperty("maxZoom")); - gtMetadata.setMinZoomStr(metadata.getProperty("minZoom")); - gtMetadata.setTypeStr(metadata.getProperty("type")); - gtMetadata.setVersion(metadata.getProperty("version")); - // index the parsed mbtiles metadata - layersMetadata.put(layerName, gtMetadata); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Parsed mbtiles metadata for layer '%s'.".formatted(layerName)); - } - } - } - } - - @Override - public Map>> getParametersMapping(String layerName) { - try { - return connectionManager.executeQuery( - metadataFile, - resultSet -> { - try { - Map>> result = new HashMap<>(); - while (resultSet.next()) { - Map params = ParametersUtils.getMap(resultSet.getString(1)); - result.put(ParametersUtils.getId(params), Optional.of(params)); - } - return result; - } catch (Exception exception) { - throw Utils.exception(exception, "Error reading result set."); - } - }, - "SELECT value FROM metadata WHERE layerName = ? AND key like 'parameters.%';", - layerName); - } catch (Exception exception) { - // probably because the metadata table doesn't exists - if (LOGGER.isLoggable(Level.SEVERE)) { - LOGGER.log(Level.SEVERE, "Error getting metadata from file '%s'.".formatted(metadataFile), exception); - } - return Collections.emptyMap(); - } - } - - void persistParameterMap(TileObject stObj) { - if (Objects.nonNull(stObj.getParametersId())) { - putLayerMetadata( - stObj.getLayerName(), - "parameters." + stObj.getParametersId(), - ParametersUtils.getKvp(stObj.getParameters())); - } - } -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/MbtilesConfigurationProvider.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/MbtilesConfigurationProvider.java deleted file mode 100644 index 994ad5aad4..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/MbtilesConfigurationProvider.java +++ /dev/null @@ -1,33 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import com.thoughtworks.xstream.XStream; -import org.geowebcache.config.Info; -import org.geowebcache.config.XMLConfigurationProvider; - -/** Configuration provider for the mbtiles blob store */ -public class MbtilesConfigurationProvider implements XMLConfigurationProvider { - - @Override - public XStream getConfiguredXStream(XStream xs) { - xs.alias("MbtilesBlobStore", MbtilesInfo.class); - return xs; - } - - @Override - public boolean canSave(Info i) { - return i instanceof MbtilesInfo; - } -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/MbtilesInfo.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/MbtilesInfo.java deleted file mode 100644 index a0dce23311..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/MbtilesInfo.java +++ /dev/null @@ -1,100 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.io.Serial; -import org.geowebcache.layer.TileLayerDispatcher; -import org.geowebcache.locks.LockProvider; -import org.geowebcache.storage.BlobStore; -import org.geowebcache.storage.StorageException; - -/** Holder for the properties needed to configure a mbtiles blob store. */ -public class MbtilesInfo extends SqliteInfo { - @Serial - private static final long serialVersionUID = -6618985107587790155L; - - public MbtilesInfo() { - super(); - } - - public MbtilesInfo(String id) { - super(id); - } - - private String mbtilesMetadataDirectory; - - private int executorConcurrency = 5; - - private Boolean gzipVector = false; - - public String getMbtilesMetadataDirectory() { - return mbtilesMetadataDirectory; - } - - public void setMbtilesMetadataDirectory(String mbtilesMetadataDirectory) { - this.mbtilesMetadataDirectory = mbtilesMetadataDirectory; - } - - public int getExecutorConcurrency() { - return executorConcurrency; - } - - public void setExecutorConcurrency(int executorConcurrency) { - this.executorConcurrency = executorConcurrency; - } - - public boolean isGzipVector() { - return gzipVector != null && gzipVector; - } - - public void setGzipVector(boolean gzipVector) { - this.gzipVector = gzipVector; - } - - @Override - public BlobStore createInstance(TileLayerDispatcher layers, LockProvider lockProvider) throws StorageException { - return new MbtilesBlobStore(this, super.getConnectionManager()); - } - - @Override - public String toString() { - return "MBTiles BlobStore"; - } - - @Override - public int hashCode() { - final int prime = 31; - int result = super.hashCode(); - result = prime * result + executorConcurrency; - result = prime * result + ((gzipVector == null) ? 0 : gzipVector.hashCode()); - result = prime * result + ((mbtilesMetadataDirectory == null) ? 0 : mbtilesMetadataDirectory.hashCode()); - return result; - } - - @Override - public boolean equals(Object obj) { - if (this == obj) return true; - if (!super.equals(obj)) return false; - if (getClass() != obj.getClass()) return false; - MbtilesInfo other = (MbtilesInfo) obj; - if (executorConcurrency != other.executorConcurrency) return false; - if (gzipVector == null) { - if (other.gzipVector != null) return false; - } else if (!gzipVector.equals(other.gzipVector)) return false; - if (mbtilesMetadataDirectory == null) { - if (other.mbtilesMetadataDirectory != null) return false; - } else if (!mbtilesMetadataDirectory.equals(other.mbtilesMetadataDirectory)) return false; - return true; - } -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/OperationsRest.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/OperationsRest.java deleted file mode 100644 index 8b34d653a2..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/OperationsRest.java +++ /dev/null @@ -1,199 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.io.BufferedOutputStream; -import java.io.File; -import java.io.FileInputStream; -import java.io.FileOutputStream; -import java.nio.file.Files; -import java.util.UUID; -import java.util.logging.Level; -import java.util.logging.Logger; -import java.util.zip.ZipEntry; -import java.util.zip.ZipInputStream; -import org.apache.commons.io.FileUtils; -import org.geotools.util.logging.Logging; -import org.geowebcache.config.BlobStoreConfiguration; -import org.geowebcache.config.BlobStoreInfo; -import org.geowebcache.config.ServerConfiguration; -import org.geowebcache.layer.TileLayerDispatcher; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.http.HttpStatus; -import org.springframework.http.ResponseEntity; -import org.springframework.stereotype.Controller; -import org.springframework.web.bind.annotation.RequestMapping; -import org.springframework.web.bind.annotation.RequestMethod; -import org.springframework.web.bind.annotation.RequestParam; -import org.springframework.web.bind.annotation.ResponseBody; -import org.springframework.web.multipart.MultipartFile; - -@Controller -@RequestMapping("**/sqlite") -public class OperationsRest { - - private static Logger LOGGER = Logging.getLogger(OperationsRest.class.getName()); - - @Autowired - private TileLayerDispatcher tileLayerDispatcher; - - @Autowired - private BlobStoreConfiguration blobConfiguration; - - @Autowired - private ServerConfiguration gwcConfiguration; - - @RequestMapping(value = "/replace", method = RequestMethod.POST) - public @ResponseBody ResponseEntity replace( - @RequestParam(value = "layer") String layer, - @RequestParam(value = "destination", required = false) String destination, - @RequestParam(value = "source", required = false) String source, - @RequestParam(value = "file", required = false) MultipartFile uploadedFile) { - // we need to close this resources at the end - File workingDirectory = null; - File file = null; - try { - // create a temporary working directory (may not be used) - workingDirectory = Files.createTempDirectory("replace-operation-").toFile(); - // finding the blobstore associated t our layer - SqliteBlobStore blobStore = getBlobStoreForLayer(layer); - if (blobStore == null) { - return ResponseEntity.status(HttpStatus.BAD_REQUEST) - .body("No SQLite store could be associated with provided layer."); - } - // finding the file or the directory that will be used to replace - if (uploadedFile != null && !uploadedFile.isEmpty()) { - // it was an upload file - file = handleFileUpload(uploadedFile, workingDirectory); - } else if (source != null) { - // the file is already present - file = new File(source); - } - if (file == null || !file.exists()) { - return ResponseEntity.status(HttpStatus.BAD_REQUEST).body("Provided file is NULL or doesn't exists."); - } - // if we have a zip file we need to unzip it - file = unzipFileIfNeeded(file, workingDirectory); - if (file.isDirectory()) { - // we invoke the replace directory variant - blobStore.replace(file); - } else { - if (destination == null) { - return ResponseEntity.status(HttpStatus.BAD_REQUEST) - .body("Destination is required for single files."); - } - // we replace the single file - blobStore.replace(file, destination); - } - } catch (Exception exception) { - if (LOGGER.isLoggable(Level.SEVERE)) { - LOGGER.log(Level.SEVERE, "Error executing the replace operation.", exception); - } - return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(exception.getMessage()); - } finally { - // cleaning everything - FileUtils.deleteQuietly(workingDirectory); - } - return ResponseEntity.status(HttpStatus.OK).body(null); - } - - private File handleFileUpload(MultipartFile uploadedFile, File workingDirectory) throws Exception { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Handling file upload."); - } - // getting the uploaded file content - File outputFile = new File(workingDirectory, UUID.randomUUID().toString()); - byte[] bytes = uploadedFile.getBytes(); - try (BufferedOutputStream stream = new BufferedOutputStream(new FileOutputStream(outputFile))) { - stream.write(bytes); - } - return outputFile; - } - - private File unzipFileIfNeeded(File file, File workingDirectory) throws Exception { - if (file.isDirectory()) { - // is not a zip file so nothing to do - return file; - } - try (FileInputStream fileInput = new FileInputStream(file); - ZipInputStream zipInput = new ZipInputStream(fileInput)) { - ZipEntry zipEntry = zipInput.getNextEntry(); - if (zipEntry == null) { - // is not a zip file nothing to do - return file; - } - // is a zip file we need to extract is content - return unzip(zipInput, zipEntry, workingDirectory); - } - } - - private File unzip(ZipInputStream zipInputStream, ZipEntry zipEntry, File workingDirectory) throws Exception { - // output directory for our zip file content - File outputDirectory = new File(workingDirectory, UUID.randomUUID().toString()); - outputDirectory.mkdir(); - // unzipping all zip file entries - while (zipEntry != null) { - File outputFile = new File(outputDirectory, zipEntry.getName()); - if (zipEntry.isDirectory()) { - // this entry is a directory, so we only need to create the directory - outputFile.mkdir(); - } else { - // is a file we need to extract is content - extractFile(zipInputStream, outputFile); - } - zipInputStream.closeEntry(); - zipEntry = zipInputStream.getNextEntry(); - } - return outputDirectory; - } - - private void extractFile(ZipInputStream inputStream, File outputFile) throws Exception { - // extracting zip entry file content - byte[] bytes = new byte[1024]; - try (FileOutputStream outputStream = new FileOutputStream(outputFile)) { - int read; - while ((read = inputStream.read(bytes)) != -1) { - outputStream.write(bytes, 0, read); - } - } - } - - private SqliteBlobStore getBlobStoreForLayer(String layerName) throws Exception { - // let's find layer associated store - String blobStoreId = tileLayerDispatcher.getTileLayer(layerName).getBlobStoreId(); - BlobStoreInfo blobStoreConfig = null; - for (BlobStoreInfo candidateBlobStoreConfig : blobConfiguration.getBlobStores()) { - if (blobStoreId == null) { - // we need to find the default configuration - if (candidateBlobStoreConfig.isDefault()) { - // this is the default configuration, we are done - blobStoreConfig = candidateBlobStoreConfig; - break; - } - } - if (candidateBlobStoreConfig.getName().equals(blobStoreId)) { - // we need to find a specific store by is id - blobStoreConfig = candidateBlobStoreConfig; - break; - } - } - if (blobStoreConfig == null || !(blobStoreConfig instanceof SqliteInfo)) { - // no store found or the store is not an sqlite store - return null; - } - // returning an instance of found store - return (SqliteBlobStore) - blobStoreConfig.createInstance(tileLayerDispatcher, gwcConfiguration.getLockProvider()); - } -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/SqliteBlobStore.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/SqliteBlobStore.java deleted file mode 100644 index 389df52dae..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/SqliteBlobStore.java +++ /dev/null @@ -1,85 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.io.File; -import org.geowebcache.storage.BlobStore; - -/** Base class for all SQLite based blob stores. */ -abstract class SqliteBlobStore implements BlobStore { - - private final SqliteInfo configuration; - - protected final FileManager fileManager; - protected final SqliteConnectionManager connectionManager; - - protected SqliteBlobStore(SqliteInfo configuration, SqliteConnectionManager connectionManager) { - this.configuration = configuration; - - this.fileManager = new FileManager( - configuration.getRootDirectoryFile(), - configuration.getTemplatePath(), - configuration.getRowRangeCount(), - configuration.getColumnRangeCount()); - this.connectionManager = connectionManager; - } - - SqliteInfo getConfiguration() { - return configuration; - } - - /** Will move the new file to is destination. */ - void replace(File newFile, String destination) { - // we ask the connection manager to replace the files - File destinationFile = new File(configuration.getRootDirectoryFile(), destination); - connectionManager.replace(destinationFile, newFile); - } - - /** Merging the given directory with this blob store directory. */ - void replace(File directory) { - // we may have more that one top files - File[] files = directory.listFiles(); - if (files == null) { - // no files, nothing to do - return; - } - for (File file : files) { - // walk the file tree of each top file - walkDirectoryTreeAndReplace(file, ""); - } - } - - /** Helper method that will recursively perform a replace operation using the directory files. */ - private void walkDirectoryTreeAndReplace(File current, String relativePath) { - String newRelativePath = relativePath + File.separator + current.getName(); - if (!current.isDirectory()) { - // we have a file let's perform the replace operation - File destinationFile = new File(configuration.getRootDirectoryFile(), newRelativePath); - connectionManager.replace(destinationFile, current); - return; - } - // is a directory let's handle all the existing files - File[] files = current.listFiles(); - if (files == null) { - // no files, nothing to do - return; - } - for (File file : files) { - walkDirectoryTreeAndReplace(file, newRelativePath); - } - } - - @Override - public void destroy() {} -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/SqliteConnectionManager.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/SqliteConnectionManager.java deleted file mode 100644 index 6782bdbace..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/SqliteConnectionManager.java +++ /dev/null @@ -1,480 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.io.File; -import java.nio.file.Files; -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.util.ArrayList; -import java.util.Collections; -import java.util.List; -import java.util.Map; -import java.util.UUID; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.locks.ReentrantReadWriteLock; -import java.util.logging.Level; -import java.util.logging.Logger; -import org.apache.commons.io.FileUtils; -import org.geotools.util.logging.Logging; - -/** - * Manages the connections to sqlite databases files taking care of the concurrent access. The concurrent access are - * managed by JVM if two JVMs access the same database file the result is unpredictable. - */ -public final class SqliteConnectionManager { - - private static Logger LOGGER = Logging.getLogger(SqliteConnectionManager.class.getName()); - - private final ConcurrentHashMap pool = new ConcurrentHashMap<>(); - - private volatile boolean stopPoolReaper = false; - - public SqliteConnectionManager(SqliteInfo configuration) { - this(configuration.getPoolSize(), configuration.getPoolReaperIntervalMs()); - } - - SqliteConnectionManager(long poolSize, long poolReaperIntervalMs) { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Initiating connection poll: [poolSize='%d', poolReaperIntervalMs='%d']." - .formatted(poolSize, poolReaperIntervalMs)); - } - // let's load the sqlite driver - try { - Class.forName("org.sqlite.JDBC"); - } catch (Exception exception) { - throw Utils.exception(exception, "Error initiating sqlite driver."); - } - // computing some values used by the pool reaper - double poolSizeThreshold = poolSize * 0.9; - double connectionsToRemove = poolSize * 0.1; - // starting the connection pool reaper - new Thread(() -> { - while (!stopPoolReaper) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Current pool size is '%d' and threshold is '%f'." - .formatted(pool.size(), poolSizeThreshold)); - } - if (pool.size() > poolSizeThreshold) { - // we exceed the pool size threshold, time to reap the less used - // connections - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Reaping connections, current pool size %d.".formatted(pool.size())); - } - List pooledConnections = new ArrayList<>(pool.values()); - Collections.sort(pooledConnections); - for (int i = 0; i < connectionsToRemove && i < pooledConnections.size(); i++) { - pooledConnections.get(i).reapConnection(); - } - } - try { - Thread.sleep(poolReaperIntervalMs); - } catch (Exception exception) { - Thread.currentThread().interrupt(); - if (LOGGER.isLoggable(Level.WARNING)) { - LOGGER.warning("Pool reaper interrupted."); - } - } - } - if (LOGGER.isLoggable(Level.WARNING)) { - LOGGER.warning("Pool reaper stop."); - } - }) - .start(); - } - - /** Helper interface to submit work. */ - interface Work { - void doWork(Connection connection); - } - - /** Helper interface to submit work that needs to return something. */ - interface WorkWithResult { - T doWork(Connection connection); - } - - /** Helper interface to submit work that needs to return something. */ - interface ResultExtractor { - T extract(ResultSet resultSet) throws SQLException; - } - - /** Submit an SQL statement to be executed. */ - void executeSql(File file, String sql, Object... parameters) { - doWork(file, false, connection -> { - executeSql(connection, sql, parameters); - }); - } - - /** Submit an SQL statement to be executed with the provided connection. */ - void executeSql(Connection connection, String sql, Object... parameters) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Executing SQL '%s'.".formatted(sql)); - } - try (PreparedStatement statement = connection.prepareStatement(sql)) { - for (int i = 0; i < parameters.length; i++) { - statement.setObject(i + 1, parameters[i]); - } - statement.execute(); - } catch (Exception exception) { - throw Utils.exception(exception, "Error executing SQL '%s'.", sql); - } - } - - /** Submit a query to be executed. */ - T executeQuery(File file, ResultExtractor extractor, String query, Object... parameters) { - return doWork(file, true, connection -> { - return executeQuery(connection, extractor, query, parameters); - }); - } - - /** Submit a query to be executed with the provided connection. */ - T executeQuery(Connection connection, ResultExtractor extractor, String query, Object... parameters) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Executing query '%s'.".formatted(query)); - } - try (PreparedStatement statement = connection.prepareStatement(query)) { - for (int i = 0; i < parameters.length; i++) { - statement.setObject(i + 1, parameters[i]); - } - return extractor.extract(statement.executeQuery()); - } catch (Exception exception) { - throw Utils.exception(exception, "Error executing query '%s'.", query); - } - } - - /** Submit some work to be executed. */ - void doWork(File file, boolean readOnly, Work work) { - doWork(file, readOnly, (WorkWithResult) connection -> { - work.doWork(connection); - return null; - }); - } - - /** Submit some work to be executed that need to return something. */ - T doWork(File file, boolean readOnly, WorkWithResult work) { - if (readOnly) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Starting work on file '%s' in readonly mode.".formatted(file)); - } - } else { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Starting work on file '%s' in write mode.".formatted(file)); - } - } - // let's find or instantiate on the fly a pool connection for the current file - PooledConnection pooledConnection = getPooledConnection(file); - // acquiring the proper lock on the pooled connection (read or write lock) - pooledConnection = readOnly - ? pooledConnection.getReadLockOnValidConnection() - : pooledConnection.getWriteLockOnValidConnection(); - // the PooledConnection seems to manage the actual connection - @SuppressWarnings("PMD.CloseResource") - ExtendedConnection connection = pooledConnection.getExtendedConnection(); - try { - // do the work - T result = work.doWork(connection); - if (!connection.closeInvoked()) { - // the work didn't close the connection, this is fine unless the connection was - // retained for future usage - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Close was not invoked on extended connection."); - } - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Work on file '%s' is done.".formatted(file)); - } - return result; - } finally { - // releasing the acquired lock - if (readOnly) { - pooledConnection.releaseReadLock(); - } else { - pooledConnection.releaseWriteLock(); - } - } - } - - void replace(File currentFile, File newFile) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Replacing file '%s' with file '%s'.".formatted(currentFile, newFile)); - } - PooledConnection currentPooledConnection = - getPooledConnection(currentFile).getWriteLockOnValidConnection(); - try { - currentPooledConnection.reapConnection(); - if (currentFile.exists()) { - Files.delete(currentFile.toPath()); - } - FileUtils.moveFile(newFile, currentFile); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("File '%s' replaced with file '%s'.".formatted(currentFile, newFile)); - } - } catch (Exception exception) { - throw Utils.exception(exception, "Error replacing file '%s' with file '%s'.", currentFile, newFile); - } finally { - currentPooledConnection.releaseWriteLock(); - } - } - - void delete(File file) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Deleting file '%s'.".formatted(file)); - } - if (!file.exists()) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("File '%s' doesn't exists.".formatted(file)); - } - return; - } - PooledConnection pooledConnection = getPooledConnection(file).getWriteLockOnValidConnection(); - try { - pooledConnection.closeConnection(); - FileUtils.deleteQuietly(file); - pool.remove(file); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("File '%s' deleted.".formatted(file)); - } - } catch (Exception exception) { - throw Utils.exception(exception, "Error deleting file '%s'.", file); - } finally { - pooledConnection.releaseWriteLock(); - } - } - - void rename(File currentFile, File newFile) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Renaming file '%s' to '%s'.".formatted(currentFile, newFile)); - } - PooledConnection pooledConnection = getPooledConnection(currentFile).getWriteLockOnValidConnection(); - try { - pooledConnection.closeConnection(); - pool.remove(currentFile); - FileUtils.moveFile(currentFile, newFile); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("File '%s' renamed to '%s'.".formatted(currentFile, newFile)); - } - } catch (Exception exception) { - throw Utils.exception(exception, "Renaming file '%s' to '%s'.", currentFile, newFile); - } finally { - pooledConnection.releaseWriteLock(); - } - } - - public Map getPool() { - return pool; - } - - void reapAllConnections() { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Reaping all connections."); - } - pool.values().forEach(PooledConnection::reapConnection); - } - - void stopPoolReaper() { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Stopping the pool reaper."); - } - stopPoolReaper = true; - } - - private PooledConnection getPooledConnection(File file) { - try { - PooledConnection pooledConnection = pool.get(file); - if (pooledConnection != null) { - // a connection already exists - return pooledConnection; - } - // creating a new pooled connection for the database file - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Creating pooled connection to file '%s'.".formatted(file)); - } - pooledConnection = new PooledConnection(file); - pooledConnection.getWriteLock(); - try { - PooledConnection existing = pool.putIfAbsent(file, pooledConnection); - if (existing != null) { - // someone create a pooled connection for this file in the meantime - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Connection to file '%s' already exists, closing this one.".formatted(file)); - } - pooledConnection.closeConnection(); - return existing; - } - // effectively open a connection to the database file - pooledConnection.init(); - return pooledConnection; - } finally { - pooledConnection.releaseWriteLock(); - } - } catch (Exception exception) { - throw Utils.exception(exception, "Error opening connection to file '%s'.", file); - } - } - - /** Helper class that contains all the info associated to an open connection. */ - private final class PooledConnection implements Comparable { - - private final File file; - private Connection connection; - - private final ReentrantReadWriteLock lock; - - private long lastAccess; - private volatile boolean closed; - - PooledConnection(File file) { - this.file = file; - lock = new ReentrantReadWriteLock(); - closed = true; - } - - void init() { - connection = openConnection(file); - lastAccess = System.currentTimeMillis(); - closed = false; - } - - @Override - public int compareTo(PooledConnection other) { - if (this.lastAccess < other.lastAccess) { - return -1; - } - return 1; - } - - ExtendedConnection getExtendedConnection() { - lastAccess = System.currentTimeMillis(); - return new ExtendedConnection(connection); - } - - void reapConnection() { - getWriteLock(); - closeConnection(); - pool.remove(file); - releaseWriteLock(); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Connection to file '%s' reaped.".formatted(file)); - } - } - - void closeConnection() { - if (!closed) { - // this connection is open let's close it - try { - connection.close(); - closed = true; - } catch (Exception exception) { - throw Utils.exception("Error closing connection to file '%s'.", file); - } - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Connection to file '%s' closed.".formatted(file)); - } - } - } - - void getReadLock() { - String logId = ""; - if (LOGGER.isLoggable(Level.FINE)) { - logId = UUID.randomUUID().toString(); - LOGGER.fine("[%s] Waiting for read lock on file '%s'.".formatted(logId, file)); - } - lock.readLock().lock(); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("[%s] Read lock on file '%s' obtained.".formatted(logId, file)); - } - } - - PooledConnection getReadLockOnValidConnection() { - getReadLock(); - if (!closed) { - // this connection is ok - return this; - } - releaseReadLock(); - // this connection was closed in the meantime we need to create a new one (trying 10 - // times) - for (int i = 0; i < 10; i++) { - PooledConnection pooledConnection = SqliteConnectionManager.this.getPooledConnection(file); - // obtain the read lock - pooledConnection.getReadLock(); - if (!pooledConnection.closed) { - return pooledConnection; - } - } - throw Utils.exception("Could not obtain a valid connection to file '%s'.", file); - } - - void releaseReadLock() { - lock.readLock().unlock(); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Read lock on file '%s' released.".formatted(file)); - } - } - - void getWriteLock() { - String logId = ""; - if (LOGGER.isLoggable(Level.FINE)) { - logId = UUID.randomUUID().toString(); - LOGGER.fine("[%s] Waiting for write lock on file '%s'.".formatted(logId, file)); - } - lock.writeLock().lock(); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("[%s] Write lock on file '%s' obtained.".formatted(logId, file)); - } - } - - PooledConnection getWriteLockOnValidConnection() { - getWriteLock(); - if (!closed) { - // this connection is ok - return this; - } - releaseWriteLock(); - // this connection was closed in the meantime we need to create a new one (trying 10 - // times) - for (int i = 0; i < 10; i++) { - PooledConnection pooledConnection = SqliteConnectionManager.this.getPooledConnection(file); - // obtain the write lock - pooledConnection.getWriteLock(); - if (!pooledConnection.closed) { - return pooledConnection; - } - } - throw Utils.exception("Could not obtain a valid connection to file '%s'.", file); - } - - void releaseWriteLock() { - lock.writeLock().unlock(); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Write lock on file '%s' released.".formatted(file)); - } - } - - private Connection openConnection(File file) { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Opening connection to file '%s'.".formatted(file)); - } - Utils.createFileParents(file); - try { - return DriverManager.getConnection("jdbc:sqlite:" + file.getPath()); - } catch (Exception exception) { - throw Utils.exception(exception, "Error opening connection to file '%s'.", file); - } - } - } -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/SqliteInfo.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/SqliteInfo.java deleted file mode 100644 index 574d1d45ad..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/SqliteInfo.java +++ /dev/null @@ -1,171 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.io.File; -import java.io.Serial; -import java.util.UUID; -import org.geowebcache.config.BlobStoreInfo; - -/** Holder for the common properties needed to configure a sqlite based blob store. */ -public abstract class SqliteInfo extends BlobStoreInfo { - @Serial - private static final long serialVersionUID = 2300159159094621077L; - - public SqliteInfo() { - this(UUID.randomUUID().toString()); - } - - public SqliteInfo(String id) { - super(id); - } - - private transient SqliteConnectionManager connectionManager; - - private String rootDirectory; - - private String templatePath = Utils.buildPath("{layer}", "{grid}{format}{params}", "{z}_{x}_{y}.sqlite"); - - private long poolSize = 1000; - - private long poolReaperIntervalMs = 500; - - private long rowRangeCount = 250; - - private long columnRangeCount = 250; - - private boolean eagerDelete = false; - - private boolean useCreateTime = true; - - public File getRootDirectoryFile() { - File file = new File(rootDirectory); - if (!file.exists()) { - file.mkdirs(); - } - return file; - } - - public String getRootDirectory() { - return rootDirectory; - } - - public void setRootDirectory(String rootDirectory) { - this.rootDirectory = rootDirectory; - } - - public String getTemplatePath() { - return templatePath; - } - - public void setTemplatePath(String templatePath) { - this.templatePath = templatePath; - } - - public long getPoolSize() { - return poolSize; - } - - public void setPoolSize(long poolSize) { - this.poolSize = poolSize; - } - - public long getPoolReaperIntervalMs() { - return poolReaperIntervalMs; - } - - public void setPoolReaperIntervalMs(long poolReaperIntervalMs) { - this.poolReaperIntervalMs = poolReaperIntervalMs; - } - - public long getRowRangeCount() { - return rowRangeCount; - } - - public void setRowRangeCount(long rowRangeCount) { - this.rowRangeCount = rowRangeCount; - } - - public long getColumnRangeCount() { - return columnRangeCount; - } - - public void setColumnRangeCount(long columnRangeCount) { - this.columnRangeCount = columnRangeCount; - } - - public boolean eagerDelete() { - return eagerDelete; - } - - public void setEagerDelete(boolean eagerDelete) { - this.eagerDelete = eagerDelete; - } - - public boolean useCreateTime() { - return useCreateTime; - } - - public void setUseCreateTime(boolean useCreateTime) { - this.useCreateTime = useCreateTime; - } - - @Override - public String getLocation() { - return rootDirectory; - } - - protected synchronized SqliteConnectionManager getConnectionManager() { - if (connectionManager == null) { - connectionManager = new SqliteConnectionManager(this); - } - return connectionManager; - } - - @Override - public int hashCode() { - final int prime = 31; - int result = super.hashCode(); - result = prime * result + (int) (columnRangeCount ^ (columnRangeCount >>> 32)); - result = prime * result + (eagerDelete ? 1231 : 1237); - result = prime * result + (int) (poolReaperIntervalMs ^ (poolReaperIntervalMs >>> 32)); - result = prime * result + (int) (poolSize ^ (poolSize >>> 32)); - result = prime * result + ((rootDirectory == null) ? 0 : rootDirectory.hashCode()); - result = prime * result + (int) (rowRangeCount ^ (rowRangeCount >>> 32)); - result = prime * result + ((templatePath == null) ? 0 : templatePath.hashCode()); - result = prime * result + (useCreateTime ? 1231 : 1237); - return result; - } - - @Override - public boolean equals(Object obj) { - if (this == obj) return true; - if (!super.equals(obj)) return false; - if (getClass() != obj.getClass()) return false; - SqliteInfo other = (SqliteInfo) obj; - if (columnRangeCount != other.columnRangeCount) return false; - if (eagerDelete != other.eagerDelete) return false; - if (poolReaperIntervalMs != other.poolReaperIntervalMs) return false; - if (poolSize != other.poolSize) return false; - if (rootDirectory == null) { - if (other.rootDirectory != null) return false; - } else if (!rootDirectory.equals(other.rootDirectory)) return false; - if (rowRangeCount != other.rowRangeCount) return false; - if (templatePath == null) { - if (other.templatePath != null) return false; - } else if (!templatePath.equals(other.templatePath)) return false; - if (useCreateTime != other.useCreateTime) return false; - return true; - } -} diff --git a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/Utils.java b/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/Utils.java deleted file mode 100644 index f73174c584..0000000000 --- a/geowebcache/sqlite/src/main/java/org/geowebcache/sqlite/Utils.java +++ /dev/null @@ -1,123 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import com.google.errorprone.annotations.FormatMethod; -import java.io.File; -import java.util.HashMap; -import java.util.Map; -import java.util.Objects; -import java.util.logging.Logger; -import org.apache.commons.io.IOUtils; -import org.apache.commons.lang3.StringUtils; -import org.geotools.util.logging.Logging; -import org.geowebcache.io.ByteArrayResource; -import org.geowebcache.io.Resource; - -/** Some utilities objects and functions used internally. */ -final class Utils { - - private static Logger LOGGER = Logging.getLogger(Utils.class.getName()); - - // if we get the windows path separator we need to escape it for regex use - static final String REGEX_FILE_SEPARATOR = File.separator.equals("\\") ? "\\\\" : File.separator; - - private Utils() {} - - static final class Tuple { - - final T first; - final U second; - - private Tuple(T first, U second) { - this.first = first; - this.second = second; - } - - static Tuple tuple(R first, S second) { - return new Tuple<>(first, second); - } - - @Override - public boolean equals(Object object) { - if (this == object) return true; - if (object == null || getClass() != object.getClass()) return false; - Tuple tuple = (Tuple) object; - return Objects.equals(first, tuple.first) && Objects.equals(second, tuple.second); - } - - @Override - public int hashCode() { - return Objects.hash(first, second); - } - - @Override - public String toString() { - return "Tuple{" + "first=" + first + ", second=" + second + '}'; - } - } - - @SafeVarargs - static Map tuplesToMap(Tuple... tuples) { - Map map = new HashMap<>(); - for (Tuple tuple : tuples) { - map.put(tuple.first, tuple.second); - } - return map; - } - - @FormatMethod - static void check(boolean condition, String message, Object... arguments) { - if (!condition) { - throw exception(message, arguments); - } - } - - @FormatMethod - static RuntimeException exception(String message, Object... arguments) { - String finalMessage = message.formatted(arguments); - LOGGER.severe(finalMessage); - return new RuntimeException(finalMessage); - } - - @FormatMethod - static RuntimeException exception(Exception exception, String message, Object... arguments) { - String finalMessage = message.formatted(arguments); - LOGGER.severe(finalMessage); - return new RuntimeException(finalMessage, exception); - } - - static byte[] resourceToByteArray(Resource resource) { - try { - return IOUtils.toByteArray(resource.getInputStream()); - } catch (Exception exception) { - throw exception(exception, "Error converting resource to byte array."); - } - } - - static Resource byteArrayToResource(byte[] data) { - return new ByteArrayResource(data); - } - - static String buildPath(String... pathParts) { - return StringUtils.join(pathParts, File.separator); - } - - static void createFileParents(File file) { - File parentFile = file.getParentFile(); - if (parentFile != null) { - parentFile.mkdirs(); - } - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/FileManagerTest.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/FileManagerTest.java deleted file mode 100644 index e4b70152a6..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/FileManagerTest.java +++ /dev/null @@ -1,279 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import static org.geowebcache.sqlite.Utils.Tuple; -import static org.geowebcache.sqlite.Utils.Tuple.tuple; -import static org.geowebcache.sqlite.Utils.tuplesToMap; -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.containsInAnyOrder; -import static org.hamcrest.Matchers.is; -import static org.hamcrest.Matchers.notNullValue; - -import java.io.File; -import java.util.Collections; -import java.util.List; -import java.util.Map; -import org.geowebcache.mime.MimeType; -import org.geowebcache.storage.TileObject; -import org.geowebcache.storage.TileRange; -import org.junit.Test; - -public final class FileManagerTest extends TestSupport { - - @Test - public void testGetLayerAssociatedFilesWithMultipleLevels() throws Exception { - String pathTemplate = - Utils.buildPath("tiles", "{format}", "{style}", "zoom-{z}", "{layer}", "ranges-{x}-{y}.sqlite"); - File fileAsiaA = createFileInRootDir( - Utils.buildPath("tiles", "png", "default", "zoom-10", "asia", "ranges-0-500.sqlite")); - File fileAsiaB = createFileInRootDir( - Utils.buildPath("tiles", "png", "default", "zoom-11", "asia", "ranges-100-500.sqlite")); - File fileAsiaC = createFileInRootDir( - Utils.buildPath("tiles", "png", "dark-borders", "zoom-10", "asia", "ranges-0-500.sqlite")); - File fileAsiaD = createFileInRootDir( - Utils.buildPath("tiles", "jpeg", "dark-borders", "zoom-10", "asia", "ranges-0-500.sqlite")); - File fileAfricaA = createFileInRootDir( - Utils.buildPath("tiles", "png", "default", "zoom-10", "africa", "ranges-0-500.sqlite")); - File fileAmericaA = createFileInRootDir( - Utils.buildPath("tiles", "png", "dark-borders", "zoom-10", "america", "ranges-0-500.sqlite")); - File fileAmericaB = createFileInRootDir( - Utils.buildPath("tiles", "jpeg", "dark-borders", "zoom-10", "america", "ranges-0-500.sqlite")); - File fileEuropeA = createFileInRootDir( - Utils.buildPath("tiles", "gif", "dark-borders", "zoom-15", "europe", "ranges-4000-500.sqlite")); - FileManager fileManager = new FileManager(getRootDirectory(), pathTemplate, 500, 500); - List asiaFiles = fileManager.getFiles("asia"); - assertThat(asiaFiles, containsInAnyOrder(fileAsiaA, fileAsiaB, fileAsiaC, fileAsiaD)); - List africaFiles = fileManager.getFiles("africa"); - assertThat(africaFiles, containsInAnyOrder(fileAfricaA)); - List americaFiles = fileManager.getFiles("america"); - assertThat(americaFiles, containsInAnyOrder(fileAmericaA, fileAmericaB)); - List europeFiles = fileManager.getFiles("europe"); - assertThat(europeFiles, containsInAnyOrder(fileEuropeA)); - } - - @Test - public void testGetLayerAssociatedFilesWithNoLayer() throws Exception { - String pathTemplate = Utils.buildPath("tiles", "{z}", "tiles-{x}.sqlite"); - File fileA = createFileInRootDir(Utils.buildPath("tiles", "10", "tiles-0.sqlite")); - File fileB = createFileInRootDir(Utils.buildPath("tiles", "10", "tiles-500.sqlite")); - File fileC = createFileInRootDir(Utils.buildPath("tiles", "11", "tiles-0.sqlite")); - File fileD = createFileInRootDir(Utils.buildPath("tiles", "11", "tiles-500.sqlite")); - FileManager fileManager = new FileManager(getRootDirectory(), pathTemplate, 1000, 1000); - List filesA = fileManager.getFiles((String) null); - assertThat(filesA, containsInAnyOrder(fileA, fileB, fileC, fileD)); - List filesB = fileManager.getFiles("something"); - assertThat(filesB, containsInAnyOrder(fileA, fileB, fileC, fileD)); - } - - @Test - public void testLayerAssociatedFilesWithOnlyOneLevel() throws Exception { - String pathTemplate = "{layer}.sqlite"; - File asiaFile = createFileInRootDir("asia.sqlite"); - File americaFile = createFileInRootDir("america.sqlite"); - File africaFile = createFileInRootDir("africa.sqlite"); - File australiaFile = createFileInRootDir("australia.sqlite"); - FileManager fileManager = new FileManager(getRootDirectory(), pathTemplate, 1000, 1000); - List asiaFiles = fileManager.getFiles("asia"); - assertThat(asiaFiles, containsInAnyOrder(asiaFile)); - List americaFiles = fileManager.getFiles("america"); - assertThat(americaFiles, containsInAnyOrder(americaFile)); - List africaFiles = fileManager.getFiles("africa"); - assertThat(africaFiles, containsInAnyOrder(africaFile)); - List australiaFiles = fileManager.getFiles("australia"); - assertThat(australiaFiles, containsInAnyOrder(australiaFile)); - } - - @Test - public void testGetGridSetAndLayerAssociatedFiles() throws Exception { - String pathTemplate = Utils.buildPath( - "tiles", "{grid}", "{format}", "{style}", "zoom-{z}", "{layer}", "ranges-{x}-{y}.sqlite"); - File fileAsiaGrid1A = createFileInRootDir( - Utils.buildPath("tiles", "grid1", "png", "default", "zoom-10", "asia", "ranges-0-500.sqlite")); - File fileAsiaGrid2B = createFileInRootDir( - Utils.buildPath("tiles", "grid2", "png", "default", "zoom-11", "asia", "ranges-100-500.sqlite")); - File fileAsiaGrid3C = createFileInRootDir( - Utils.buildPath("tiles", "grid3", "png", "dark-borders", "zoom-10", "asia", "ranges-0-500.sqlite")); - File fileAsiaGrid3D = createFileInRootDir( - Utils.buildPath("tiles", "grid3", "jpeg", "dark-borders", "zoom-10", "asia", "ranges-0-500.sqlite")); - File fileAfricaGrid1A = createFileInRootDir( - Utils.buildPath("tiles", "grid1", "png", "default", "zoom-10", "africa", "ranges-0-500.sqlite")); - File fileAmericaGrid1A = createFileInRootDir( - Utils.buildPath("tiles", "grid1", "png", "dark-borders", "zoom-10", "america", "ranges-0-500.sqlite")); - File fileAmericaGrid2B = createFileInRootDir( - Utils.buildPath("tiles", "grid2", "jpeg", "dark-borders", "zoom-10", "america", "ranges-0-500.sqlite")); - File fileEuropeGrid1A = createFileInRootDir(Utils.buildPath( - "tiles", "grid1", "gif", "dark-borders", "zoom-15", "europe", "ranges-4000-500.sqlite")); - FileManager fileManager = new FileManager(getRootDirectory(), pathTemplate, 500, 500); - List asiaFilesGrid1 = fileManager.getFiles("asia", "grid1"); - assertThat(asiaFilesGrid1, containsInAnyOrder(fileAsiaGrid1A)); - List asiaFilesGrid2 = fileManager.getFiles("asia", "grid2"); - assertThat(asiaFilesGrid2, containsInAnyOrder(fileAsiaGrid2B)); - List asiaFilesGrid3 = fileManager.getFiles("asia", "grid3"); - assertThat(asiaFilesGrid3, containsInAnyOrder(fileAsiaGrid3C, fileAsiaGrid3D)); - List africaFilesGrid1 = fileManager.getFiles("africa", "grid1"); - assertThat(africaFilesGrid1, containsInAnyOrder(fileAfricaGrid1A)); - List americaFilesGrid1 = fileManager.getFiles("america", "grid1"); - assertThat(americaFilesGrid1, containsInAnyOrder(fileAmericaGrid1A)); - List americaFilesGrid2 = fileManager.getFiles("america", "grid2"); - assertThat(americaFilesGrid2, containsInAnyOrder(fileAmericaGrid2B)); - List europeFilesGrid1 = fileManager.getFiles("europe", "grid1"); - assertThat(europeFilesGrid1, containsInAnyOrder(fileEuropeGrid1A)); - } - - @Test - public void testSimplePathTemplate() throws Exception { - genericTestPath( - Utils.buildPath("tiles", "{grid}", "layers", "{layer}.sqlite"), - Utils.buildPath("tiles", "wgs84", "layers", "europe.sqlite"), - null, - null, - "europe", - "wgs84", - null); - } - - @Test - public void testPathTemplateWithRanges() throws Exception { - genericTestPath( - Utils.buildPath("{grid}", "tiles", "{layer}", "zoom-{z}", "ranges_{x}-{y}.sqlite"), - Utils.buildPath("wgs84", "tiles", "usa", "zoom-15", "ranges_3000-1000.sqlite"), - null, - new long[] {3024, 1534, 15}, - "usa", - "wgs84", - null); - } - - @Test - public void testPathTemplateWithParameters() throws Exception { - genericTestPath( - Utils.buildPath("{background}-tiles", "{grid}", "{layer}-{style}", "zoom-{z}", "ranges{x}{y}.sqlite"), - Utils.buildPath("blue-tiles", "wgs84", "america-default", "zoom-15", "ranges30001000.sqlite"), - null, - new long[] {3024, 1534, 15}, - "america", - "wgs84", - null, - tuple("style", "default"), - tuple("background", "blue")); - } - - @Test - public void testPathTemplateWithTileObject() throws Exception { - TileObject tile = TileObject.createCompleteTileObject( - "africa", - new long[] {7050, 5075, 11}, - "wgs84", - "jpeg", - tuplesToMap(tuple("style", "dark borders"), tuple("background", "blue")), - null); - String pathTemplate = - Utils.buildPath("{format}-tiles", "{grid}", "{layer}-{style}", "zoom-{z}", "ranges-{x}_{y}.sqlite"); - String expectedPath = - Utils.buildPath("jpeg-tiles", "wgs84", "africa-dark_borders", "zoom-11", "ranges-7000_4000.sqlite"); - FileManager fileManager = new FileManager(getRootDirectory(), pathTemplate, 2000, 1000); - File file = fileManager.getFile(tile); - assertThat(file.getCanonicalPath(), is(getRootDirectoryPath() + File.separator + expectedPath)); - } - - @Test - public void testPathTemplateWithInvalidChars() throws Exception { - genericTestPath( - Utils.buildPath("{grid}", "{format}", "{layer}.sqlite"), - Utils.buildPath("epsg_4326", "image_png", "europe_borders.sqlite"), - null, - null, - "europe\\borders", - "epsg:4326", - "image/png"); - } - - @Test - public void testGetFilesByTileRange() throws Exception { - String pathTemplate = Utils.buildPath("tiles", "{z}", "tiles-{x}-{y}.sqlite"); - FileManager fileManager = new FileManager(getRootDirectory(), pathTemplate, 10, 10); - File file1 = createFileInRootDir(Utils.buildPath("tiles", "5", "tiles-0-0.sqlite")); - File file2 = createFileInRootDir(Utils.buildPath("tiles", "5", "tiles-10-0.sqlite")); - File file3 = createFileInRootDir(Utils.buildPath("tiles", "5", "tiles-0-10.sqlite")); - File file4 = createFileInRootDir(Utils.buildPath("tiles", "5", "tiles-10-10.sqlite")); - File file5 = createFileInRootDir(Utils.buildPath("tiles", "6", "tiles-10-0.sqlite")); - long[][] rangeBounds = { - {9, 9, 12, 13, 5}, - {16, 5, 16, 5, 6} - }; - TileRange tileRange = new TileRange( - "layer", "EPSG:4326", 5, 6, rangeBounds, MimeType.createFromExtension("png"), Collections.emptyMap()); - Map> files = fileManager.getFiles(tileRange); - assertThat(files, notNullValue()); - assertThat(files.size(), is(5)); - checkFileExistsAndContainsTiles(files, file1, new long[] {9, 9, 9, 9, 5}); - checkFileExistsAndContainsTiles(files, file2, new long[] {10, 9, 12, 9, 5}); - checkFileExistsAndContainsTiles(files, file3, new long[] {9, 10, 9, 13, 5}); - checkFileExistsAndContainsTiles(files, file4, new long[] {10, 10, 12, 13, 5}); - checkFileExistsAndContainsTiles(files, file5, new long[] {16, 5, 16, 5, 6}); - } - - @Test - public void testPathSeparatorsAreConvertedToOsOnes() { - String pathTemplate = "something-{layer}\\{grid-something}-something{grid}\\-el/se{format}/something-{params}"; - FileManager fileManager = new FileManager(getRootDirectory(), pathTemplate, 250, 250); - File file = fileManager.getFile( - "parameters_id", new long[] {0, 0, 0}, "layer_name", "grid_set", "format_name", Collections.emptyMap()); - assertThat( - file.toString(), - is(buildRootFile( - "something-layer_name", - "null-somethinggrid_set", - "-el", - "seformat_name", - "something-parameters_id") - .toString())); - } - - private void checkFileExistsAndContainsTiles(Map> files, File file, long[]... expectedTiles) { - List tiles = files.get(file); - assertThat(tiles, notNullValue()); - assertThat(tiles.size(), is(expectedTiles.length)); - assertThat(tiles, containsInAnyOrder(expectedTiles)); - } - - @SafeVarargs - private final void genericTestPath( - String pathTemplate, - String expectedPath, - String parametersId, - long[] xyz, - String layerName, - String gridSetId, - String format, - Tuple... tuples) { - genericTestPath( - pathTemplate, expectedPath, parametersId, xyz, layerName, gridSetId, format, tuplesToMap(tuples)); - } - - private void genericTestPath( - String pathTemplate, - String expectedPath, - String parametersId, - long[] xyz, - String layerName, - String gridSetId, - String format, - Map parameters) { - FileManager fileManager = new FileManager(getRootDirectory(), pathTemplate, 1000, 1000); - File file = fileManager.getFile(parametersId, xyz, layerName, gridSetId, format, parameters); - assertThat(file.getPath(), is(new File(getRootDirectory(), expectedPath).getPath())); - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesBlobStoreConformanceTest.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesBlobStoreConformanceTest.java deleted file mode 100644 index 27b5a5673f..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesBlobStoreConformanceTest.java +++ /dev/null @@ -1,40 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Kevin Smith, Boundless, 2017 - */ -package org.geowebcache.sqlite; - -import org.geowebcache.storage.AbstractBlobStoreTest; -import org.junit.Rule; -import org.junit.rules.TemporaryFolder; - -public class MbtilesBlobStoreConformanceTest extends AbstractBlobStoreTest { - - @Rule - public TemporaryFolder temp = new TemporaryFolder(); - - @Override - public void createTestUnit() throws Exception { - this.store = new MbtilesBlobStore(getDefaultConfiguration()); - } - - protected MbtilesInfo getDefaultConfiguration() { - MbtilesInfo configuration = new MbtilesInfo(); - configuration.setPoolSize(1000); - configuration.setRootDirectory(temp.getRoot().getPath()); - configuration.setTemplatePath( - Utils.buildPath("{grid}", "{layer}", "{params}", "{format}", "{z}", "tiles-{x}-{y}.sqlite")); - configuration.setRowRangeCount(500); - configuration.setColumnRangeCount(500); - return configuration; - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesBlobStoreSuitabilityTest.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesBlobStoreSuitabilityTest.java deleted file mode 100644 index ccda7f4cb9..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesBlobStoreSuitabilityTest.java +++ /dev/null @@ -1,59 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Kevin Smith, Boundless, 2018 - */ -package org.geowebcache.sqlite; - -import static org.geowebcache.util.FileMatchers.directoryContaining; -import static org.geowebcache.util.FileMatchers.named; -import static org.hamcrest.Matchers.hasItem; - -import java.io.File; -import java.io.IOException; -import org.geowebcache.blobstore.file.FileBasedBlobStoreSuitabilityTest; -import org.geowebcache.storage.BlobStore; -import org.hamcrest.Matcher; -import org.junit.experimental.theories.DataPoints; - -public class MbtilesBlobStoreSuitabilityTest extends FileBasedBlobStoreSuitabilityTest { - - protected static File withMetadata(File dir) throws IOException { - new File(dir, "metadata.sqlite").createNewFile(); - return dir; - } - - @DataPoints - public static File[] persistenceLocations() throws Exception { - return new File[] { - newDir(), - emptyDir(), - withMetadata(emptyDir()), - dirWithFile(), - withMetadata(dirWithFile()), - dirWithDir(), - withMetadata(dirWithDir()) - }; - } - - @Override - @SuppressWarnings("unchecked") - protected Matcher existing() { - return directoryContaining((Matcher) hasItem(named("metadata.sqlite"))); - } - - @Override - public BlobStore create(Object dir) throws Exception { - MbtilesInfo info = new MbtilesInfo(); - info.setRootDirectory(((File) dir).getAbsolutePath()); - return new MbtilesBlobStore(info); - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesBlobStoreTest.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesBlobStoreTest.java deleted file mode 100644 index d760e4864a..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesBlobStoreTest.java +++ /dev/null @@ -1,337 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import static org.geowebcache.sqlite.Utils.Tuple.tuple; -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.containsInAnyOrder; -import static org.hamcrest.Matchers.greaterThan; -import static org.hamcrest.Matchers.is; -import static org.hamcrest.Matchers.notNullValue; -import static org.hamcrest.Matchers.nullValue; - -import java.io.File; -import java.util.ArrayList; -import java.util.Collections; -import java.util.List; -import org.geowebcache.mime.MimeType; -import org.geowebcache.storage.CompositeBlobStore; -import org.geowebcache.storage.SuitabilityCheckRule; -import org.geowebcache.storage.TileObject; -import org.geowebcache.storage.TileRange; -import org.junit.Rule; -import org.junit.Test; - -public final class MbtilesBlobStoreTest extends TestSupport { - - @Rule - public SuitabilityCheckRule suitability = - SuitabilityCheckRule.system(CompositeBlobStore.StoreSuitabilityCheck.NONE); - - @Test - public void testTilePutGetDeleteOperations() throws Exception { - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // create the tile that will be stored - TileObject putTile = TileObject.createCompleteTileObject( - "africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null, stringToResource("IMAGE-10-50-5")); - // storing the tile - store.put(putTile); - // create a query tile to get the previous stored tile - TileObject getTile = - TileObject.createQueryTileObject("africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null); - // checking if the tile was retrieved - assertThat(store.get(getTile), is(true)); - // checking if the blob data was updated - assertThat(getTile.getBlob(), notNullValue()); - assertThat(resourceToString(getTile.getBlob()), is("IMAGE-10-50-5")); - // sanity check of create time - assertThat(getTile.getCreated(), greaterThan(System.currentTimeMillis() - 60000)); - // delete and check if the tile was deleted - assertThat(store.delete(putTile), is(true)); - // query the deleted tile - getTile = TileObject.createQueryTileObject("africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null); - // check that the tile was deleted - assertThat(store.get(getTile), is(false)); - assertThat(getTile.getBlob(), nullValue()); - } - - @Test - public void testTileMetadataOperations() throws Exception { - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // let's store some metadata - store.putLayerMetadata("america", "background", "blue"); - store.putLayerMetadata("america", "style", "america-style"); - store.putLayerMetadata("australia", "background", "green"); - store.putLayerMetadata("australia", "style", "australia-style"); - // checking if the metadata was correctly stored - assertThat(store.getLayerMetadata("america", "background"), is("blue")); - assertThat(store.getLayerMetadata("america", "style"), is("america-style")); - assertThat(store.getLayerMetadata("australia", "background"), is("green")); - assertThat(store.getLayerMetadata("australia", "style"), is("australia-style")); - } - - @Test - public void testTileMetadataOperationsWithNull() throws Exception { - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // let's store some metadata with null values - store.putLayerMetadata(null, null, null); - store.putLayerMetadata("america", "style", null); - store.putLayerMetadata("australia", null, "green"); - // checking if the metadata was correctly stored - assertThat(store.getLayerMetadata("america", "style"), nullValue()); - assertThat(store.getLayerMetadata("australia", null), nullValue()); - assertThat(store.getLayerMetadata(null, null), nullValue()); - } - - @Test - public void testDeleteLayerOperation() throws Exception { - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // created database files for different layers - File asia1 = createFileInRootDir(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-0-500.sqlite")); - File asia2 = createFileInRootDir(Utils.buildPath("grid2", "asia", "image_png", "11", "tiles-100-500.sqlite")); - File asia3 = createFileInRootDir(Utils.buildPath("grid3", "asia", "image_png", "10", "tiles-0-500.sqlite")); - File asia4 = createFileInRootDir(Utils.buildPath("grid3", "asia", "image_jpeg", "10", "tiles-0-500.sqlite")); - File africa1 = createFileInRootDir(Utils.buildPath("grid1", "africa", "image_png", "10", "tiles-0-500.sqlite")); - File america1 = - createFileInRootDir(Utils.buildPath("grid1", "america", "image_png", "10", "tiles-0-500.sqlite")); - File america2 = - createFileInRootDir(Utils.buildPath("grid2", "america", "image_jpeg", "10", "tiles-0-500.sqlite")); - File europe1 = - createFileInRootDir(Utils.buildPath("grid1", "europe", "image_gif", "15", "tiles-4000-500.sqlite")); - // deleting layer asia and europe - store.delete("asia"); - store.delete("europe"); - assertThat(asia1.exists(), is(false)); - assertThat(asia2.exists(), is(false)); - assertThat(asia3.exists(), is(false)); - assertThat(asia4.exists(), is(false)); - assertThat(africa1.exists(), is(true)); - assertThat(america1.exists(), is(true)); - assertThat(america2.exists(), is(true)); - assertThat(europe1.exists(), is(false)); - } - - @Test - public void testDeleteLayerWithGridSetOperation() throws Exception { - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // created database files for different layers - File asia1 = createFileInRootDir(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-0-500.sqlite")); - File asia2 = createFileInRootDir(Utils.buildPath("grid2", "asia", "image_png", "11", "tiles-100-500.sqlite")); - File asia3 = createFileInRootDir(Utils.buildPath("grid3", "asia", "image_png", "10", "tiles-0-500.sqlite")); - File asia4 = createFileInRootDir(Utils.buildPath("grid3", "asia", "image_jpeg", "10", "tiles-0-500.sqlite")); - File africa1 = createFileInRootDir(Utils.buildPath("grid1", "africa", "image_png", "10", "tiles-0-500.sqlite")); - File america1 = - createFileInRootDir(Utils.buildPath("grid1", "america", "image_png", "10", "tiles-0-500.sqlite")); - File america2 = - createFileInRootDir(Utils.buildPath("grid2", "america", "image_jpeg", "10", "tiles-0-500.sqlite")); - File europe1 = - createFileInRootDir(Utils.buildPath("grid1", "europe", "image_gif", "15", "tiles-4000-500.sqlite")); - // deleting layer asia grid set grid3 and america grid set grid2 - store.deleteByGridsetId("asia", "grid3"); - store.deleteByGridsetId("america", "grid2"); - assertThat(asia1.exists(), is(true)); - assertThat(asia2.exists(), is(true)); - assertThat(asia3.exists(), is(false)); - assertThat(asia4.exists(), is(false)); - assertThat(africa1.exists(), is(true)); - assertThat(america1.exists(), is(true)); - assertThat(america2.exists(), is(false)); - assertThat(europe1.exists(), is(true)); - } - - @Test - public void testDeleteLayerWithTileRangeEager() throws Exception { - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - // activate eager mode - configuration.setEagerDelete(true); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // create the tile range - long[][] rangeBounds = { - {0, 490, 10, 500, 10}, - {800, 950, 1005, 1020, 11} - }; - TileRange tileRange = new TileRange( - "asia", "grid1", 10, 11, rangeBounds, MimeType.createFromExtension("png"), Collections.emptyMap()); - // created layer database files - File asia1 = createFileInRootDir(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-0-0.sqlite")); - File asia2 = createFileInRootDir(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-0-500.sqlite")); - File asia3 = createFileInRootDir(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-500-0.sqlite")); - File asia4 = createFileInRootDir(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-500-500.sqlite")); - File asia5 = createFileInRootDir(Utils.buildPath("grid1", "asia", "image_png", "11", "tiles-1000-1000.sqlite")); - // deleting tiles range - store.delete(tileRange); - assertThat(asia1.exists(), is(false)); - assertThat(asia2.exists(), is(false)); - assertThat(asia3.exists(), is(true)); - assertThat(asia4.exists(), is(true)); - assertThat(asia5.exists(), is(false)); - } - - @Test - public void testDeleteLayerWithTileRangeNoEager() throws Exception { - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // create the tile range - long[][] rangeBounds = { - {0, 10, 5, 15, 10}, - {980, 950, 1005, 1020, 11} - }; - TileRange tileRange = new TileRange( - "asia", "grid1", 10, 11, rangeBounds, MimeType.createFromExtension("png"), Collections.emptyMap()); - // created layer database files - File asia1 = buildRootFile(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-0-0.sqlite")); - File asia2 = buildRootFile(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-500-500.sqlite")); - // create some tiles - store.put(TileObject.createCompleteTileObject( - "asia", new long[] {3, 12, 10}, "grid1", "image/png", null, stringToResource("IMAGE-10-50-10"))); - store.put(TileObject.createCompleteTileObject( - "asia", new long[] {510, 550, 10}, "grid1", "image/png", null, stringToResource("IMAGE-510-550-10"))); - // deleting tiles range - store.delete(tileRange); - assertThat(asia1.exists(), is(true)); - assertThat(asia2.exists(), is(true)); - // check that the correct tiles were deleted - assertThat( - store.get( - TileObject.createQueryTileObject("asia", new long[] {10, 50, 10}, "grid1", "image/png", null)), - is(false)); - assertThat( - store.get(TileObject.createQueryTileObject( - "asia", new long[] {510, 550, 10}, "grid1", "image/png", null)), - is(true)); - } - - @Test - public void testLayerExistsOperation() throws Exception { - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // created layer database file - createFileInRootDir(Utils.buildPath("grid1", "europe", "image_png", "10", "tiles-0-0.sqlite")); - // checking if layer exists - assertThat(store.layerExists("europe"), is(true)); - assertThat(store.layerExists("asia"), is(false)); - } - - @Test - public void testRenameOperation() throws Exception { - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // create layers database files - File asia1 = createFileInRootDir(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-0-0.sqlite")); - File asia2 = createFileInRootDir(Utils.buildPath("grid1", "asia", "image_png", "10", "tiles-0-500.sqlite")); - File europe1 = createFileInRootDir(Utils.buildPath("grid1", "europe", "image_png", "10", "tiles-500-0.sqlite")); - // rename asia layers - store.rename("asia", "australia"); - assertThat(asia1.exists(), is(false)); - assertThat(asia2.exists(), is(false)); - assertThat(europe1.exists(), is(true)); - assertThat( - buildRootFile("grid1", "australia", "image_png", "10", "tiles-0-0.sqlite") - .exists(), - is(true)); - assertThat( - buildRootFile("grid1", "australia", "image_png", "10", "tiles-0-500.sqlite") - .exists(), - is(true)); - } - - @Test - // TODO: remove when upgrading to Hamcrest 2 - public void testOpeningDatabaseFileWithMbtilesMetadata() throws Exception { - // create and instantiate mbtiles metadata - File mbtilesMetadataDirectory = buildRootFile("mbtiles-metadata"); - File mbtilesMetadataFile = new File(mbtilesMetadataDirectory, "europe_asia.properties"); - String mbtilesMetadata = "attribution=some attribution" - + System.lineSeparator() - + "bounds=-180,-90,180,90" - + System.lineSeparator() - + "description=some description" - + System.lineSeparator() - + "maxZoom=10" - + System.lineSeparator() - + "minZoom=0" - + System.lineSeparator() - + "type=base_layer" - + System.lineSeparator() - + "version=1.0" - + System.lineSeparator(); - writeToFile(mbtilesMetadataFile, mbtilesMetadata); - // instantiating the store - MbtilesInfo configuration = getDefaultConfiguration(); - configuration.setMbtilesMetadataDirectory(mbtilesMetadataDirectory.getPath()); - SqliteConnectionManager connectionManager = new SqliteConnectionManager(configuration); - MbtilesBlobStore store = new MbtilesBlobStore(configuration, connectionManager); - addStoresToClean(store); - // create the tile that will be stored - TileObject putTile = TileObject.createCompleteTileObject( - "europe:asia", new long[] {5, 5, 5}, "EPSG:4326", "image/png", null, stringToResource("IMAGE-5-5-5")); - // storing the tile - store.put(putTile); - // checking if the file exists - File file = buildRootFile("EPSG_4326", "europe_asia", "image_png", "5", "tiles-0-0.sqlite"); - assertThat(file.exists(), is(true)); - // checking that the user provided metadata was properly inserted - connectionManager.executeQuery( - file, - resultSet -> { - // extract the metadata values from the result set - List> foundMetadata = new ArrayList<>(); - while (resultSet.next()) { - foundMetadata.add(tuple(resultSet.getString(1), resultSet.getString(2))); - } - // let's see if we have the expected metadata - assertThat(foundMetadata.size(), is(10)); - // the provided minZoom and maxZoom are ignored (geotools computes the real - // values) - assertThat( - foundMetadata, - containsInAnyOrder( - tuple("name", "europe:asia"), - tuple("version", "1.0"), - tuple("description", "some description"), - tuple("attribution", "some attribution"), - tuple("type", "base_layer"), - tuple("format", "png"), - tuple("bounds", "-180.0,-90.0,180.0,90.0"), - tuple("minzoom", "5"), - tuple("maxzoom", "5"), - tuple("json", "null"))); - return null; - }, - "SELECT name, value FROM metadata;"); - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesPBFTest.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesPBFTest.java deleted file mode 100644 index d8e9f2d82d..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/MbtilesPBFTest.java +++ /dev/null @@ -1,101 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Kevin Smith, Boundless, Copyright 2016 - */ -package org.geowebcache.sqlite; - -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.is; -import static org.junit.Assert.assertTrue; - -import java.io.File; -import java.io.FileOutputStream; -import java.io.InputStream; -import java.io.OutputStream; -import java.net.URL; -import org.apache.commons.io.IOUtils; -import org.geotools.mbtiles.MBTilesFileVectorTileTest; -import org.geowebcache.mime.ApplicationMime; -import org.geowebcache.storage.CompositeBlobStore; -import org.geowebcache.storage.SuitabilityCheckRule; -import org.geowebcache.storage.TileObject; -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; - -public class MbtilesPBFTest extends TestSupport { - File file; - String layer; - - @Rule - public SuitabilityCheckRule suitability = - SuitabilityCheckRule.system(CompositeBlobStore.StoreSuitabilityCheck.NONE); - - @Before - public void copyData() throws Exception { - file = buildRootFile("planet.mbtiles"); - URL template = org.geotools.mbtiles.MBTilesFileVectorTileTest.class.getResource("planet.mbtiles"); - try (InputStream in = template.openStream(); - OutputStream out = new FileOutputStream(file)) { - IOUtils.copy(in, out); - } - layer = "planet"; - } - - @Test - public void testGetTileDefaultUnzip() throws Exception { - MbtilesInfo configuration = getDefaultConfiguration(); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - TileObject tile = TileObject.createQueryTileObject( - layer, new long[] {0, 0, 0}, "EPSG:900913", ApplicationMime.mapboxVector.getFormat(), null); - assertThat(store.get(tile), is(true)); - try (InputStream is = MBTilesFileVectorTileTest.class.getResourceAsStream("tile_data.pbf.gz")) { - assertTrue(IOUtils.contentEquals(tile.getBlob().getInputStream(), is)); - } - } - - @Test - public void testGetTileDoUnzip() throws Exception { - MbtilesInfo configuration = getDefaultConfiguration(); - configuration.setGzipVector(true); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - TileObject tile = TileObject.createQueryTileObject( - layer, new long[] {0, 0, 0}, "EPSG:900913", ApplicationMime.mapboxVector.getFormat(), null); - assertThat(store.get(tile), is(true)); - try (InputStream is = MBTilesFileVectorTileTest.class.getResourceAsStream("tile_data.pbf")) { - assertTrue(IOUtils.contentEquals(tile.getBlob().getInputStream(), is)); - } - } - - @Test - public void testGetTileNoUnzip() throws Exception { - MbtilesInfo configuration = getDefaultConfiguration(); - configuration.setGzipVector(false); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - TileObject tile = TileObject.createQueryTileObject( - layer, new long[] {0, 0, 0}, "EPSG:900913", ApplicationMime.mapboxVector.getFormat(), null); - assertThat(store.get(tile), is(true)); - try (InputStream is = MBTilesFileVectorTileTest.class.getResourceAsStream("tile_data.pbf.gz")) { - assertTrue(IOUtils.contentEquals(tile.getBlob().getInputStream(), is)); - } - } - - @Override - protected MbtilesInfo getDefaultConfiguration() { - MbtilesInfo config = super.getDefaultConfiguration(); - config.setTemplatePath("{layer}.mbtiles"); - return config; - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/OperationsRestTest.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/OperationsRestTest.java deleted file mode 100644 index be4ad7ae00..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/OperationsRestTest.java +++ /dev/null @@ -1,274 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import static org.geowebcache.sqlite.Utils.Tuple.tuple; -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.is; -import static org.hamcrest.Matchers.notNullValue; -import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; - -import java.io.File; -import java.io.FileInputStream; -import java.io.FileOutputStream; -import java.io.IOException; -import java.nio.file.FileVisitResult; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.SimpleFileVisitor; -import java.nio.file.attribute.BasicFileAttributes; -import java.util.List; -import java.util.zip.ZipEntry; -import java.util.zip.ZipOutputStream; -import org.apache.commons.io.FileUtils; -import org.geowebcache.config.BlobStoreInfo; -import org.geowebcache.config.XMLConfiguration; -import org.geowebcache.sqlite.Utils.Tuple; -import org.geowebcache.storage.CompositeBlobStore; -import org.geowebcache.storage.SuitabilityCheckRule; -import org.geowebcache.storage.TileObject; -import org.junit.After; -import org.junit.Rule; -import org.junit.Test; -import org.junit.runner.RunWith; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.mock.web.MockMultipartFile; -import org.springframework.test.context.ActiveProfiles; -import org.springframework.test.context.ContextConfiguration; -import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; -import org.springframework.test.context.web.WebAppConfiguration; -import org.springframework.test.web.servlet.MockMvc; -import org.springframework.test.web.servlet.request.MockMvcRequestBuilders; -import org.springframework.test.web.servlet.setup.MockMvcBuilders; -import org.springframework.web.context.WebApplicationContext; - -@WebAppConfiguration -@ContextConfiguration(classes = OperationsRestWebConfig.class) -@RunWith(SpringJUnit4ClassRunner.class) -@ActiveProfiles("test") -public class OperationsRestTest extends TestSupport { - - @Autowired - private WebApplicationContext webApplicationContext; - - @Rule - public SuitabilityCheckRule suitability = - SuitabilityCheckRule.system(CompositeBlobStore.StoreSuitabilityCheck.NONE); - - @After - public void afterClass() throws Exception { - closeSqliteStoresConnections(); - super.afterTest(); - FileUtils.deleteQuietly(OperationsRestWebConfig.ROOT_DIRECTORY); - } - - @Test - public void testMultipleFilesUploadReplace() throws Exception { - // creates some database files for the tests - Tuple> testFiles = createTestFiles(); - File rootDirectory = testFiles.first; - try (FileInputStream fileA = new FileInputStream(new File(rootDirectory, testFiles.second.first)); - FileInputStream fileB = new FileInputStream(new File(rootDirectory, testFiles.second.second))) { - // perform the rest request - MockMultipartFile fileUploadA = new MockMultipartFile("file", fileA); - MockMultipartFile fileUploadB = new MockMultipartFile("file", fileB); - MockMvc mockMvc = - MockMvcBuilders.webAppContextSetup(webApplicationContext).build(); - // first request - mockMvc.perform(MockMvcRequestBuilders.multipart("/sqlite/replace") - .file(fileUploadA) - .param("layer", "europe") - .param("destination", testFiles.second.first)) - .andExpect(status().is(200)); - // second request - mockMvc.perform(MockMvcRequestBuilders.multipart("/sqlite/replace") - .file(fileUploadB) - .param("layer", "europe") - .param("destination", testFiles.second.second)) - .andExpect(status().is(200)); - // check that files were replaced - checkThatStoreContainsReplacedTiles(testFiles.second.first, testFiles.second.second); - } - } - - @Test - public void testZipFileUploadReplace() throws Exception { - // creates some database files for the tests - Tuple> testFiles = createTestFiles(); - File rootDirectory = testFiles.first; - // zip store files - File tempDirectory = Files.createTempDirectory("gwc-").toFile(); - addFilesToDelete(tempDirectory); - File zipFile = new File(tempDirectory, "replace.zip"); - zipDirectory(Path.of(rootDirectory.getPath()), zipFile); - try (FileInputStream zipFileInputStream = new FileInputStream(zipFile)) { - // perform the rest request - MockMultipartFile zipUpload = new MockMultipartFile("file", zipFileInputStream); - MockMvc mockMvc = - MockMvcBuilders.webAppContextSetup(webApplicationContext).build(); - // execute request - mockMvc.perform(MockMvcRequestBuilders.multipart("/sqlite/replace") - .file(zipUpload) - .param("layer", "europe")) - .andExpect(status().is(200)); - // check that files were replaced - checkThatStoreContainsReplacedTiles(testFiles.second.first, testFiles.second.second); - } - } - - @Test - public void testLocalDirectoryReplace() throws Exception { - // creates some database files for the tests - Tuple> testFiles = createTestFiles(); - File rootDirectory = testFiles.first; - // perform the rest request - MockMvc mockMvc = - MockMvcBuilders.webAppContextSetup(webApplicationContext).build(); - // execute request - mockMvc.perform(MockMvcRequestBuilders.multipart("/sqlite/replace") - .param("layer", "europe") - .param("source", rootDirectory.getPath())) - .andExpect(status().is(200)); - // check that files were replaced - checkThatStoreContainsReplacedTiles(testFiles.second.first, testFiles.second.second); - } - - private Tuple> createTestFiles() throws Exception { - // instantiating the stores - File rootDirectoryA = OperationsRestWebConfig.ROOT_DIRECTORY; - addFilesToDelete(rootDirectoryA); - if (Files.exists(OperationsRestWebConfig.ROOT_DIRECTORY.toPath())) { - Files.delete(OperationsRestWebConfig.ROOT_DIRECTORY.toPath()); - } - Files.createDirectories(OperationsRestWebConfig.ROOT_DIRECTORY.toPath()); - - File rootDirectoryB = Files.createTempDirectory("gwc-").toFile(); - addFilesToDelete(rootDirectoryB); - MbtilesInfo configurationA = new MbtilesInfo(); - configurationA.setRootDirectory(rootDirectoryA.getPath()); - configurationA.setTemplatePath(Utils.buildPath("{grid}", "{layer}", "{format}", "{z}", "tiles-{x}-{y}.sqlite")); - MbtilesInfo configurationB = new MbtilesInfo(); - configurationB.setRootDirectory(rootDirectoryB.getPath()); - configurationB.setTemplatePath(Utils.buildPath("{grid}", "{layer}", "{format}", "{z}", "tiles-{x}-{y}.sqlite")); - MbtilesBlobStore storeA = new MbtilesBlobStore(configurationA); - MbtilesBlobStore storeB = new MbtilesBlobStore(configurationB); - addStoresToClean(storeA, storeB); - // create the tiles that will be stored - TileObject putTileA = TileObject.createCompleteTileObject( - "africa", - new long[] {10, 50, 5}, - "EPSG:4326", - "image/png", - null, - TestSupport.stringToResource("IMAGE-10-50-5-A")); - TileObject putTileB = TileObject.createCompleteTileObject( - "africa", - new long[] {10, 50, 5}, - "EPSG:4326", - "image/png", - null, - TestSupport.stringToResource("IMAGE-10-50-5-B")); - TileObject putTileC = TileObject.createCompleteTileObject( - "africa", - new long[] {10, 5050, 15}, - "EPSG:4326", - "image/png", - null, - TestSupport.stringToResource("IMAGE-15-5050-5-B")); - // storing the tile - storeA.put(putTileA); - storeB.put(putTileB); - storeB.put(putTileC); - // make sure no sqlite file is in use to allow the move operation - storeA.clear(); - storeB.clear(); - // return relative paths - String relativePathA = Utils.buildPath("EPSG_4326", "africa", "image_png", "5", "tiles-0-0.sqlite"); - String relativePathB = Utils.buildPath("EPSG_4326", "africa", "image_png", "15", "tiles-0-5000.sqlite"); - return tuple(rootDirectoryB, tuple(relativePathA, relativePathB)); - } - - private void checkThatStoreContainsReplacedTiles(String relativePathA, String relativePathB) throws Exception { - // instantiating the store - File rootDirectory = OperationsRestWebConfig.ROOT_DIRECTORY; - MbtilesInfo configuration = new MbtilesInfo(); - configuration.setRootDirectory(rootDirectory.getPath()); - configuration.setTemplatePath(Utils.buildPath("{grid}", "{layer}", "{format}", "{z}", "tiles-{x}-{y}.sqlite")); - MbtilesBlobStore store = new MbtilesBlobStore(configuration); - addStoresToClean(store); - // checking that all the files are present - File fileA = new File(OperationsRestWebConfig.ROOT_DIRECTORY, relativePathA); - File fileB = new File(OperationsRestWebConfig.ROOT_DIRECTORY, relativePathB); - assertThat(fileA.exists(), is(true)); - assertThat(fileB.exists(), is(true)); - // let's query the store to see if we get the replaced tiles - TileObject getTile = - TileObject.createQueryTileObject("africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null); - assertThat(store.get(getTile), is(true)); - assertThat(getTile.getBlob(), notNullValue()); - assertThat(TestSupport.resourceToString(getTile.getBlob()), is("IMAGE-10-50-5-B")); - // let's query the second tile - getTile = TileObject.createQueryTileObject("africa", new long[] {10, 5050, 15}, "EPSG:4326", "image/png", null); - assertThat(store.get(getTile), is(true)); - assertThat(getTile.getBlob(), notNullValue()); - assertThat(TestSupport.resourceToString(getTile.getBlob()), is("IMAGE-15-5050-5-B")); - } - - private void zipDirectory(Path directoryToZip, File outputZipFile) throws IOException { - try (FileOutputStream fileOutputStream = new FileOutputStream(outputZipFile); - ZipOutputStream zipOutputStream = new ZipOutputStream(fileOutputStream)) { - Files.walkFileTree(directoryToZip, new SimpleFileVisitor<>() { - - @Override - public FileVisitResult visitFile(Path file, BasicFileAttributes fileAttributes) throws IOException { - zipOutputStream.putNextEntry( - new ZipEntry(directoryToZip.relativize(file).toString())); - Files.copy(file, zipOutputStream); - zipOutputStream.closeEntry(); - return FileVisitResult.CONTINUE; - } - - @Override - public FileVisitResult preVisitDirectory(Path directory, BasicFileAttributes attrs) throws IOException { - if (directory.equals(directoryToZip)) { - return FileVisitResult.CONTINUE; - } - // the zip structure is not tied the OS file separator - zipOutputStream.putNextEntry( - new ZipEntry(directoryToZip.relativize(directory).toString() + "/")); - zipOutputStream.closeEntry(); - return FileVisitResult.CONTINUE; - } - }); - } - } - - /** - * Helper method that simply makes sure that all the connections to the sqlite databases are closed allowing file - * operations to be performed on the databases files. - */ - private void closeSqliteStoresConnections() { - XMLConfiguration configuration = webApplicationContext.getBean(XMLConfiguration.class); - assertThat(configuration, notNullValue()); - List blobStoresConfig = configuration.getBlobStores(); - assertThat(blobStoresConfig, notNullValue()); - // let's iterate over all the available blob stores configurations - for (BlobStoreInfo blobStoreConfig : blobStoresConfig) { - if (blobStoreConfig instanceof SqliteInfo) { - // we have a sqlite based blob store, let's close all the connections - ((MbtilesInfo) blobStoreConfig).getConnectionManager().reapAllConnections(); - } - } - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/OperationsRestWebConfig.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/OperationsRestWebConfig.java deleted file mode 100644 index cf377d1155..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/OperationsRestWebConfig.java +++ /dev/null @@ -1,84 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.when; - -import java.io.File; -import java.nio.file.Files; -import java.util.ArrayList; -import java.util.List; -import org.geowebcache.GeoWebCacheException; -import org.geowebcache.config.BlobStoreInfo; -import org.geowebcache.config.XMLConfiguration; -import org.geowebcache.grid.GridSetBroker; -import org.geowebcache.layer.TileLayer; -import org.geowebcache.layer.TileLayerDispatcher; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.ComponentScan; -import org.springframework.context.annotation.Configuration; -import org.springframework.context.annotation.Profile; -import org.springframework.web.multipart.MultipartResolver; -import org.springframework.web.multipart.commons.CommonsMultipartResolver; -import org.springframework.web.servlet.config.annotation.EnableWebMvc; -import org.springframework.web.servlet.config.annotation.WebMvcConfigurationSupport; - -@Configuration -@ComponentScan({"org.geowebcache.sqlite"}) -@EnableWebMvc -@Profile("test") -public class OperationsRestWebConfig extends WebMvcConfigurationSupport { - - static File ROOT_DIRECTORY; - - static { - try { - ROOT_DIRECTORY = Files.createTempDirectory("gwc-").toFile(); - } catch (Exception exception) { - throw Utils.exception(exception, "Error creating temporary directory."); - } - } - - @Bean - public MultipartResolver multipartResolver() { - CommonsMultipartResolver multipartResolver = new CommonsMultipartResolver(); - return multipartResolver; - } - - @Bean - public TileLayerDispatcher tileLayerDispatcher() throws GeoWebCacheException { - TileLayerDispatcher tileLayerDispatcher = mock(TileLayerDispatcher.class); - TileLayer tileLayer = mock(TileLayer.class); - when(tileLayer.getBlobStoreId()).thenReturn("mbtiles-store"); - when(tileLayerDispatcher.getTileLayer("europe")).thenReturn(tileLayer); - return tileLayerDispatcher; - } - - @Bean - public XMLConfiguration xmlConfiguration() { - XMLConfiguration gwcConfiguration = mock(XMLConfiguration.class); - List mbtilesStoreConfig = new ArrayList<>(); - MbtilesInfo configuration = new MbtilesInfo("mbtiles-store"); - configuration.setRootDirectory(ROOT_DIRECTORY.getPath()); - mbtilesStoreConfig.add(configuration); - when(gwcConfiguration.getBlobStores()).thenReturn(mbtilesStoreConfig); - return gwcConfiguration; - } - - @Bean - public GridSetBroker gwcGridSetBroker() { - return mock(GridSetBroker.class); - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/SqliteBlobStoreTest.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/SqliteBlobStoreTest.java deleted file mode 100644 index cb86ae8acf..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/SqliteBlobStoreTest.java +++ /dev/null @@ -1,129 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.is; -import static org.hamcrest.Matchers.notNullValue; - -import java.io.File; -import java.nio.file.Files; -import org.apache.commons.io.FileUtils; -import org.geowebcache.storage.TileObject; -import org.junit.Test; - -public final class SqliteBlobStoreTest extends TestSupport { - - @Test - public void testFileReplaceOperation() throws Exception { - // instantiating the stores - File rootDirectoryA = Files.createTempDirectory("gwc-").toFile(); - File rootDirectoryB = Files.createTempDirectory("gwc-").toFile(); - addFilesToDelete(rootDirectoryA, rootDirectoryB); - MbtilesInfo configurationA = getDefaultConfiguration(); - configurationA.setRootDirectory(rootDirectoryA.getPath()); - MbtilesInfo configurationB = getDefaultConfiguration(); - configurationB.setRootDirectory(rootDirectoryB.getPath()); - MbtilesBlobStore storeA = new MbtilesBlobStore(configurationA); - MbtilesBlobStore storeB = new MbtilesBlobStore(configurationB); - addStoresToClean(storeA, storeB); - // create the tiles that will be stored - TileObject putTileA = TileObject.createCompleteTileObject( - "africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null, stringToResource("IMAGE-10-50-5-A")); - TileObject putTileB = TileObject.createCompleteTileObject( - "africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null, stringToResource("IMAGE-10-50-5-B")); - // storing the tile - storeA.put(putTileA); - storeB.put(putTileB); - // make sure connections to the created stores a re closed - storeA.clear(); - storeB.clear(); - // check that database files exists - String relativePath = Utils.buildPath("EPSG_4326", "africa", "image_png", "5", "tiles-0-0.sqlite"); - File fileA = new File(rootDirectoryA, relativePath); - File fileB = new File(rootDirectoryB, relativePath); - assertThat(fileA.exists(), is(true)); - assertThat(fileB.exists(), is(true)); - // replace store A database file with store B database file - storeA.replace(fileB, relativePath); - // let's query store A to see if we get store B tile - TileObject getTile = - TileObject.createQueryTileObject("africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null); - assertThat(storeA.get(getTile), is(true)); - assertThat(getTile.getBlob(), notNullValue()); - assertThat(resourceToString(getTile.getBlob()), is("IMAGE-10-50-5-B")); - // clean up - FileUtils.deleteQuietly(rootDirectoryA); - FileUtils.deleteQuietly(rootDirectoryB); - } - - @Test - public void testDirectoryReplaceOperation() throws Exception { - // instantiating the stores - File rootDirectoryA = Files.createTempDirectory("replace-tests-a-").toFile(); - File rootDirectoryB = Files.createTempDirectory("replace-tests-a-").toFile(); - addFilesToDelete(rootDirectoryA, rootDirectoryB); - MbtilesInfo configurationA = getDefaultConfiguration(); - configurationA.setRootDirectory(rootDirectoryA.getPath()); - MbtilesInfo configurationB = getDefaultConfiguration(); - configurationB.setRootDirectory(rootDirectoryB.getPath()); - MbtilesBlobStore storeA = new MbtilesBlobStore(configurationA); - MbtilesBlobStore storeB = new MbtilesBlobStore(configurationB); - addStoresToClean(storeA, storeB); - // create the tiles that will be stored - TileObject putTileA = TileObject.createCompleteTileObject( - "africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null, stringToResource("IMAGE-10-50-5-A")); - TileObject putTileB = TileObject.createCompleteTileObject( - "africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null, stringToResource("IMAGE-10-50-5-B")); - TileObject putTileC = TileObject.createCompleteTileObject( - "africa", - new long[] {10, 5050, 15}, - "EPSG:4326", - "image/png", - null, - stringToResource("IMAGE-15-5050-5-B")); - // storing the tile - storeA.put(putTileA); - storeB.put(putTileB); - storeB.put(putTileC); - // make sure connections to the created stores a re closed - storeA.clear(); - storeB.clear(); - // check that database files exists - String relativePathA = Utils.buildPath("EPSG_4326", "africa", "image_png", "5", "tiles-0-0.sqlite"); - String relativePathB = Utils.buildPath("EPSG_4326", "africa", "image_png", "15", "tiles-0-5000.sqlite"); - File fileA = new File(rootDirectoryA, relativePathA); - File fileB = new File(rootDirectoryB, relativePathA); - File fileC = new File(rootDirectoryB, relativePathB); - assertThat(fileA.exists(), is(true)); - assertThat(fileB.exists(), is(true)); - assertThat(fileC.exists(), is(true)); - // replace store A database files with store B database files - storeA.replace(rootDirectoryB); - // let's query store A to see if we get store B tiles - TileObject getTile = - TileObject.createQueryTileObject("africa", new long[] {10, 50, 5}, "EPSG:4326", "image/png", null); - assertThat(storeA.get(getTile), is(true)); - assertThat(getTile.getBlob(), notNullValue()); - assertThat(resourceToString(getTile.getBlob()), is("IMAGE-10-50-5-B")); - // let's query the second tile - getTile = TileObject.createQueryTileObject("africa", new long[] {10, 5050, 15}, "EPSG:4326", "image/png", null); - assertThat(storeA.get(getTile), is(true)); - assertThat(getTile.getBlob(), notNullValue()); - assertThat(resourceToString(getTile.getBlob()), is("IMAGE-15-5050-5-B")); - // clean up - FileUtils.deleteQuietly(rootDirectoryA); - FileUtils.deleteQuietly(rootDirectoryB); - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/SqliteConnectionManagerTest.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/SqliteConnectionManagerTest.java deleted file mode 100644 index a2d96ee631..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/SqliteConnectionManagerTest.java +++ /dev/null @@ -1,241 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import static org.geowebcache.sqlite.Utils.Tuple; -import static org.geowebcache.sqlite.Utils.Tuple.tuple; -import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.is; -import static org.hamcrest.Matchers.notNullValue; -import static org.hamcrest.Matchers.nullValue; - -import com.google.errorprone.annotations.FormatMethod; -import java.io.File; -import java.sql.Connection; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.util.ArrayList; -import java.util.List; -import java.util.Random; -import java.util.UUID; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.Future; -import java.util.concurrent.TimeUnit; -import java.util.logging.Level; -import java.util.logging.Logger; -import org.geotools.util.logging.Logging; -import org.geowebcache.storage.StorageException; -import org.junit.After; -import org.junit.Before; -import org.junit.Ignore; -import org.junit.Test; - -public final class SqliteConnectionManagerTest extends TestSupport { - - private static Logger LOGGER = Logging.getLogger(SqliteConnectionManagerTest.class.getName()); - - private List connectionManagersToClean; - - @Override - @Before - public void beforeTest() throws Exception { - super.beforeTest(); - connectionManagersToClean = new ArrayList<>(); - } - - @Override - @After - public void afterTest() throws Exception { - for (SqliteConnectionManager connectionManager : connectionManagersToClean) { - try { - connectionManager.reapAllConnections(); - connectionManager.stopPoolReaper(); - } catch (Exception exception) { - // nothing that we can do, so just ignoring this exception - } - } - super.afterTest(); - } - - @Test - public void testGetConnection() throws StorageException { - SqliteConnectionManager connectionManager = new SqliteConnectionManager(Integer.MAX_VALUE, 1000); - connectionManagersToClean.add(connectionManager); - connectionManager.doWork(buildRootFile("tiles", "data_base.sqlite"), true, connection -> { - insertInTestTable(connection, "name", "europe"); - }); - connectionManager.reapAllConnections(); - assertThat(connectionManager.getPool().size(), is(0)); - connectionManager.doWork(buildRootFile("tiles", "data_base.sqlite"), true, connection -> { - String value = getFromTestTable(connection, "name"); - assertThat(value, notNullValue()); - assertThat(value, is("europe")); - closeConnectionQuietly(connection); - }); - } - - @Test - @Ignore - public void testMultiThreadsWithSingleFile() throws Exception { - genericMultiThreadsTest(10, 500, Integer.MAX_VALUE, buildRootFile("data_base_a.sqlite")); - } - - @Test - @Ignore - public void testMultiThreadsWithMultipleFiles() throws Exception { - genericMultiThreadsTest( - 10, - 500, - 10, - buildRootFile("data_base_a.sqlite"), - buildRootFile("data_base_b.sqlite"), - buildRootFile("data_base_c.sqlite"), - buildRootFile("data_base_d.sqlite"), - buildRootFile("data_base_e.sqlite")); - } - - @Test - @Ignore - public void testMultiThreadsWithMultipleFilesWithCacheLimit() throws Exception { - genericMultiThreadsTest( - 10, - 500, - 1, - buildRootFile("data_base_a.sqlite"), - buildRootFile("data_base_b.sqlite"), - buildRootFile("data_base_c.sqlite"), - buildRootFile("data_base_d.sqlite"), - buildRootFile("data_base_e.sqlite")); - } - - @Test - @Ignore - public void testReplaceOperation() throws Exception { - SqliteConnectionManager connectionManager = new SqliteConnectionManager(Integer.MAX_VALUE, 1000); - connectionManagersToClean.add(connectionManager); - File file1 = buildRootFile("tiles", "data_base_1.sqlite"); - Utils.createFileParents(file1); - File file2 = buildRootFile("tiles", "data_base_2.sqlite"); - Utils.createFileParents(file2); - connectionManager.doWork(file1, false, connection -> { - insertInTestTable(connection, "name", "europe"); - closeConnectionQuietly(connection); - }); - file2.createNewFile(); - connectionManager.replace(file1, file2); - connectionManager.doWork(file1, false, connection -> { - createTestTable(connection); - String value = getFromTestTable(connection, "name"); - assertThat(value, nullValue()); - closeConnectionQuietly(connection); - }); - } - - private void genericMultiThreadsTest(int threadsNumber, int workersNumber, long poolSize, File... files) - throws Exception { - SqliteConnectionManager connectionManager = new SqliteConnectionManager(poolSize, 10); - connectionManagersToClean.add(connectionManager); - ExecutorService executor = Executors.newFixedThreadPool(threadsNumber); - Random random = new Random(); - List>> results = new ArrayList<>(); - for (int i = 0; i < workersNumber; i++) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Submitted worker '%d'\\'%d'.".formatted(i, workersNumber)); - } - executor.submit(() -> { - File file = files[random.nextInt(files.length)]; - String key = UUID.randomUUID().toString(); - return connectionManager.doWork(file, false, connection -> { - insertInTestTable(connection, key, "value-" + key); - closeConnectionQuietly(connection); - return tuple(file, key); - }); - }); - } - executor.shutdown(); - executor.awaitTermination(60, TimeUnit.SECONDS); - connectionManager.reapAllConnections(); - assertThat(connectionManager.getPool().size(), is(0)); - for (Future> result : results) { - File file = result.get().first; - String key = result.get().second; - connectionManager.doWork(file, true, connection -> { - String value = getFromTestTable(connection, key); - assertThat(value, notNullValue()); - assertThat(value, is("value-" + key)); - closeConnectionQuietly(connection); - }); - } - } - - private static void closeConnectionQuietly(Connection connection) { - try { - connection.close(); - } catch (Exception exception) { - throw Utils.exception(exception, "Error closing connection."); - } - } - - private static void createTestTable(Connection connection) { - execute( - connection, - "CREATE TABLE IF NOT EXISTS test " - + "(key text, value text, CONSTRAINT pk_metadata PRIMARY KEY(key));"); - } - - private static void insertInTestTable(Connection connection, String key, String value) { - createTestTable(connection); - execute(connection, "INSERT INTO test VALUES ('%s', '%s');", key, value); - } - - private static String getFromTestTable(Connection connection, String key) { - return new ExecuteQuery(connection, "SELECT value FROM test WHERE key = '%s' ORDER BY key;", key) { - - String result; - - @Override - public void extract(ResultSet resultSet) throws Exception { - if (resultSet.next()) { - result = resultSet.getString(1); - } - } - }.result; - } - - @FormatMethod - private static void execute(Connection connection, String sql, Object... arguments) { - String finalSql = sql.formatted(arguments); - try (PreparedStatement statement = connection.prepareStatement(finalSql)) { - statement.execute(); - } catch (Exception exception) { - throw Utils.exception(exception, "Error executing SQL '%s'.", finalSql); - } - } - - private abstract static class ExecuteQuery { - - public abstract void extract(ResultSet resultSet) throws Exception; - - @FormatMethod - public ExecuteQuery(Connection connection, String query, Object... arguments) { - String finalQuery = query.formatted(arguments); - try (PreparedStatement statement = connection.prepareStatement(finalQuery)) { - extract(statement.executeQuery()); - } catch (Exception exception) { - throw Utils.exception(exception, "Error executing query '%s'.", finalQuery); - } - } - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/SqlitlePerf.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/SqlitlePerf.java deleted file mode 100644 index 9680e3e22a..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/SqlitlePerf.java +++ /dev/null @@ -1,408 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.io.File; -import java.nio.file.Files; -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.util.Random; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.TimeUnit; -import java.util.logging.Level; -import java.util.logging.Logger; -import org.apache.commons.io.FileUtils; -import org.geotools.util.logging.Logging; -import org.geowebcache.storage.BlobStore; -import org.geowebcache.storage.TileObject; -import org.geowebcache.storage.blobstore.file.FileBlobStore; - -/** - * Measures the performance of the {@link SqliteConnectionManager} and the {@link MbtilesBlobStore} against using a raw - * query. - */ -final class SqlitlePerf { - - private static Logger LOGGER = Logging.getLogger(SqlitlePerf.class.getName()); - - // number of workers that will be used to perform the selects - static final int WORKERS = 10; - // number of tiles to store and retrieve - static final int TILES = 1000000; - - public static void main(String[] args) throws Exception { - // initiate sqlite drive - Class.forName("org.sqlite.JDBC"); - // create the directory that will contain all created files - File rootDirectory = Files.createTempDirectory("gwc-").toFile(); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Root directory '%s'.".formatted(rootDirectory)); - } - // seeding file system - long[][] tiles = new long[TILES][3]; - File seedDirectory = seedFileSystem(rootDirectory, tiles); - fileStore(seedDirectory, tiles); - FileUtils.deleteDirectory(seedDirectory); - // seeding a database with some random tiles - File seedFile = createSeedFile(rootDirectory, tiles); - // select tiles using a raw connection - rawSqlitle(rootDirectory, seedFile, tiles); - // select tiles using the connection manager - pooledSqlitle(rootDirectory, seedFile, tiles); - // select tiles using the mbtiles blobstore - mbtilesStore(rootDirectory, seedFile, tiles); - // cleaning everything - FileUtils.deleteDirectory(rootDirectory); - } - - /** Select the created tiles using a raw connection. */ - private static void rawSqlitle(File rootDirectory, File seedFile, long[][] tiles) throws Exception { - // creating a new database by copying the seeded one - File databaseFile = new File(rootDirectory, "raw_perf_test.sqlite"); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Start raw select from file '%s'.".formatted(databaseFile)); - } - FileUtils.copyFile(seedFile, databaseFile); - // submitting the select tasks - ExecutorService executor = Executors.newFixedThreadPool(WORKERS); - long startTime = System.currentTimeMillis(); - try (Connection connection = DriverManager.getConnection("jdbc:sqlite:" + seedFile.getPath())) { - for (int i = 0; i < tiles.length; i++) { - long[] tile = tiles[i]; - executor.submit((Runnable) () -> getTile(connection, tile)); - if (i != 0 && i % 10000 == 0) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Submitted %d select tasks.".formatted(i)); - } - } - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Submitted %d select tasks.".formatted(TILES)); - } - // lets wait for the workers to finish - executor.shutdown(); - executor.awaitTermination(5, TimeUnit.MINUTES); - // computing some stats - long endTime = System.currentTimeMillis(); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Tiles raw select time '%d'.".formatted(endTime - startTime)); - } - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info( - "Tiles raw selected per second '%f'.".formatted(TILES / (float) (endTime - startTime) * 1000)); - } - } - FileUtils.deleteQuietly(databaseFile); - } - - /** Select the created tiles using a connection provide by the connection manager. */ - private static void pooledSqlitle(File rootDirectory, File seedFile, long[][] tiles) throws Exception { - // creating a new database by copying the seeded one - File databaseFile = new File(rootDirectory, "pooled_perf_test.sqlite"); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Start pooled select from file '%s'.".formatted(databaseFile)); - } - FileUtils.copyFile(seedFile, databaseFile); - // submitting the select tasks - ExecutorService executor = Executors.newFixedThreadPool(WORKERS); - SqliteConnectionManager connectionManager = new SqliteConnectionManager(10, 2000); - long startTime = System.currentTimeMillis(); - for (int i = 0; i < tiles.length; i++) { - long[] tile = tiles[i]; - executor.submit(() -> connectionManager.doWork(databaseFile, true, connection -> { - getTile(connection, tile); - })); - if (i != 0 && i % 10000 == 0) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Submitted %d select tasks.".formatted(i)); - } - } - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Submitted %d select tasks.".formatted(TILES)); - } - // lets wait for the workers to finish - executor.shutdown(); - executor.awaitTermination(5, TimeUnit.MINUTES); - // computing some stats - long endTime = System.currentTimeMillis(); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Tiles pooled select time '%d'.".formatted(endTime - startTime)); - } - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info( - "Tiles pooled selected per second '%f'.".formatted(TILES / (float) (endTime - startTime) * 1000)); - } - // clean everything - connectionManager.reapAllConnections(); - connectionManager.stopPoolReaper(); - FileUtils.deleteQuietly(databaseFile); - } - - /** Retrieve the created tiles using the mbtiles blobstore. */ - private static void mbtilesStore(File rootDirectory, File seedFile, long[][] tiles) throws Exception { - // creating a new database by copying the seeded one - File databaseFile = - new File(rootDirectory, Utils.buildPath("grid", "layer", "image_png", "mbtiles_perf_test.sqlite")); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Start mbtiles select from file '%s'.".formatted(databaseFile)); - } - FileUtils.copyFile(seedFile, databaseFile); - // submitting the select tasks - ExecutorService executor = Executors.newFixedThreadPool(WORKERS); - long startTime = System.currentTimeMillis(); - // mbtiles store configuration - MbtilesInfo configuration = new MbtilesInfo(); - configuration.setRootDirectory(rootDirectory.getPath()); - configuration.setTemplatePath(Utils.buildPath("{grid}", "{layer}", "{format}", "mbtiles_perf_test.sqlite")); - configuration.setUseCreateTime(false); - // instantiate the mbtiles blobstore - SqliteConnectionManager connectionManager = new SqliteConnectionManager(10, 2000); - MbtilesBlobStore mbtilesBlobStore = new MbtilesBlobStore(configuration, connectionManager); - for (int i = 0; i < tiles.length; i++) { - long[] tile = tiles[i]; - executor.submit(() -> { - TileObject mbtile = TileObject.createQueryTileObject("layer", tile, "grid", "image/png", null); - try { - mbtilesBlobStore.get(mbtile); - } catch (Exception exception) { - throw Utils.exception(exception, "Error retrieving tile '%s'.", mbtile); - } - }); - if (i != 0 && i % 10000 == 0) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Submitted %d select tasks.".formatted(i)); - } - } - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Submitted %d select tasks.".formatted(TILES)); - } - // lets wait for the workers to finish - executor.shutdown(); - executor.awaitTermination(5, TimeUnit.MINUTES); - // computing some stats - long endTime = System.currentTimeMillis(); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Tiles mbtiles blobstore select time '%d'.".formatted(endTime - startTime)); - } - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Tiles mbtiles blobstore selected per second '%f'." - .formatted(TILES / (float) (endTime - startTime) * 1000)); - } - // clean everything - connectionManager.reapAllConnections(); - connectionManager.stopPoolReaper(); - FileUtils.deleteQuietly(databaseFile); - } - - /** Retrieve the created tiles using the file blobstore. */ - private static void fileStore(File seedDirectory, long[][] tiles) throws Exception { - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Start reading from directory'%s'.".formatted(seedDirectory)); - } - // submitting the read tasks - ExecutorService executor = Executors.newFixedThreadPool(WORKERS); - long startTime = System.currentTimeMillis(); - // instantiate the file blobstore - BlobStore fileBlobStore = new FileBlobStore(seedDirectory.getPath()); - for (int i = 0; i < tiles.length; i++) { - long[] tile = tiles[i]; - executor.submit(() -> { - TileObject mbtile = TileObject.createQueryTileObject("layer", tile, "grid", "image/png", null); - try { - fileBlobStore.get(mbtile); - } catch (Exception exception) { - throw Utils.exception(exception, "Error retrieving tile '%s'.", mbtile); - } - }); - if (i != 0 && i % 10000 == 0) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Submitted %d read tasks.".formatted(i)); - } - } - } - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Submitted %d read tasks.".formatted(TILES)); - } - // lets wait for the workers to finish - executor.shutdown(); - executor.awaitTermination(5, TimeUnit.MINUTES); - // computing some stats - long endTime = System.currentTimeMillis(); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Tiles file blobstore read time '%d'.".formatted(endTime - startTime)); - } - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Tiles file blobstore reads per second '%f'." - .formatted(TILES / (float) (endTime - startTime) * 1000)); - } - } - - /** Store random tiles in the filesystem. */ - private static File seedFileSystem(File rootDirectory, long[][] tiles) throws Exception { - // creating the root directory where tiles will be saved - File seedDirectory = new File(rootDirectory, "tiles"); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Start seeding file system '%s'.".formatted(seedDirectory)); - } - BlobStore fileBlobStore = new FileBlobStore(seedDirectory.getPath()); - // start seeding the tiles - long startTime = System.currentTimeMillis(); - for (int i = 0; i < TILES; i++) { - Tile tile = Tile.random(); - tiles[i][0] = tile.x; - tiles[i][1] = tile.y; - tiles[i][2] = tile.z; - fileBlobStore.put(TileObject.createCompleteTileObject( - "layer", - new long[] {tile.x, tile.y, tile.z}, - "epsg:4326", - "image/png", - null, - Utils.byteArrayToResource(tile.data))); - if (i != 0 && i % 10000 == 0) { - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Stored %d tiles.".formatted(i)); - } - } - } - long endTime = System.currentTimeMillis(); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Insert time '%d' (batch mode).".formatted(endTime - startTime)); - } - return seedDirectory; - } - - /** Seeding a file with random tiles. */ - private static File createSeedFile(File rootDirectory, long[][] tiles) throws Exception { - // creating the database that will be seeded - File seedFile = new File(rootDirectory, "seed_perf_test.sqlite"); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Start seeding file '%s'.".formatted(seedFile)); - } - try (Connection connection = DriverManager.getConnection("jdbc:sqlite:" + seedFile.getPath())) { - String createTableSql = "CREATE TABLE IF NOT EXISTS tiles (zoom_level integer, tile_column integer, " - + "tile_row integer, tile_data blob, CONSTRAINT pk_tiles PRIMARY KEY(zoom_level, tile_column,tile_row));"; - executeSql(connection, createTableSql); - // start seeding wrapped in a transaction (improves performance) - long startTime = System.currentTimeMillis(); - executeSql(connection, "BEGIN TRANSACTION;"); - String sql = "INSERT OR REPLACE INTO tiles VALUES(?, ?, ?, ?);"; - // insert the tiles in batches - try (PreparedStatement statement = connection.prepareStatement(sql)) { - for (int i = 0; i < TILES; i++) { - Tile tile = Tile.random(); - tiles[i][0] = tile.x; - tiles[i][1] = tile.y; - tiles[i][2] = tile.z; - statement.setLong(1, tile.z); - statement.setLong(2, tile.x); - statement.setLong(3, tile.y); - statement.setBytes(4, tile.data); - statement.addBatch(); - if (i != 0 && i % 10000 == 0) { - statement.executeBatch(); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Inserted batch %d.".formatted(i)); - } - } - } - statement.executeBatch(); - if (LOGGER.isLoggable(Level.FINE)) { - LOGGER.fine("Inserted batch %d.".formatted(TILES)); - } - } catch (Exception exception) { - throw Utils.exception(exception, "Error executing SQL '%s'.", sql); - } - // clean everything - executeSql(connection, "END TRANSACTION;"); - long endTime = System.currentTimeMillis(); - if (LOGGER.isLoggable(Level.INFO)) { - LOGGER.info("Insert time '%d' (batch mode).".formatted(endTime - startTime)); - } - } - return seedFile; - } - - /** Helper method that fetches a tile form the database using the provided connection. */ - private static byte[] getTile(Connection connection, long[] xyz) { - String sql = "SELECT tile_data FROM tiles WHERE zoom_level = ? AND tile_column = ? AND tile_row = ?;"; - try (PreparedStatement statement = connection.prepareStatement(sql)) { - statement.setLong(1, xyz[2]); - statement.setLong(2, xyz[0]); - statement.setLong(3, xyz[1]); - try (ResultSet resultSet = statement.executeQuery()) { - if (resultSet.next()) { - // the tile exists - byte[] data = resultSet.getBytes(1); - if (data.length != 2024) { - // the data doesn't have the expected size - if (LOGGER.isLoggable(Level.SEVERE)) { - LOGGER.log( - Level.SEVERE, "Tile %d-%d-%d data is not valid.".formatted(xyz[2], xyz[0], xyz[1])); - } - } - // the tile data looks good - return data; - } else { - // the tile was not found - if (LOGGER.isLoggable(Level.SEVERE)) { - LOGGER.log(Level.SEVERE, "Failed to load tile %d-%d-%d.".formatted(xyz[2], xyz[0], xyz[1])); - } - return null; - } - } - } catch (Exception exception) { - throw Utils.exception(exception, "Error executing SQL '%s'.", sql); - } - } - - /** Helper method that executes an SQL statement using the provided connection. */ - private static void executeSql(Connection connection, String sql) { - try (PreparedStatement statement = connection.prepareStatement(sql)) { - statement.execute(); - } catch (Exception exception) { - throw Utils.exception(exception, "Error executing SQL '%s'.", sql); - } - } - - /** Helper class that stores a tile information. */ - private static final class Tile { - - private static final Random random = new Random(); - - final long x; - final long y; - final long z; - final byte[] data; - - private Tile(long x, long y, long z, byte[] data) { - this.x = x; - this.y = y; - this.z = z; - this.data = data; - } - - /** Creates a random tile. */ - static Tile random() { - byte[] data = new byte[2024]; - random.nextBytes(data); - return new Tile(random.nextInt(1000000), random.nextInt(1000000), random.nextInt(10), data); - } - } -} diff --git a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/TestSupport.java b/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/TestSupport.java deleted file mode 100644 index 86e4ab5f9e..0000000000 --- a/geowebcache/sqlite/src/test/java/org/geowebcache/sqlite/TestSupport.java +++ /dev/null @@ -1,133 +0,0 @@ -/** - * This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General - * Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any - * later version. - * - *

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied - * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - * - *

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see - * . - * - * @author Nuno Oliveira, GeoSolutions S.A.S., Copyright 2016 - */ -package org.geowebcache.sqlite; - -import java.io.File; -import java.io.FileWriter; -import java.nio.file.Files; -import java.util.ArrayList; -import java.util.List; -import org.apache.commons.io.FileUtils; -import org.geowebcache.io.ByteArrayResource; -import org.geowebcache.io.Resource; -import org.junit.After; -import org.junit.Before; - -/** Helper class with some utilities methods for using during the tests. */ -public abstract class TestSupport { - - private File rootDirectory; - - private List filesToDelete; - private List storesToClean; - - @Before - public void beforeTest() throws Exception { - rootDirectory = Files.createTempDirectory("gwc-").toFile(); - filesToDelete = new ArrayList<>(); - filesToDelete.add(rootDirectory); - storesToClean = new ArrayList<>(); - } - - @After - public void afterTest() throws Exception { - // let's first make sure all the stores connection were removed - for (MbtilesBlobStore store : storesToClean) { - try { - store.destroy(); - } catch (Exception exception) { - // there is nothing we can do - } - } - // deleting all the files to delete - for (File file : filesToDelete) { - if (file.exists()) { - FileUtils.forceDelete(file); - } - } - } - - protected File getRootDirectory() { - return rootDirectory; - } - - protected String getRootDirectoryPath() { - try { - return rootDirectory.getCanonicalPath(); - } catch (Exception exception) { - throw Utils.exception(exception, "Error getting root directory canonical path."); - } - } - - protected File createFileInRootDir(String path) throws Exception { - File file = new File(rootDirectory, path); - if (file.exists()) { - return null; - } - File parent = file.getParentFile(); - if (parent != null) { - parent.mkdirs(); - } - file.createNewFile(); - return file; - } - - protected File buildRootFile(String... pathParts) { - return new File(rootDirectory, Utils.buildPath(pathParts)); - } - - protected File buildFile(String... pathParts) { - return new File(Utils.buildPath(pathParts)); - } - - protected void writeToFile(File file, String content) { - Utils.createFileParents(file); - try (FileWriter writer = new FileWriter(file)) { - writer.write(content); - writer.flush(); - } catch (Exception exception) { - throw Utils.exception(exception, "Error creating or writing content to file '%s'.", file); - } - } - - protected MbtilesInfo getDefaultConfiguration() { - MbtilesInfo configuration = new MbtilesInfo(); - configuration.setPoolSize(1000); - configuration.setRootDirectory(getRootDirectory().getPath()); - configuration.setTemplatePath(Utils.buildPath("{grid}", "{layer}", "{format}", "{z}", "tiles-{x}-{y}.sqlite")); - configuration.setRowRangeCount(500); - configuration.setColumnRangeCount(500); - return configuration; - } - - protected static Resource stringToResource(String string) { - return new ByteArrayResource(string.getBytes()); - } - - protected static String resourceToString(Resource resource) { - return new String(Utils.resourceToByteArray(resource)); - } - - protected void addStoresToClean(MbtilesBlobStore... stores) { - for (MbtilesBlobStore store : stores) { - storesToClean.add(store); - } - } - - protected void addFilesToDelete(File... files) { - for (File file : files) { - filesToDelete.add(file); - } - } -} diff --git a/geowebcache/sqlite/src/test/resources/logging.properties b/geowebcache/sqlite/src/test/resources/logging.properties deleted file mode 100644 index d2ddbda2b1..0000000000 --- a/geowebcache/sqlite/src/test/resources/logging.properties +++ /dev/null @@ -1,10 +0,0 @@ -handlers=java.util.logging.ConsoleHandler - -java.util.logging.ConsoleHandler.level = INFO -java.util.logging.ConsoleHandler.formatter = org.geotools.util.logging.MonolineFormatter -org.geotools.util.logging.MonolineFormatter.source = logger:long - -.level= INFO -org.geotools.level = WARNING -org.geowebcache.level = INFO -org.geowebcache.config.XMLConfiguration.level = INFO \ No newline at end of file diff --git a/geowebcache/web/pom.xml b/geowebcache/web/pom.xml index 36f336ca5d..02bdf24ad5 100644 --- a/geowebcache/web/pom.xml +++ b/geowebcache/web/pom.xml @@ -85,11 +85,6 @@ gwc-aws-s3 ${project.version} - - org.geowebcache - gwc-sqlite - ${project.version} - org.geowebcache gwc-swift diff --git a/geowebcache/web/src/main/webapp/WEB-INF/geowebcache-servlet.xml b/geowebcache/web/src/main/webapp/WEB-INF/geowebcache-servlet.xml index 065c91f9e3..15dc6e685b 100644 --- a/geowebcache/web/src/main/webapp/WEB-INF/geowebcache-servlet.xml +++ b/geowebcache/web/src/main/webapp/WEB-INF/geowebcache-servlet.xml @@ -21,7 +21,6 @@ - diff --git a/geowebcache/web/src/main/webapp/WEB-INF/geowebcache-sqlite-context.xml b/geowebcache/web/src/main/webapp/WEB-INF/geowebcache-sqlite-context.xml deleted file mode 100644 index 4c1198f932..0000000000 --- a/geowebcache/web/src/main/webapp/WEB-INF/geowebcache-sqlite-context.xml +++ /dev/null @@ -1,16 +0,0 @@ - - - - - - -