Merge pull request #270 from Trivadis/feature/issue-269-sqlcl-23.4.0

Feature/issue 269 sqlcl 23.4.0

Author: PhilippSalvisberg

README.md

@@ -6,11 +6,13 @@ This repository provides formatter settings for the [coding style rules](https:/
 
 Settings are primarily provided for
 
-- [Oracle SQLcl, Version 23.3.0](https://www.oracle.com/tools/downloads/sqlcl-downloads.html)
+- [Oracle SQLcl, Version 23.4.0](https://www.oracle.com/tools/downloads/sqlcl-downloads.html)
 - [Oracle SQL Developer, Version 23.1.1](https://www.oracle.com/tools/downloads/sqldev-downloads.html) (requires `dbtools-common.jar` from SQLcl 23.3.0)
 
 These settings have been defined and tested with the product versions mentioned above. They might not work in other versions.
 
+Please note, that these settings cannot be used in [Oracle SQL Developer for VSCode 23.4.0](https://marketplace.visualstudio.com/items?itemName=Oracle.sql-developer). And it does not work in the embedded version of SQLcl 23.3.1.0, which cannot run JavaScript. 
+
 JDK 11 is required for SQLDev and SQLcl. The standalone tvdformat.jar works with JDK 11 or newer.
 
 See [releases](https://github.com/Trivadis/plsql-formatter-settings/releases) for settings supporting older versions.
@@ -21,7 +23,9 @@ SQL Developer is slowly reaching the end of its life cycle. The days when SQL De
 
 A new SQLcl version typically comes with enhancements and bug fixes in the area of PL/SQL and SQL grammar. And this also requires an adaptation of the formatting rule due to symbol name changes etc. And as a result, these formatting rules can actually only be used in SQLcl. 
 
-However, we know that the grammars and the formatter are provided in a JAR called `dbtools-common.jar`. And this JAR file also exists in the SQL Developer distribution. This means that in order to be able to use the current formatting rules in SQL Developer, we have to copy the `dbtools-common.jar` file from SQLcl to SQL Developer as follows:
+However, we know that the grammars and the formatter are provided in a JAR called `dbtools-common.jar`. And this JAR file also exists in the SQL Developer distribution. This means that in order to be able to use the current formatting rules in SQL Developer, we have to copy the `dbtools-common.jar` file from SQLcl to SQL Developer. Unfortunatelly the classes are not 100% compatible with the SQL Developer. As a a result, we have to keep some original classes, which complicates the patching process a bit.
+
+Here's the full procedure to use `dbtools-common.jar` from SQLcl 23.4.0 in SQL Developer 23.1.1.345:
 
 1. Quit SQL Developer
 
@@ -35,11 +39,31 @@ However, we know that the grammars and the formatter are provided in a JAR calle
 
    Find the `dbtools-common.jar` in your SQLcl installation. In our case it's in `/usr/local/bin/sqlcl/lib`. Copy the file to the SQL Developer’s directory (where `dbtools-common.original.jar` is located).
 
-4. Start SQL Developer
+4. Patch SQLcl's `dbtools-common.jar`
+
+   Open a terminal window in the `lib` folder where `dbtools-common.original.jar` and `dbtools-commmon.jar` are located and run the following commands to copy the class `oracle.dbtools.parser.ParserNode.class` to `dbtools-common.jar`:
+
+   ```bash
+   jar -xvf dbtools-common.original.jar oracle/dbtools/parser/ParseNode.class
+   jar -u0vMf dbtools-common.jar oracle/dbtools
+   rm -rf oracle
+   ```
+
+   This step is necessary to ensure that the "Code Outline" continues to work in SQL Developer.
+
+5. Clear SQL Developer's cache
+
+   Delete the following directory:
+
+   - On Windows: `%APPDATA%\SQL Developer\system23.1.1.345.2114\system_cache`
+   - On other platforms: `$HOME/.sqldeveloper/system23.1.1.345.2114/system_cache`
+
+   This step is necessary to ensure no other version of `dbtools-common.jar` is used from the cache. You will all
+   window settings. However, all other preferences including your connections are preserved.
 
-   Open an editor and test if the formatter is working.
+6. Start SQL Developer
 
-We have successfully tested this procedure with SQL Developer 23.1.1. However, there were cases in the past where this did not work. For example, using the JAR of SQLcl 20.3.0 in SQL Developer 20.2.0. Replacing a JAR requires a certain level of compatiblity. We therefore expect that this procedure will no longer work with an upcoming version of SQLcl. We will update this section as soon as we know more.
+   Open an editor and test if the formatter and code outline is working.
 
 ## Deviating Settings
 
@@ -137,7 +161,7 @@ SQL Developer uses its own parse tree query language called Arbori for its advan
 
 #### Links
 
-- [SQL Developer 22.2 User Guide, Code Editor: Format](https://docs.oracle.com/en/database/oracle/sql-developer/22.2/rptug/sql-developer-concepts-usage.html#GUID-9421DA6E-A48A-427B-88C9-4414D83EC9D1__CODEEDITORFORMAT-C73DB981)
+- [SQL Developer 23.1 User Guide, Code Editor: Format](https://docs.oracle.com/en/database/oracle/sql-developer/23.1/rptug/sql-developer-concepts-usage.html#GUID-9421DA6E-A48A-427B-88C9-4414D83EC9D1__CODEEDITORFORMAT-C73DB981)
 - [Formatting Code With SQL Developer](https://www.salvis.com/blog/2020/04/13/formatting-code-with-sql-developer/)
 - [Formatter Callback Functions](https://www.salvis.com/blog/2020/11/12/formatter-callback-functions/)
 - [Arbori Starter Manual](https://vadimtropashko.files.wordpress.com/2017/02/arbori-starter-manual.pdf)

settings/sql_developer/trivadis_custom_format.arbori

@@ -17,7 +17,7 @@
 include "std.arbori"
 
 /**
- * Lightweight Formatter for SQL Developer and SQLcl, version 23.3.1-SNAPSHOT
+ * Lightweight Formatter for SQL Developer and SQLcl, version 23.4.0-SNAPSHOT
  * The idea is to keep the code formatted "as is" and apply chosen formatting rules only.
  *
  * The Arbori program is processed from top to bottom.
@@ -157,9 +157,22 @@ d1_initialize_timer:
 dontFormatNode:
       [node) numeric_literal
     | [node) path             -- expected in SQL*Plus, SQLcl commands only
+    | [node) "create_java"
 ->
 ;
 
+enableKeepQutoedIdentifiersForCall:
+    runOnce
+-> {
+    var keepQuotedIdentifiersForCall = false;
+}
+
+enforceKeepQuotedIdentifiersForCall:
+    [node) "create_java"
+-> {
+    keepQuotedIdentifiersForCall = true;
+}
+
 -- --------------------------------------------------------------------------------------------------------------------
 -- I5: Define global variables and functions.
 -- --------------------------------------------------------------------------------------------------------------------
@@ -638,6 +651,7 @@ i10_determine_and_normalize_line_separators:
 --    - The identifier is not part of a code section for which the formatter is disabled.
 --    - The identifier is not part of a conditional compilation block.
 --    - The identifier is not a reserved keyword
+--    - The variable keepQuotedIdentifiersForCall is true (set at the beginning, disabled for Java)
 --
 -- Quoted identifiers look like strings in Java and other languages. The parser in SQLDev 22.2.0 does not
 -- understand Java stored procedures. Trying to format a Java stored procedure leads to a syntax error.
@@ -759,7 +773,7 @@ i11_enforce_nonquoted_identifiers:
     // get custom option keepQuotedIdentifiers (might not exist)
     var keepQuotedIdentifiers = struct.options.get("keepQuotedIdentifiers");
 
-    if (keepQuotedIdentifiers == null || !keepQuotedIdentifiers) {
+    if ((keepQuotedIdentifiers == null || !keepQuotedIdentifiers) && !keepQuotedIdentifiersForCall) {
         findAndConvertQuotedIdentifiers();
     }
 }

sqlcl/format.js

@@ -51,7 +51,7 @@ var javaSqlEarley = Java.type("oracle.dbtools.parser.plsql.SqlEarley");
 var javaProgram = Java.type("oracle.dbtools.arbori.Program");
 
 var getVersion = function() {
-    return "23.3.1-SNAPSHOT";
+    return "23.4.0-SNAPSHOT";
 }
 
 var getFiles = function (rootPath, extensions, ignoreMatcher) {
@@ -759,8 +759,12 @@ var unregisterTvdFormat = function () {
         .stream().map(function(l) {return l.getClass()}).collect(javaCollectors.toSet());
     // re-register all commands except for class TvdFormat and remaining (not removed) listener classes
     for (var i in listeners) {
-        if (listeners.get(i).toString() !== "TvdFormat" && !remainingListeners.contains(listeners.get(i).getClass())) {
-            javaCommandRegistry.addForAllStmtsListener(listeners.get(i).getClass());
+        try {
+            if (listeners.get(i).toString() !== "TvdFormat" && !remainingListeners.contains(listeners.get(i).getClass())) {
+                javaCommandRegistry.addForAllStmtsListener(listeners.get(i).getClass());
+            }
+        } catch (e) {
+            // ignore, assume toString() cannot be evaluated and therefore a class that does not need to be registered.
         }
     }
 }

standalone/README.md

@@ -31,13 +31,13 @@ The parameters are the same as for the [SQLcl command `tvdformat`](../sqlcl/READ
 
 A native image is a platform specific executable. It does not require a JDK at runtime. A native image uses less resources and is faster. The following images can be produced with a GraalVM JDK 17:
 
-OS      | amd64 (Intel)?  | aarch64 (ARM)? |
-------- | :-------------: | :------------: |
-macOS   | yes             | yes            |
-Linux   | yes             | yes            |
-Windows | yes             | -              |
+| OS      | amd64 (Intel)? | aarch64 (ARM)? |
+|---------|:--------------:|:--------------:|
+| macOS   |      yes       |      yes       |
+| Linux   |      yes       |      yes       |
+| Windows |      yes       |       -        |
 
-Currently there is no way to produce an ARM based (aarch64) native image for Windows. 
+Currently, there is no way to produce an ARM based (aarch64) native image for Windows. 
 
 Native images are not part of a release. You have to build them yourself as described [below](#how-to-build).
 
@@ -49,67 +49,11 @@ To run a native image open a terminal window and type
 
 The parameters are the same as for the [executable JAR](#executable-jar).
 
-## Important: Native Images Do not Work Anymore
-
-Starting with SQLcl 23.1.0 it is not possible to build native images on all platforms anymore.
-The reason is that the underlying parser uses AWT/Swing components (for whatever reason).
-AWT/Swing components are supported on AMD/Linux platforms only by GraalVM.
-As a result this build process will produce a non-working native image on most platforms.
-
-You will see the following stacktrace similar to the following when running the resulting native image 
-(based on SQLcl 23.3.0 on macOS 14.1.2 with an M-series chip):
-
-```
-Exception in thread "main" javax.script.ScriptException: java.lang.Exception: java.lang.UnsatisfiedLinkError: no awt in java.library.path
-	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibraryRelative(NativeLibrarySupport.java:136)
-	at java.base@17.0.9/java.lang.ClassLoader.loadLibrary(ClassLoader.java:50)
-	at java.base@17.0.9/java.lang.Runtime.loadLibrary0(Runtime.java:818)
-	at java.base@17.0.9/java.lang.System.loadLibrary(System.java:1989)
-	at java.desktop@17.0.9/java.awt.Toolkit$2.run(Toolkit.java:1388)
-	at java.desktop@17.0.9/java.awt.Toolkit$2.run(Toolkit.java:1386)
-	at java.base@17.0.9/java.security.AccessController.executePrivileged(AccessController.java:171)
-	at java.base@17.0.9/java.security.AccessController.doPrivileged(AccessController.java:318)
-	at java.desktop@17.0.9/java.awt.Toolkit.loadLibraries(Toolkit.java:1385)
-	at java.desktop@17.0.9/java.awt.Toolkit.initStatic(Toolkit.java:1423)
-	at java.desktop@17.0.9/java.awt.Toolkit.<clinit>(Toolkit.java:1397)
-	at java.desktop@17.0.9/java.awt.Component.<clinit>(Component.java:624)
-	at java.base@17.0.9/java.lang.Class.ensureInitialized(DynamicHub.java:579)
-	at java.base@17.0.9/java.lang.Class.ensureInitialized(DynamicHub.java:579)
-	at java.base@17.0.9/java.lang.Class.ensureInitialized(DynamicHub.java:579)
-	at java.base@17.0.9/java.lang.Class.ensureInitialized(DynamicHub.java:579)
-	at java.base@17.0.9/java.lang.Class.ensureInitialized(DynamicHub.java:579)
-	at oracle.dbtools.parser.Visual.<init>(Visual.java:404)
-	at oracle.dbtools.parser.Grammar.parseGrammarFile(Grammar.java:172)
-	at oracle.dbtools.parser.Grammar.parseGrammarFile(Grammar.java:152)
-	at oracle.dbtools.arbori.Program.getRules(Program.java:285)
-	at oracle.dbtools.arbori.Program.getArboriParser(Program.java:194)
-	at oracle.dbtools.arbori.Program.compile(Program.java:338)
-	at oracle.dbtools.arbori.Program.compile(Program.java:327)
-	at oracle.dbtools.arbori.SqlProgram.<init>(SqlProgram.java:32)
-	at oracle.dbtools.app.Format$3.<init>(Format.java:343)
-	at oracle.dbtools.app.Format.format(Format.java:343)
-	at <js>.formatInSandbox(<eval>:168)
-	at <js>.formatFile(<eval>:670)
-	at <js>.formatFiles(<eval>:683)
-	at <js>.run(<eval>:737)
-	at <js>.:program(<eval>:803)
-	at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:403)
-	at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:485)
-	at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:427)
-	at com.trivadis.plsql.formatter.TvdFormat.run(TvdFormat.java:36)
-	at com.trivadis.plsql.formatter.TvdFormat.main(TvdFormat.java:49)
-```
-
-This is really sad. However, there is nothing we can do. 
-We have to wait until the SQL Developer team decides to provide a parser that does not require AWT/Swing components (as in previous versions)
-or until the GraalVM team provides a native-image build process that works on all platforms with AWT/Swing.
-See also [macOS: no awt in java.library.path](https://github.com/oracle/graal/issues/4124).
-
 ## How to Build
 
-1. [Download](https://www.oracle.com/tools/downloads/sqlcl-downloads.html) and install SQLcl 23.3.0
-2. [Download](https://github.com/graalvm/graalvm-ce-builds/releases/tag/jdk-17.0.9) and install the GraalVM JDK 17.0.9
-3. Go to the `bin` directory of the GraalVM JDK and run `./gu install js native-image visualvm`. For native image on Windows you need to [download](https://visualstudio.microsoft.com/downloads/) Visual Studio Community 2022 and install the C++ compiler. Use `x64 Native Tools Command Prompt for VS 2022` to get a terminal window with the correct environment.
+1. [Download](https://www.oracle.com/tools/downloads/sqlcl-downloads.html) and install SQLcl 23.4.0
+2. [Download](https://github.com/graalvm/graalvm-ce-builds/releases/tag/jdk-17.0.9) and install the GraalVM JDK 21.0.2
+3. For native image on Windows you need to [download](https://visualstudio.microsoft.com/downloads/) Visual Studio Community 2022 and install the C++ compiler. Use `x64 Native Tools Command Prompt for VS 2022` to get a terminal window with the correct environment.
 4. [Download](https://maven.apache.org/download.cgi) and install Apache Maven 3.9.6
 5. [Download](https://git-scm.com/downloads) and install a git command line client
 6. Clone the plsql-formatter-settings repository. The repository uses symbolic links. On Windows you have to use `git clone -c core.symlinks=true https://github.com/Trivadis/plsql-formatter-settings.git` as Administrator to make it work. See also [Symbolic Links in Windows](https://github.com/git-for-windows/git/wiki/Symbolic-Links) for more information.
@@ -138,14 +82,14 @@ See also [macOS: no awt in java.library.path](https://github.com/oracle/graal/is
 
     You can define the following optional parameters: 
 
-    | Parameter                  | Value   | Meaning |
-    | -------------------------- | ------- | ------- |
-    | `skip.native`              | `true`  | Do not produce a native image (default) |
-    |                            | `false` | Produce a native image |
-    | `skipTests`                | `true`  | Run tests (default) |
-    |                            | `false` | Do not run tests |
+    | Parameter                  | Value   | Meaning                                           |
+    | -------------------------- | ------- |---------------------------------------------------|
+    | `skip.native`              | `true`  | Do not produce a native image (default)           |
+    |                            | `false` | Produce a native image                            |
+    | `skipTests`                | `true`  | Do not run tests                                  |
+    |                            | `false` | Run tests (default)                               |
     | `disable.logging`          | `true`  | Disable logging message during test run (default) |
-    |                            | `false` | Enable logging message during test run |
+    |                            | `false` | Enable logging message during test run            |
 
     Here's a fully qualified example to produce a native image and run all integration tests:
 

standalone/install_sqlcl_libs.sh

@@ -15,7 +15,7 @@ if ! test -f "${SQLCL_LIBDIR}/dbtools-common.jar"; then
 fi
 
 # define common Maven properties
-SQLCL_VERSION="23.3.0"
+SQLCL_VERSION="23.4.0"
 
 # install JAR files in local Maven repository, these libs are not available in public Maven repositories
 mvn install:install-file -Dfile=$SQLCL_LIBDIR/dbtools-common.jar -DgeneratePom=true \