Enhancement: Prevent Tag List Pollution by Enforcing Alphabetical Tag Starts and Improving Code-Block Stripping

[gun666vald]

Title:

Enhancement: Prevent Tag List Pollution by Enforcing Alphabetical Tag Starts and Improving Code-Block Stripping

---

Issue Description:

1. Problem Statement

Flatnotes-Enhanced automatically indexes and populates tags in the sidebar on-the-fly by parsing any string matching #tagname inside your markdown files [2]. However, technical notes frequently contain:

• Hex Colors: e.g., primary_color to #125d0d
• Numerical Identifiers / Port Numbers: e.g., #1, #2, #14496
• Configuration / Shell Comments: e.g., #allow, #bash, #service, #requires inside code blocks.

Because the current backend tag regex (TAGS_RE) accepts any alphanumeric character as a starting character (using [a-zA-Z0-9_]), these values are falsely indexed as tags [4]. This pollutes the sidebar tag list with "ghost tags" that cannot be removed unless the user manually alters their technical notes and configurations.

Additionally, the default code-block stripping regular expression (CODEBLOCK_RE = re.compile(r"{1,3}.*?{1,3}", re.DOTALL)) can get out of sync on pages that mix single inline backticks (e.g. `my_script.sh`) and triple backtick code blocks (e.g., ```bash ... ```). When this happens, parts of the code block remain unstripped, and the comments inside them leak into the search index.

---

2. Proposed Solution

To ensure a clean, accurate tag list on technical servers, we propose two modifications to both the frontend and the backend tag-extraction logic:

1. Alphabetical Tag Start: Change the tag regex starting character class from [a-zA-Z0-9_] to [a-zA-Z] [4]. This immediately ignores hex colors, port numbers, dates, and plain numerical values.
2. Separate Code-Block Stripping: Define the code-block regex to match triple backtick blocks and single inline backticks separately, preventing matching synchronization issues.

---

3. Proposed Implementation Changes

Backend: server/notes/file_system/file_system.py

Modify the compiled regular expressions to separate code-block patterns and enforce alphabetical tag starts:

# Line 47-51 in server/notes/file_system/file_system.py
class FileSystemNotes(BaseNotes):
    # Require an alphabetical start: [a-zA-Z0-9_] changed to [a-zA-Z]
    TAGS_RE = re.compile(r"(?:(?<=^#)|(?<=\s#))[a-zA-Z][a-zA-Z0-9_/-]*(?=\s|$)")
    
    # Split triple and single backticks to prevent parsing out-of-sync bugs
    CODEBLOCK_RE = re.compile(r"```.*?```|`[^`]*`", re.DOTALL)
    
    # Enforce alphabetical start
    TAGS_WITH_HASH_RE = re.compile(
        r"(?:(?<=^)|(?<=\s))#[a-zA-Z][a-zA-Z0-9_/-]*(?=\s|$)"
    )

Frontend: client/views/Note.vue

Modify the noteTags computed property to strip code blocks cleanly and apply the same alphabetical filter:

const noteTags = computed(() => {
  if (!note.value.content) return [];
  const tags = [];
  // Strip code blocks to prevent code comment indexing on the client-side
  const cleanContent = note.value.content.replace(/```[\s\S]*?```/g, "").replace(/`[^`]*`/g, "");
  // Require tags to start with an alphabetical letter to avoid hex colors and numbers
  const tagMatches = cleanContent.match(/(?:^|\s)#[a-zA-Z][a-zA-Z0-9_/-]*/g);
  if (tagMatches) {
    tags.push(...tagMatches.map(t => t.trim().substring(1)));
  }
  return tags;
});

---

4. Technical References (Attachments)

We have successfully implemented and tested these changes on a bare-metal environment using a custom compiler-filter patch script (patch_note.py) and a custom PKGBUILD for testing. The full, working implementations of these files are attached below for reference:

---

1. Custom Compiler-Filter Script (patch_note.py)

#!/usr/bin/env python3
import re
import sys
import os

def patch_note():
    # Define local path variables
    note_path = "client/views/Note.vue"
    fs_path = "server/notes/file_system/file_system.py"

    # =========================================================================
    # PART 1: PATCH THE FRONTEND (Note.vue) TEMPLATE AND TAG FILTERING
    # =========================================================================
    try:
        with open(note_path, 'r', encoding='utf-8') as f:
            code = f.read()
    except FileNotFoundError:
        print(f"Error: Could not locate {note_path} inside the source directory.")
        sys.exit(1)

    # 1. DYNAMICALLY EXTRACT EDIT-MODE ACTION BUTTONS
    edit_header_match = re.search(
        r'(<!-- Buttons — right aligned, never shrink -->.*?)\s*<!-- Row 2:',
        code,
        re.DOTALL
    )
    if not edit_header_match:
        print("Error: Could not locate Edit Mode buttons block in Note.vue")
        sys.exit(1)
    
    edit_header_raw = edit_header_match.group(1).strip()
    edit_buttons_html = re.sub(r'</div>\s*</div>$', '</div>', edit_header_raw, flags=re.DOTALL)

    # 2. DYNAMICALLY EXTRACT VIEW-MODE ACTION BUTTONS
    view_header_match = re.search(
        r'<!-- View Mode: fixed header with scrollable content -->.*?(<!-- Buttons — right aligned, never shrink -->.*?)\s*<!-- Row 2:',
        code,
        re.DOTALL
    )
    if not view_header_match:
        print("Error: Could not locate View Mode buttons block in Note.vue")
        sys.exit(1)
    
    view_header_raw = view_header_match.group(1).strip()
    view_buttons_html = re.sub(r'</div>\s*</div>$', '</div>', view_header_raw, flags=re.DOTALL)

    # 3. CONSTRUCT THE SINGLE-ROW METADATA + TITLE + ACTIONS TEMPLATES (Raw Strings)
    new_edit_header = r"""<!-- Edit Mode: full scrollable content -->
    <template v-if="editMode">
      <!-- ── Single-Row Unified Header (Dynamically Patched) ───────────────── -->
      <div class="shrink-0">
        <div class="flex items-center justify-between min-h-[2rem] gap-4 py-1 print:hidden">
          
          <!-- Left block: breadcrumb and title inline -->
          <div class="flex items-center gap-2 min-w-0 flex-1">
            <!-- Folder breadcrumb -->
            <div v-if="noteFolder" class="flex items-center gap-1 text-xs text-theme-text-very-muted shrink-0 truncate">
              <span v-for="(part, i) in folderParts" :key="i" class="flex items-center gap-1 min-w-0">
                <RouterLink
                  :to="{ name: 'search', query: { term: '*', folder: part.path, sortBy: 1 } }"
                  class="hover:text-theme-brand transition-colors truncate"
                >{{ part.name }}</RouterLink>
                <span v-if="i < folderParts.length - 1">/</span>
              </span>
              <span>/</span>
            </div>

            <!-- Title input inline -->
            <input
              ref="titleInputEl"
              v-model.trim="newTitle"
              class="w-full bg-theme-background outline-none text-xl font-bold truncate"
              placeholder="Title"
              @input="onTitleInput"
              @keydown="onTitleKeydown"
            />
            
            <!-- Folder path autocomplete dropdown -->
            <TagAutocomplete
              :visible="folderAcVisible"
              :matches="folderAcMatches"
              :counts="{}"
              :activeIndex="folderAcIndex"
              :anchorRect="folderAcAnchorRect"
              mode="folder"
              @choose="selectFolder"
              @hide="folderAcVisible = false"
            />
          </div>

          <!-- Dynamic Edit Buttons -->
          """ + edit_buttons_html + """
        </div>

        <hr class="my-2 border-theme-border" />
      </div>

      <!-- Content -->
      <div class="flex-1 min-h-0">"""

    new_view_header = r"""<!-- View Mode: fixed header with scrollable content -->
    <template v-else>
      <!-- Fixed Header (does not scroll) -->
      <div class="shrink-0">
        <div class="flex items-center justify-between min-h-[2rem] gap-4 py-1 print:hidden">
          
          <!-- Left block: folder path and title text inline -->
          <div class="flex items-center gap-2 min-w-0 flex-1">
            <!-- Folder breadcrumb -->
            <div v-if="noteFolder" class="flex items-center gap-1 text-xs text-theme-text-very-muted shrink-0 truncate">
              <span v-for="(part, i) in folderParts" :key="i" class="flex items-center gap-1 min-w-0">
                <RouterLink
                  :to="{ name: 'search', query: { term: '*', folder: part.path, sortBy: 1 } }"
                  class="hover:text-theme-brand transition-colors truncate"
                >{{ part.name }}</RouterLink>
                <span v-if="i < folderParts.length - 1">/</span>
              </span>
              <span>/</span>
            </div>

            <!-- Title text display -->
            <div class="text-xl font-bold truncate flex-1" :title="note.title">
              {{ noteBasename }}
            </div>
          </div>

          <!-- Dynamic View Buttons -->
          """ + view_buttons_html + """
        </div>

        <hr class="my-2 border-theme-border" />
      </div>

      <!-- Scrollable Content Area -->
      <div class="flex-1 overflow-y-auto min-h-0">"""

    # Inject style scoped overrides and dynamic client-side code-block tag stripping
    custom_style = """
<style scoped>
.pin-active {
  color: rgb(var(--theme-brand));
}
.toastui-editor-defaultUI {
  margin-top: 4px !important;
}
.toast-viewer {
  padding-bottom: 1rem !important;
}
</style>"""

    # Replace Edit Mode Header Block
    content_new, count_edit = re.subn(
        r'<!-- Edit Mode: full scrollable content -->.*?<div class="flex-1 min-h-0">',
        new_edit_header,
        code,
        flags=re.DOTALL
    )

    # Replace View Mode Header Block
    content_final, count_view = re.subn(
        r'<!-- View Mode: fixed header with scrollable content -->.*?<div class="flex-1 overflow-y-auto min-h-0">',
        new_view_header,
        content_new,
        flags=re.DOTALL
    )

    if count_edit == 0 or count_view == 0:
        print("Error: Target header templates not found in Note.vue.")
        sys.exit(1)

    # 4. LITERAL STR-REPLACE FOR VUE TAGS PROPERTY (Avoids backslash PatternError)
    old_computed_tags = r"""const noteTags = computed(() => {
  if (!note.value.content) return [];
  const tags = [];
  const tagMatches = note.value.content.match(/(?:^|\s)#[a-zA-Z0-9_/-]+/g);
  if (tagMatches) {
    tags.push(...tagMatches.map(t => t.trim().substring(1)));
  }
  return tags;
});"""

    new_computed_tags = r"""const noteTags = computed(() => {
  if (!note.value.content) return [];
  const tags = [];
  // Strip code blocks to prevent code comment indexing on front-end
  const cleanContent = note.value.content.replace(/```[\s\S]*?```/g, "").replace(/`[^`]*`/g, "");
  // Require tags to start with an alphabetical letter to avoid hex colors and numbers
  const tagMatches = cleanContent.match(/(?:^|\s)#[a-zA-Z][a-zA-Z0-9_/-]*/g);
  if (tagMatches) {
    tags.push(...tagMatches.map(t => t.trim().substring(1)));
  }
  return tags;
});"""

    # Using literal string.replace is 100% fail-safe against regex bad escapes
    content_final = content_final.replace(old_computed_tags, new_computed_tags)

    # Remove existing styled scoped and append our optimized version
    content_final = re.sub(r'<style scoped>.*?</style>', '', content_final, flags=re.DOTALL)
    content_final += custom_style

    with open(note_path, "w", encoding="utf-8") as f:
        f.write(content_final)

    print("Successfully patched client Note.vue (Layout & Tag Filters).")

    # =========================================================================
    # PART 2: PATCH THE BACKEND (server/notes/file_system/file_system.py)
    # =========================================================================
    try:
        with open(fs_path, 'r', encoding='utf-8') as f:
            fs_code = f.read()

        # Update the starting character of both TAGS_RE and TAGS_WITH_HASH_RE
        # This replaces '[a-zA-Z0-9_]' with '[a-zA-Z]', enforcing alphabetical tag starts
        fs_code_final = fs_code.replace(
            '[a-zA-Z0-9_][a-zA-Z0-9_/-]*(?=\\s|$)',
            '[a-zA-Z][a-zA-Z0-9_/-]*(?=\\s|$)'
        )

        # Update the CODEBLOCK_RE compilation inside file_system.py to separate single/triple backticks
        # This prevents code-block extraction from getting out of sync when mixed on a single page
        old_codeblock_re = 'CODEBLOCK_RE = re.compile(r"`{1,3}.*?`{1,3}", re.DOTALL)'
        new_codeblock_re = 'CODEBLOCK_RE = re.compile(r"```.*?```|`[^`]*`", re.DOTALL)'
        
        fs_code_final = fs_code_final.replace(old_codeblock_re, new_codeblock_re)

        with open(fs_path, 'w', encoding='utf-8') as f:
            f.write(fs_code_final)
        print("Successfully patched server/notes/file_system/file_system.py (Backend Tag & Codeblock Filters).")
        
    except Exception as e:
        print(f"Warning: Could not process server/notes/file_system/file_system.py modifications: {e}")

if __name__ == '__main__':
    patch_note()

---

2. Production PKGBUILD (v1.9.0-16)

# Maintainer: devome <evinedeng@hotmail.com>
# Modified for Production: ezbook Home Server (v1.9.0-16)

pkgname=flatnotes-enhanced
_pkgname=flatnotes-enhanced
pkgver=1.9.0
pkgrel=16
pkgdesc="A professional-grade, feature-rich fork of Flatnotes with folders, nested tags, and a dynamic single-row note layout."
arch=("any")
url="https://github.com/BobWs/flatnotes-enhanced"
license=("MIT")
depends=("python-aiofiles" "python-cryptography" "python-fastapi" "python-jose" "python-pyotp" "python-python-multipart" "python-qrcode" "python-whoosh" "python-sqlalchemy" "uvicorn")
makedepends=("npm")
provides=("flatnotes")
conflicts=("flatnotes")
backup=("etc/conf.d/flatnotes")

source=("${pkgname}-${pkgver}.tar.gz::${url}/archive/v${pkgver}.tar.gz"
        "flatnotes.env"
        "flatnotes.service"
        "flatnotes.sysusers"
        "flatnotes.tmpfiles"
        "patch_note.py")

sha256sums=('SKIP'
            '59e9f26dca4d316d580b86afb09afec3f949a4debfdbd4de9fdaed4868abbb78'
            '44b35a9f08962eb6e8fd7a48a9d9213cf2119e48eb5d4766a349244d221e8a15'
            '2d62cb21e34fd41277c1b9cd8692c68cf4980fd1c3d94330f2b79ddbc4349c1a'
            'dffdf187992981de979626511fba98cbcb62453a9796ffbfbffbad295ee1e15e'
            'SKIP')

prepare() {
    cd "${_pkgname}-${pkgver}"
    if [ -d "client/node_modules" ]; then
        rm -rf client/node_modules
    fi
    if [ -d "node_modules" ]; then
        rm -rf node_modules
    fi

    # Execute your python patch script to dynamically restructure the template
    msg2 "Executing patch_note.py..."
    python3 "${srcdir}/patch_note.py"
}

build() {
    cd "${_pkgname}-${pkgver}"
    npm ci --no-audit --no-fund --quiet
    npm run build
    mv client/dist/index.html .
}

package() {
    install -Dm644 "flatnotes.env"      "${pkgdir}/etc/conf.d/flatnotes"
    install -Dm644 "flatnotes.service"  "${pkgdir}/usr/lib/systemd/system/flatnotes.service"
    install -Dm644 "flatnotes.sysusers" "${pkgdir}/usr/lib/sysusers.d/flatnotes.conf"
    install -Dm644 "flatnotes.tmpfiles" "${pkgdir}/usr/lib/tmpfiles.d/flatnotes.conf"

    cd "${_pkgname}-${pkgver}"
    
    install -Dm644 LICENSE               "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE"
    install -Dm644 README.md             "${pkgdir}/usr/share/doc/${pkgname}/README.md"
    install -Dm644 index.html            "${pkgdir}/var/lib/flatnotes/index.html"
    install -Dm644 package.json          "${pkgdir}/usr/share/flatnotes/package.json"
    
    install -d "${pkgdir}/usr/share/flatnotes/client"
    cp -a client/dist "${pkgdir}/usr/share/flatnotes/client/"
    cp -a server "${pkgdir}/usr/share/flatnotes/"
    
    ln -sf "/var/lib/flatnotes/index.html" "${pkgdir}/usr/share/flatnotes/client/dist/index.html"
}

---

5. Expected Benefits

• True Tag Indexing: Purges all raw numerical, CSS hex colors, and configuration values from your tags sidebar.
• Reliable Code-Block Stripping: Splitting triple and single backticks into non-conflicting match blocks completely isolates code comments, ensuring they are skipped during tag parsing.
• Upstream Stability: The implementation is completely dynamic, allowing you to fetch new upstream releases without needing to manually reconcile code files or patch sets.

[BobWs]

@gun666vald,

Thank you for your detailed issue report and for taking the time to propose a solution! I really appreciate the effort you’ve put into this—it’s incredibly helpful.

I was already aware of the issue with tag pollution (e.g., hex colors, numerical identifiers, and code-block comments being incorrectly indexed as tags). Your proposed solution—enforcing alphabetical tag starts and improving code-block stripping—is exactly the kind of fix I was considering. You’ve saved me a lot of time by providing such a thorough explanation and even sharing the implementation details!

I’ll review your changes, test them, and include the fix in the next release. This will help ensure the tag list remains clean and functional for everyone.

Thanks again for your contribution and for using Flatnotes-Enhanced!

/Bob