Extended file server capabilities

README.md

@@ -1,28 +1,41 @@
 # PyRobusta
 
-A lightweight HTTP server library for MicroPython designed for constrained embedded systems.
+PyRobusta is a memory-conscious HTTP/1.1 server library built for embedded devices where heap usage, connection reliability, and stream processing efficiency matter. PyRobusta offers robust keep-alive connection management and efficient byte-stream processing while maintaining a predictable memory footprint.
 
-## HTTP features
-- Routing decorators
-- Fixed-size, configurable request/response buffers
+## HTTP Features
+- Routing decorators and wildcard-based URL matching
 - Multipart request and response handling
-- Chunked transfer decoding for streamed request bodies
-- Bounded-copy memory footprint
-- Finite-state-machine parser with linear sliding buffer
-- Robust byte-stream handling
+- Support for chunked encoding and streaming payloads
 - Query parameter parsing with percent encoding support
-- Persistent connections (set by **connection: keep-alive**)
+- Built-in API for uploading, downloading, and deleting files stored on the server
+- Persistent connection handling via the `Connection: keep-alive` header
+- HTTP/1.0 and HTTP/1.1 support
 - TLS support
 
+## Design Principles
+
+- Predictable memory usage through fixed-size stream buffers
+- Incremental byte-stream processing with bounded memory overhead
+- State-machine-driven request parsing for extensibility and protocol correctness
+- Reliable connection handling with keep-alive, timeouts, and transport error recovery
+- Designed specifically for MicroPython and memory-constrained embedded environments
+
+## Project Status
+
+PyRobusta is under active development. The public API is not yet considered
+stable and may change between releases.
+
+Starting with v1.0.0, backwards compatibility will be maintained within each major version. Any backwards-incompatible changes introduced before then are clearly documented in the release notes.
+
 # Installation
 
 Install PyRobusta on your MicroPython-enabled device using the mip package manager.
 
-A minimum of 40 KB free heap is required. However, for better usability and stability,\
-devices with more SRAM are strongly recommended. The ESP32-C3 SuperMini is a good\
+A minimum of 40 KB free heap is required. However, for better usability and stability,
+devices with more SRAM are strongly recommended. The ESP32-C3 SuperMini is a good
 entry-level option, providing a comfortable amount of free memory after installation.
 
-If you haven’t already set up your environment, follow the [setup guide](./docs/setup.md) to install\
+If you haven’t already set up your environment, follow the [setup guide](./docs/setup.md) to install
 mpremote and connect your device to Wi-Fi.
 
 
@@ -48,14 +61,46 @@ async def main():
 asyncio.run(main())
 ```
 
-# Access the Application
+# Verify the Installation
 
-Open a web browser and enter your device’s IP address in the address bar.\
-You should see the default homepage. Refer to the included documentation\
-for details on supported use cases and advanced features.
+Open a web browser and enter your device’s IP address in the address bar.
+
+If the server is running correctly, the default homepage will be displayed.
+Refer to the documentation for configuration options, routing, streaming
+payloads, and advanced HTTP features.
 
 ![image info](./docs/img/home_page.png)
 
+## Sample Application
+
+```python
+import asyncio
+from gc import mem_free, mem_alloc, collect
+
+import pyrobusta.server.http_server as http_server
+from pyrobusta.protocol.http import HttpEngine
+
+@HttpEngine.route("/mem-usage", "GET")
+def mem_usage(http_ctx, _):
+    collect()
+    free = mem_free()
+    used = mem_alloc()
+    usage_percentage = 100 * used / (free + used)
+    return "text/plain", (
+        f"Currently used: {usage_percentage:.2f}%\n"
+        f"Free   [bytes]: {free}\n"
+        f"Used   [bytes]: {used}\n"
+        f"Total  [bytes]: {used + free}\n"
+    )
+
+async def main():
+    server = http_server.HttpServer()
+    asyncio.create_task(server.start_socket_server())
+    while True:
+        await asyncio.sleep(1)
+
+asyncio.run(main())
+```
 
 # Configuration and Optimization
 
@@ -65,5 +110,5 @@ To fine-tune heap usage and optimize performance, see:
 
 # Development
 
-Check the provided development guide to create and deploy custom builds\
+Check the provided development guide to create and deploy custom builds
 to your device: [development guide](./docs/development.md)

docs/api.md

@@ -0,0 +1,59 @@
+***
+
+# File Management Endpoint (`/files`)
+
+This endpoint provides file management capabilities, allowing clients to upload, retrieve, and manage files through various HTTP methods. `http_files_api` must be set to `True` in pyrobusta.env to enable this API.
+
+## Summary
+
+| Method   | Path                 | Description |
+| :------- | :------------------- | :---------- |
+| `GET`    | `/files/{path}`      | Lists or retrieves metadata about files. |
+| `PUT`    | `/files/{file path}` | Uploads or overwrites a file at the specified path. |
+| `POST`   | `/files`             | Uploads multiple files in multipart/form-data. |
+| `DELETE` | `/files/{file path}` | Delete a file at the specified path. |
+
+---
+
+## Endpoint Details
+
+### 1. File Retrieval/Listing (`GET /files/{path}`)
+
+This endpoint allows general file system interaction, enabling operations such as listing directory contents and retrieving metadata as well as downloading files.
+
+*   **Method:** `GET`
+*   **Path:** `/files/{path}`
+*   **Success Response:** 200 OK.
+
+### 2. File Upload / Overwrite (`PUT /files/{file path}`)
+
+This method is used to upload a file or overwrite an existing file at a specific path.
+The upload path is restricted to /www/user_data.
+
+*   **Method:** `PUT`
+*   **Path:** `/files/{file path}`
+*   **Body:** Raw file content (e.g., binary data).
+*   **Success Response:** 201 Created.
+*   **Notes:** `transfer-encoding: chunked` is supported.
+
+### 3. File Upload (`POST /files`)
+
+This method handles general file uploads, designed for uploading multiple files with per-file chunking supported. Only multipart/form-data is accepted as a content type.
+
+The upload path is restricted to /www/user_data, however, content-disposition headers only have to specify the file name, /www/user_data is prepended by default.
+
+`http_multipart` must be set to `True` in the configuration to use this endpoint.
+
+*   **Method:** `POST`
+*   **Path:** `/files`
+*   **Body:** File content encapsulated in multipart/form-data.
+*   **Success Response:** 201 Created.
+
+### 4. File Delete (`DELETE /files/{file path}`)
+
+This method is used to delete a file at a specific path.
+The path is restricted to /www/user_data.
+
+*   **Method:** `PUT`
+*   **Path:** `/files/{file path}`
+*   **Success Response:** 204 No Content.

docs/configuration.md

@@ -3,16 +3,16 @@
 Configuration can be overridden in pyrobusta.env, in .env format. Create pyrobusta.env in the project root, and run ```make deploy-config```
 to upload it to the root directory of the target device.
 
-| Name              | Description                                                                                           | Default                       |
-|-------------------|-------------------------------------------------------------------------------------------------------|-------------------------------|
-| wifi_ssid         | Name of the Wi-Fi network. When empty, Wi-Fi is not initalized by the built-in wifi.py module.        | None                          |
-| wifi_password     | Password of the Wi-Fi network. When empty, Wi-Fi is not initalized by the built-in wifi.py module.    | None                          |
-| http_port         | Port number for HTTP.                                                                                 | 80                            |
-| https_port        | Port number for HTTPS.                                                                                | 443                           |
-| http_multipart    | Enable multipart HTTP requests/responses.                                                             | False                         |
-| http_mem_cap      | Max memory cap (% × 0.01) of usable heap for HTTP request/response stream buffers.                    | 0.1                           |
-| http_served_paths | Space delimited list of filesystem paths allowed to be served through HTTP.                           | "/www /lib/pyrobusta"         |
-| http_serve_files  | Enable/disable file serving.                                                                          | True                          |
-| socket_max_con    | Max number of socket connections of any enabled application server.                                   | 2                             |
-| tls               | Enable/disable TLS. When turned on, cert.der/key.der must be installed at the root.                   | False                         |
-| log_level         | Can be one of: warning, info, debug.                                                                  | "info"                        |
+| Name              | Description | Default |
+| :---------------- | :---------- | :------ |
+| wifi_ssid         | Name of the Wi-Fi network. When empty, Wi-Fi is not initalized by the built-in wifi.py module. | None |
+| wifi_password     | Password of the Wi-Fi network. When empty, Wi-Fi is not initalized by the built-in wifi.py module. | None |
+| http_port         | Port number for HTTP. | 80 |
+| https_port        | Port number for HTTPS. | 443 |
+| http_multipart    | Enable multipart HTTP requests/responses. | False |
+| http_mem_cap      | Max memory cap (% × 0.01) of usable heap for HTTP request/response stream buffers. | 0.1 |
+| http_served_paths | Space delimited list of filesystem paths allowed to be served through HTTP. | "/www /lib/pyrobusta" |
+| http_files_api    | Enables or disables the file management API endpoint (/files), allowing to upload, download, and list files. | False |
+| socket_max_con    | Max number of socket connections of any enabled application server. | 2 |
+| tls               | Enables or disables TLS. When turned on, cert.der/key.der must be installed at the root. | False |
+| log_level         | Can be one of: warning, info, debug. | "info" |

docs/dimensioning/http_dimensioning.md

@@ -10,7 +10,7 @@ of parameters relative to a defined baseline configuration.
 socket_max_con=1
 http_mem_cap=0.05
 http_multipart=False
-http_serve_files=True
+http_files_api=False
 tls=False
 http_port=8080
 https_port=4443

example/boot.py

@@ -0,0 +1,13 @@
+# This file is executed on every boot (including wake-boot from deepsleep)
+import asyncio
+import machine
+from os import listdir
+
+from pyrobusta.connectivity import wifi
+
+connected = wifi.initialize()
+if connected and not machine.reset_cause() == machine.SOFT_RESET:
+    if "app.py" in listdir():
+        import app
+
+        asyncio.run(app.main())

example/demo_app/app.py

@@ -56,3 +56,7 @@ async def main():
     asyncio.create_task(server.start_socket_server())
     while True:
         await asyncio.sleep(1)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

example/demo_app/boot.py

@@ -0,0 +1 @@
+../boot.py

example/mem_usage/app.py

@@ -47,3 +47,7 @@ async def main():
     asyncio.create_task(server.start_socket_server())
     while True:
         await asyncio.sleep(1)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

example/mem_usage/boot.py

@@ -0,0 +1 @@
+../boot.py

example/mip_repo/app.py

@@ -46,3 +46,7 @@ async def main():
     asyncio.create_task(server.start_socket_server())
     while True:
         await asyncio.sleep(1)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

example/mip_repo/boot.py

@@ -0,0 +1 @@
+../boot.py

src/pyrobusta/bindings/http_connection.py

@@ -96,11 +96,21 @@ async def _run_state_machine(self):
 
     async def _response_handler(self, resp_handler):
         if "closure" == type(resp_handler).__name__:
-            for is_finished in resp_handler(self._send_buf):
-                await self._flush_response()
-                if is_finished:
-                    break
-                await sleep_ms(self.STATE_MACHINE_SLEEP_MS)
+            if self._engine.get_response_header(b"transfer-encoding") == b"chunked":
+                for is_finished in resp_handler(self._send_buf):
+                    await self.write(b"%x\r\n" % self._send_buf.size())
+                    await self._flush_response()
+                    await self.write(b"\r\n")
+                    if is_finished:
+                        await self.write(b"0\r\n\r\n")
+                        break
+                    await sleep_ms(self.STATE_MACHINE_SLEEP_MS)
+            else:
+                for is_finished in resp_handler(self._send_buf):
+                    await self._flush_response()
+                    if is_finished:
+                        break
+                    await sleep_ms(self.STATE_MACHINE_SLEEP_MS)
         elif type(resp_handler).__name__ in ("FileIO", "BytesIO"):
             try:
                 while True: