chore: make the functions watching happen in the Docker container

.dockerignore

@@ -0,0 +1,3 @@
+containerization
+node_modules
+dump

containerization/Dockerfile

@@ -1,35 +1,31 @@
 ####################################################################
 #
-# Due to complexities with Yarn workspaces, this Dockerfile
-# does not create the `dist` folder. It should be mounted from
-# the host machine to the container, using the `-v` flag.
-#
-# Optionally, initial data can be setup by mounting the `/seed`
-# folder on the container.
+# When starting the container, you probably want to mount the
+# project folder, so code changes are shared with the container
+# (see commands below.) In that case, log files for each emulated
+# service are shared back to the host machine.
 #
+# TODO: check if the below is still true.
 # If you want to not get errors due to missing email templates,
-# you also need to mount those. The functions code in the
+# you also need to mount a folder. The functions code in the
 # emulator expects them to be located in the `/templates` folder
 # on the container.
 #
 # COMMANDS
-# We need some files from the root directory of the project,
-# therefore these commands should be ran from there.
+# We need files from the root directory of the project, therefore
+# these commands should be ran from there.
 #
 #   BUILD
 #   docker build -f ./containerization/Dockerfile -t backend .
 #
 #   RUN
-#   docker run -v ./functions:/app/functions -p 4001-4008:4001-4008 -it backend
+#   docker run -v ./:/app -p 4001-4008:4001-4008 -it backend
 #
-#   RUN WITH SEED DATA
-#   docker run -v ./containerization/data:/seed -v ./functions:/app/functions -p 4001-4008:4001-4008 -it backend
-#
-#   RUN WITH EMAIL TEMPLATES
+#   RUN WITH EMAIL TEMPLATES (TODO: unsure if still needed)
 #   docker run -v ./functions/src/emailNotifications/templates:/templates -v ./functions:/app/functions -p 4001-4008:4001-4008 -it backend
 #
 #   EXPORT (while the container is running)
-#   docker exec -it <conatiner_name> /app/export.js
+#   docker exec -it <conatiner_name> /app/containerization/export.cjs
 #   docker cp <conatiner_name>:/app/dump ./whatever
 #
 # HOW TO DEBUG THE CONTAINER WHILE IT IS RUNNING:
@@ -51,6 +47,9 @@
 #   Due to Docker, the Firebase emulators should run on 0.0.0.0
 #   https://stackoverflow.com/a/52518929
 #
+#   DOCUMENTATION FOR RUN
+#   https://docs.docker.com/reference/cli/docker/container/run
+#
 ####################################################################
 
 FROM node:20.9.0-bullseye-slim
@@ -74,7 +73,7 @@ HEALTHCHECK CMD curl --fail http://0.0.0.0:4001 || exit 1
 # This works but feel free to make a change.
 RUN npm install -g firebase-tools
 
-# Doing setup saves time when running the container.
+# Doing setup here saves time when running the container.
 # There are no setup commands for functions, hosting, or auth.
 RUN \
   firebase setup:emulators:ui && \
@@ -83,19 +82,39 @@ RUN \
   firebase setup:emulators:storage && \
   firebase setup:emulators:pubsub
 
-COPY ./firebase.json                    ./firebase.json
-COPY ./firebase.storage.rules           ./firebase.storage.rules
-COPY ./firestore.indexes.json           ./firestore.indexes.json
-COPY ./firestore.rules                  ./firestore.rules
-
-COPY ./containerization/export.js       ./export.js
-
-COPY ./containerization/link.js         ./link.js
-
-# This folder needs to exist because otherwise
-# the emulators error if the user did not mount
-# their own folder.
-RUN mkdir /seed
+############################################################
+#
+# We must do some weirdness because of the yarn workspace.
+#
+# The normal approach is:
+# 1) copy the package.json files
+# 2) install the dependencies
+# 3) copy the application files
+#
+# Doing it like this optimizes rebuilds, as dependenices
+# change much less often compared to application files.
+#
+# source: https://www.docker.com/blog/getting-started-with-docker-using-node-jspart-i/
+#
+# But for our situation, this results in an error when
+# trying to install the dependencies:
+#   Resolving packages...
+#   error Couldn't find any versions for "oa-components" that matches "workspace:*"
+#   info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
+#
+# So instead we must:
+# 1) copy the package.json files and application files
+# 2) install the dependencies
+#
+# This is slower but at least it works. And, ideally, once
+# the container is running, developers should not need
+# to be constantly rebuilding it.
+#
+COPY ./ ./
+#
+RUN yarn install
+#
+############################################################
 
 # These should be the ports specified in firebase.json
 EXPOSE 4001 4002 4003 4004 4005 4006 4007 4008
@@ -104,9 +123,9 @@ EXPOSE 4001 4002 4003 4004 4005 4006 4007 4008
 ENV IS_EMULATED=true
 
 CMD \
-  ./link.js & \
+  yarn workspace functions watch & \
   # Do firebase emulators:start --help for details
   firebase emulators:start \
   --project demo-community-platform-emulated \
   --only auth,functions,firestore,pubsub,storage,hosting,database \
-  --import=/seed
+  --import=/app/containerization/data

containerization/link.js

@@ -1,45 +0,0 @@
-#!/usr/bin/env node
-
-/*global require, console*/
-
-// Continuously copy the log files into a common folder
-// making it easier to export them.
-//
-// This script knows the folder structure of the container.
-//
-// Symbolic links did not work.
-
-const fs = require('fs')
-
-const isDebug = false
-
-log('setup...')
-
-const files = [
-  'database-debug.log',
-  'firebase-debug.log',
-  'firestore-debug.log',
-  'pubsub-debug.log',
-  'ui-debug.log',
-]
-
-// clear the files so the they are also cleared on the host.
-files.forEach((filename) => {
-  fs.writeFileSync('/app/' + filename, '')
-})
-
-fs.watch('/app', {}, (event, filename) => {
-  log('event: ' + event)
-  log('filename: ' + filename)
-
-  if (files.includes(filename)) {
-    log('updating... ' + filename)
-    fs.copyFileSync('/app/' + filename, '/app/logs/' + filename)
-  }
-})
-
-function log(statement) {
-  if (isDebug) {
-    console.log('[LINK-LOGS] ' + statement)
-  }
-}

containerization/logs/.gitignore

@@ -1 +0,0 @@
-*.log

docker-compose.yml

@@ -13,10 +13,7 @@ services:
     ports:
     - 4001-4008:4001-4008
     volumes:
-    - ./containerization/data:/seed
-    - ./containerization/logs:/app/logs
-    - ./functions:/app/functions
-    - ./functions/src/emailNotifications/templates:/templates
+    - ./:/app
 
   simulated-webhook-receiver:
     container_name: simulated-webhook-receiver

package.json

@@ -22,8 +22,7 @@
     "start:platform:for-emulated-backend": "yarn build:shared && vite --port 4000",
     "start:platform-ci": "yarn build:shared && vite --port 3456",
     "frontend:for-emulated-backend:watch": "concurrently --kill-others --names themes,components,platform --prefix-colors yellow,cyan,blue,magenta \"yarn start:themes\" \"yarn start:components\" \"yarn start:platform:for-emulated-backend\"",
-    "backend:emulator:watch": "concurrently --kill-others --names functions-watcher,docker-compose \"yarn workspace functions watch\" \"docker-compose up --force-recreate --build\"",
-    "backend:emulator:stop": "docker stop $(docker ps -a -q)",
+    "backend:emulator:watch": "docker-compose up --force-recreate --build",
     "build:themes": "yarn workspace oa-themes build",
     "build:components": "yarn workspace oa-components build",
     "build:vite": "tsc && vite build",

packages/documentation/docs/Backend Development/firebase-emulator.md

@@ -5,7 +5,7 @@ title: Firebase Emulator
 
 # Introduction
 
-To run backend functions locally, Firebase provides a suite of emulators to mimic most functionality seen online (e.g firestore, storage, functions, triggers etc.)
+To run backend functions locally, Firebase provides a suite of emulators to mimic most functionality seen online (e.g firestore, storage, functions, triggers, etc.)
 
 For simplicity, although each service is an individual emulator and we are running multiple services, we will refer to them all collectively as a single emulator.
 
@@ -15,7 +15,7 @@ We start the frontend and backend separately, so we have two different commands.
 
 ## Prerequisites
 
-The emulator can be a bit tricky to setup and populate with seed data, so a Docker image has been created that contains all the necessary setup.
+The emulator can be a bit tricky to setup and populate with seed data, so we have a Docker image that contains all the necessary setup.
 
 You will need to be able to run `docker-compose` commands on your local machine.
 
@@ -30,7 +30,7 @@ docker-compose -v
 
 ## Commands
 
-To make things easier, some Yarn commands were created.
+To make things easier, we offer some yarn commands.
 
 ### Starting the frontend
 
@@ -46,15 +46,7 @@ This is similar to `yarn run start` but configures the frontend to connect to th
 yarn run backend:emulator:watch
 ```
 
-This starts the Firebase emulator, loads code into it, and watches for changes. There is initial data but any changes will be lost after the emulator is stopped.
-
-### Stopping the backend
-
-Due to technical limitations of concurrently, the `CTRL+C` keyboard shortcut does not reach the emulator, so it will probably be necessary to explicitly run:
-
-```
-yarn run backend:emulator:stop
-```
+This starts the Firebase emulator, loads code, and watches for changes. The databases are populated with seed data, see the section below for details.
 
 ## Note
 
@@ -76,7 +68,7 @@ Clicking on tabs will take you to a page similar to the Firebase console, from w
 
 ## Seed data
 
-Hardcoded data is loaded into the emulator on start-up, any changes will be lost the next time the emulator is started.
+Hardcoded data is loaded into the emulator on start-up. Any changes will be lost the next time the emulator is started.
 
 You may experience some strange data issues, for example incorrectly getting error messages that a user already exists after restarting, but that is probably due to browser caching. You can verify that by using another browser; in that case the original browser's indexeddb cache would need to be manually cleared.
 
@@ -102,33 +94,29 @@ password: thanks_emulator_man
 
 ### Improving it
 
-You can improve the seed data by making changes via the application or Firebase user interface, exporting the data, and making a pull request. This will help make development and testing easier for you and others in the future.
+You can improve the seed data by making changes via the application or Firebase user interface, exporting the data, and making a pull request. This will help make development and testing easier for you and others.
 
 1. Get the container name using `docker ps`.
 
 2. Run the export script:
 
 ```
-docker exec -it <container_name> /app/export.js
+docker exec -it <container_name> /app/containerization/export.cjs
 ```
 
-3. Transfer the data from the container to your machine:
+3. Transfer the data from the container to your host machine:
 
 ```
-docker cp <container_name>:/app/dump ./functions/data/
+docker cp <container_name>:/app/dump ./whatever
 ```
 
-4. Delete the current `emulator` folder.
-
-5. Rename the `dump` folder to `emulator`.
-
-6. But note, each folder in the export must be checked into Git. If not, this will cause problems when the emulator tries to start. By default, Git does not track empty folders, so you must force it to track it by adding a .gitkeep file to the folder.
+4. Then replace the content of `./containerization/data` with it. But note, each folder in the export must be checked into Git. If not, this will cause problems when the emulator tries to start. By default, Git does not track empty folders, so you must force it to track it by adding a `.gitkeep` file to the folder.
 
 ## Function development
 
 ### Writing functions code
 
-The emulator binds to the `./functions/dist` folder so that changes made will be reflected in the emulator. On Linux these changes should be picked up immediately. On Windows the changes are not always detected and may require spinning the emulator down and then starting back up.
+When you make changes to the functions code, this is shared to the container and then the emulator. These changes should be picked up almost immediately.
 
 ### Invoking functions
 
@@ -152,9 +140,7 @@ Read more about [connecting your app to the Cloud Functions Emulator](https://fi
 
 ### Accessing logs
 
-If the emulator throws an error you may want to check generated debug.log files. These will exist in the container in the root `/app` folder.
-
-By default, when using the commands above, the log files will sync to the `./functions/logs` folder on the host machine.
+If the emulator throws an error you may want to check generated debug.log files. These are generated on the container and shared to your machine. Just check the root folder, for example: `firestore-debug.log`
 
 You can access the file system within the docker container directly using the
 [Remote-Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extension for vscode, and running the command to `attach to running container`.