Development workflow

Prerequisites

In order to code on Orchest, you need to have the following installed on your system:

Next, you will need to setup your development environment.

Development environment

Run the code below to install all dependencies needed for incremental development, building the docs, running tests and automatically running pre-commit hooks:

# Make sure you are inside the orchest root directory.
npm run setup --install && \
    pnpm i && \
    # Install dependencies to build the docs
    python3 -m pip install -r docs/requirements.txt && \
    # Install pre-commit hooks
    pre-commit install && \
    # Dependencies to run unit tests
    sudo apt-get install -y default-libmysqlclient-dev

IDE & language servers

Note

👉 This section is for VS Code and pyright users.

Who doesn’t like to use the smarts of their IDE by using features such as auto complete, go to definition, find all references etc. For everyone who is using VS Code (or the pyright language server to be more precise) the different services contain their own pyrightconfig.json file that configures these features. All that is needed is to install the dependencies of the services in the correct virtual environment. This is done using:

scripts/run_tests.sh

Next you can create a workspace file that sets up VS Code to use the right Python interpreters (do note that this won’t include all the files defined in the Orchest repo), e.g.:

{
    "folders": [
        {
            "path": "services/orchest-api"
        },
        {
            "path": "services/orchest-webserver"
        },
        {
            "path": "services/base-images/runnable-shared"
        },
        {
            "path": "services/orchest-ctl"
        },
        {
            "path": "services/session-sidecar"
        },
        {
            "path": "services/memory-server"
        },
        {
            "name": "orchest-sdk",
            "path": "orchest-sdk/python"
        },
        {
            "name": "internal lib Python",
            "path": "lib/python/orchest-internals/"
        }
    ],
    "settings": {}
}

Building Orchest

Last but not least, Orchest needs to be build from source:

scripts/build_container.sh

# By default the scripts builds all containers in parallel. To learn
# more about other options, such as building without cache, check out
# the first lines of the script first.
head -45 scripts/build_container.sh

Building the docs

Our docs are build using Read the Docs with Sphinx and written in reStructuredText.

To build the docs, run:

cd docs
make html

Tip

👉 If you didn’t follow the prerequisites, then make sure you’ve installed the needed requirements to builds the docs:

python3 -m pip install -r docs/requirements.txt

Incremental development

Warning

🚨 For incremental development to work in WSL2, Docker must be installed within the WSL2 environment itself.

Now that you have Orchest and all devevelopment dependencies installed you ready to start Orchest in dev mode by using the --dev flag. This way code changes are instantly reflected, without having to build the containers again (although it is good practice to rebuild all containers before committing your changes).

# In case any new dependencies were changed or added they need to
# be installed.
pnpm i

# Run the client dev server for hot reloading. Note: This command
# does not finish.
pnpm run dev

# Start Orchest in a new terminal window.
./orchest start --dev

With --dev the repository code from the filesystem is mounted (and thus adhering to git branches) to the appropriate paths in the Docker containers. This allows for active code changes being reflected inside the application.

A few additional notes about running Orchest with the --dev flag:

  • All Flask applications are run in development mode.

  • Only the orchest-webserver, auth-server, file-manager and orchest-api support code changes to be instantly reflected. For code changes to other services you will have to rebuild the container and restart Orchest. To re-build a specific container (e.g. orchest-webserver), run the following command:

scripts/build_containers.sh -i orchest-webserver

Note

🎉 Awesome! Everything is set up now and you are ready to start coding. Have a look at our best practices and our GitHub to find interesting issues to work on.

Before committing

Make sure your development environment is set up correctly (see prerequisites) so that pre-commit can automatically take care of running the appropriate formatters and linters when running git commit. Lastly, it is good practice to rebuild all containers (and restart Orchest) to do some manual testing and running the unit tests to make sure your changes didn’t break anything:

# Rebuild containers to do manual testing.
scripts/build_containers.sh

# Run unit tests.
scripts/run_tests.sh

In our CI we also run all of these checks together with integration tests to make sure the codebase remains stable. To read more about testing, check out the testing section.

Opening a PR

Note

When opening a PR please change the base in which you want to merge from master to dev. The GitHub docs describe how this can be done.

We use gitflow as our branching model with master and dev being the described master and develop branches respectively. Therefore, we require PRs to be merged into dev instead of master.

When opening the PR a checklist will automatically appear to guide you to successfully completing your PR 🏁.

Database schema migrations

Whenever one of the services’s database models (in their respective models.py) have been changed, a database migration has to be performed so that all existing users are unaffected by the schema change on update (since they can then be automatically migrated to the latest version).

# Depending on the service that requires schema changes.
scripts/migration_manager.sh orchest-api migrate
scripts/migration_manager.sh orchest-webserver migrate

# For more options run:
scripts/migration_manager.sh --help

Testing

Unit tests

The unit tests (in particular for the orchest-api and orchest-webserver) run against a real database. This, together with additional setup, and the running of all unit tests is done using the following script:

scripts/run_tests.sh

At this moment we only have unit tests for the Python code.

Tip

👉 If you didn’t follow the prerequisites, then make sure you’ve installed the needed requirements to run the unit tests:

sudo apt install default-libmysqlclient-dev

Note

For isolation dependencies for the different services are installed within their respective virtual environments inside the .venvs folder.

Integration tests

Warning

🚨 Running integration tests will remove all content of the userdir directory along with all built environments (the provided script will ask you to confirm before proceeding).

The integration tests are build using Cypress and can be run using:

scripts/run_integration_tests.sh

Running all the integration tests can take some time, depending on the host running the tests but also on the browser version, run-times have been observed to range from 15 to 30 minutes.

Tip

👉 Adding the -g option opens the Cypress GUI. Use --help to see more options.

Troubleshooting

The script takes care of starting Orchest if it isn’t already. On the other hand, if Orchest is already started, then the script expects Orchest to be running on its default port 8000.