upstream/forgejo
1
0
Fork 0
mirror of https://codeberg.org/forgejo/forgejo.git synced 2026-02-19 00:07:49 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
forgejo/.forgejo/workflows/testing.yml
Mathieu Fenniak cc47a4057f ci: introduce semgrep to prevent using xorm.Sync() incorrectly in new migrations (#11142) 
Adds a CI check which detects any usage of xorm's `Sync` method that doesn't include `IgnoreDropIndices: true`, and causes an error.

`semgrep` is a semantic grep tool that allows for the relatively easy authoring of linting tools that are customized to a project's specific needs, rather than generic like `golangci` and related tools.  Although `semgrep` offers a suite of out-of-the-box rules (and a paid set of rules), neither of those are used here -- only one Forgejo-specific rule is added in `.semgrep/xorm.yaml`.

My intent with this change is to introduce the idea and infrastructure of `semgrep` with a single minimal rule.  Once in-place, this will become a tool that we can use when we recognize bad coding patterns and wish to correct them permanently, rather than relying on human code review.  While generic linting tools do this well for general patterns, this will allow Forgejo to apply domain-specific checks.  For example, in #11112, an error indicates that it might be appropriate for us to always use `.StorageEngine("InnoDB")` when using an xorm engine -- if we made that determination, it could be cemented in-place with a `semgrep` rule relatively easily.

This specific rule looks for any access for xorm's `Sync` or `SyncWithOptions` methods on the `*xorm.Engine` or `*xorm.Session`.  They are then considered errors if they don't include `IgnoreDropIndices: true`.  This is *typically* correct and safe, but can also be ignored when specifically needed.  In the `.semgrep/tests` folder, test code is added which validates that the `semgrep` rule matches the expected patterns; this self-test is run before `semgrep` runs on the PR in CI.

As a demonstration, when `IgnoreDropIndices` is removed from a migration, here's an error: https://codeberg.org/forgejo/forgejo/actions/runs/135750/jobs/12/attempt/1

```
    models/forgejo_migrations/v14b_add-action_run-preexecutionerrorcode.go
   ❯❯❱ semgrep.xorm-sync-missing-ignore-drop-indices
          xorm Sync operation may drop indices if used on an incomplete bean definition for an existing table.
          Use SyncWithOptions with IgnoreDropIndices: true instead.

           22┆ _, err := x.SyncWithOptions(xorm.SyncOptions{}, new(ActionRun))
```

## Checklist

The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).

### Tests

- I added test coverage for Go changes...
  - [ ] in their respective `*_test.go` for unit tests.
  - [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
  - [ ] in `web_src/js/*.test.js` if it can be unit tested.
  - [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).

### Documentation

- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.

### Release notes

- [ ] This change will be noticed by a Forgejo user or admin (feature, bug fix, performance, etc.). I suggest to include a release note for this change.
- [x] This change is not visible to a Forgejo user or admin (refactor, dependency upgrade, etc.). I think there is no need to add a release note for this change.

*The decision if the pull request will be shown in the release notes is up to the mergers / release team.*

The content of the `release-notes/<pull request number>.md` file will serve as the basis for the release notes. If the file does not exist, the title of the pull request will be used instead.

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/11142
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
2026-02-07 21:52:43 +01:00

316 lines
12 KiB
YAML
Raw Blame History

name: testing
enable-email-notifications: true
on:
pull_request:
push:
branches:
- 'forgejo*'
- 'v*/forgejo*'
workflow_dispatch:
jobs:
backend-checks:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
runs-on: docker
container:
image: 'data.forgejo.org/oci/node:24-bookworm'
options: --tmpfs /tmp:exec,noatime
steps:
- name: event info
run: |
cat <<'EOF'
${{ toJSON(github) }}
EOF
- uses: https://data.forgejo.org/actions/checkout@v6
- uses: ./.forgejo/workflows-composite/setup-env
- run: su forgejo -c 'make deps-backend deps-tools'
- run: su forgejo -c 'make --always-make -j$(nproc) lint-backend tidy-check swagger-check lint-swagger fmt-check swagger-validate' # ensure the "go-licenses" make target runs
- uses: ./.forgejo/workflows-composite/build-backend
frontend-checks:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
runs-on: docker
container:
image: 'data.forgejo.org/oci/node:24-bookworm'
options: --tmpfs /tmp:exec,noatime
steps:
- uses: https://data.forgejo.org/actions/checkout@v6
- run: make deps-frontend
- run: make lint-frontend
- run: make checks-frontend
- name: make test-frontend-coverage
run: |
# Usage of `dayjs` can be impacted by local system timezone and can be sensitive to DST differences; since
# frontend tests are very short they're run twice with varying DST rules to reduce regression risk.
TZ=Europe/Berlin make test-frontend-coverage
TZ=America/Edmonton make test-frontend-coverage
- run: make frontend
- name: Install zstd for cache saving
# works around https://github.com/actions/cache/issues/1169, because the
# consuming job has zstd and doesn't restore the cache otherwise
run: |
apt-get update -qq
apt-get -q install -qq -y zstd
- name: 'Cache frontend build for playwright testing'
uses: https://data.forgejo.org/actions/cache/save@v5
with:
path: ${{github.workspace}}/public/assets
key: frontend-build-${{ github.sha }}
test-unit:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
runs-on: docker
needs: [backend-checks, frontend-checks]
container:
image: 'data.forgejo.org/oci/node:24-bookworm'
options: --tmpfs /tmp:exec,noatime
services:
elasticsearch:
image: data.forgejo.org/oci/bitnami/elasticsearch:7
options: --tmpfs /bitnami/elasticsearch/data
env:
discovery.type: single-node
ES_JAVA_OPTS: '-Xms512m -Xmx512m'
minio:
image: data.forgejo.org/oci/bitnami/minio:2024.8.17
options: >-
--hostname gitea.minio --tmpfs /bitnami/minio/data:noatime
env:
MINIO_DOMAIN: minio
MINIO_ROOT_USER: 123456
MINIO_ROOT_PASSWORD: 12345678
steps:
- uses: https://data.forgejo.org/actions/checkout@v6
- uses: ./.forgejo/workflows-composite/setup-env
- name: install git >= 2.42
uses: ./.forgejo/workflows-composite/apt-install-from
with:
packages: git
- name: test release-notes-assistant.sh
run: |
apt-get -q install -qq -y jq
./release-notes-assistant.sh test_main
- uses: ./.forgejo/workflows-composite/build-backend
- run: |
su forgejo -c 'make test-backend test-check'
timeout-minutes: 120
env:
RACE_ENABLED: 'true'
TAGS: bindata
TEST_ELASTICSEARCH_URL: http://elasticsearch:9200
TEST_MINIO_ENDPOINT: minio:9000
test-e2e:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
runs-on: docker
needs: [backend-checks, frontend-checks]
container:
image: 'data.forgejo.org/oci/playwright:latest'
options: --tmpfs /tmp:exec,noatime
steps:
- uses: https://data.forgejo.org/actions/checkout@v6
with:
fetch-depth: 20
- uses: ./.forgejo/workflows-composite/setup-env
- name: 'Restore frontend build'
uses: https://data.forgejo.org/actions/cache/restore@v5
id: cache-frontend
with:
path: ${{github.workspace}}/public/assets
key: frontend-build-${{ github.sha }}
- name: 'Build frontend (if not cached)'
if: steps.cache-frontend.outputs.cache-hit != 'true'
run: |
su forgejo -c 'make deps-frontend frontend'
- uses: ./.forgejo/workflows-composite/build-backend
- name: Decide to run all tests
id: run-all
if: contains(github.event.pull_request.labels.*.name, 'run-all-playwright-tests') || contains(github.event.pull_request.title, 'playwright')
run: |
echo "all=1" >> "$GITHUB_OUTPUT"
- name: Get changed files
id: changed-files
uses: https://data.forgejo.org/tj-actions/changed-files@v47
with:
separator: '\n'
- run: |
su forgejo -c 'make generate test-e2e-sqlite'
timeout-minutes: 120
env:
USE_REPO_TEST_DIR: 1
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
CHANGED_FILES: ${{steps.changed-files.outputs.all_changed_files}}
RUN_ALL: ${{steps.run-all.all}}
- name: Upload test artifacts on failure
if: failure()
uses: https://data.forgejo.org/forgejo/upload-artifact@v4
with:
name: test-artifacts.zip
path: tests/e2e/test-artifacts/
retention-days: 3
test-remote-cacher:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
runs-on: docker
needs: [backend-checks, frontend-checks, test-unit]
container:
image: 'data.forgejo.org/oci/node:24-bookworm'
options: --tmpfs /tmp:exec,noatime
name: ${{ format('test-remote-cacher ({0})', matrix.cacher.name) }}
strategy:
matrix:
cacher:
- name: redis
image: data.forgejo.org/oci/bitnami/redis:7.2
options: --tmpfs /bitnami/redis/data:noatime
- name: redict
image: registry.redict.io/redict:7.3.0-scratch
options: --tmpfs /data:noatime
- name: valkey
image: data.forgejo.org/oci/bitnami/valkey:7.2
options: --tmpfs /bitnami/redis/data:noatime
- name: garnet
image: ghcr.io/microsoft/garnet-alpine:1.0.14
options: --tmpfs /data:noatime
services:
cacher:
image: ${{ matrix.cacher.image }}
options: ${{ matrix.cacher.options }}
env:
ALLOW_EMPTY_PASSWORD: 'yes' # redis & valkey will immediately shutdown with no defined password unless overridden
steps:
- uses: https://data.forgejo.org/actions/checkout@v6
- uses: ./.forgejo/workflows-composite/setup-env
- name: install git >= 2.42
uses: ./.forgejo/workflows-composite/apt-install-from
with:
packages: git
- uses: ./.forgejo/workflows-composite/build-backend
- run: |
su forgejo -c 'make test-remote-cacher test-check'
timeout-minutes: 120
env:
RACE_ENABLED: 'true'
TAGS: bindata
TEST_REDIS_SERVER: cacher:6379
test-mysql:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
runs-on: docker
needs: [backend-checks, frontend-checks]
container:
image: 'data.forgejo.org/oci/node:24-bookworm'
options: --tmpfs /tmp:exec,noatime
services:
mysql:
image: 'data.forgejo.org/oci/bitnami/mysql:8.4'
env:
ALLOW_EMPTY_PASSWORD: yes
MYSQL_DATABASE: testgitea
#
# See also https://codeberg.org/forgejo/forgejo/issues/976
#
MYSQL_EXTRA_FLAGS: --innodb-adaptive-flushing=OFF --innodb-buffer-pool-size=4G --innodb-log-buffer-size=128M --innodb-flush-log-at-trx-commit=0 --innodb-flush-log-at-timeout=30 --innodb-flush-method=nosync --innodb-fsync-threshold=1000000000 --disable-log-bin
options: --tmpfs /bitnami/mysql/data:noatime
steps:
- uses: https://data.forgejo.org/actions/checkout@v6
- uses: ./.forgejo/workflows-composite/setup-env
- name: install dependencies & git >= 2.42
uses: ./.forgejo/workflows-composite/apt-install-from
with:
packages: git git-lfs
- uses: ./.forgejo/workflows-composite/build-backend
- run: |
su forgejo -c 'make test-mysql-migration test-mysql'
timeout-minutes: 120
env:
USE_REPO_TEST_DIR: 1
test-pgsql:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
runs-on: docker
needs: [backend-checks, frontend-checks]
container:
image: 'data.forgejo.org/oci/node:24-bookworm'
options: --tmpfs /tmp:exec,noatime
services:
minio:
image: data.forgejo.org/oci/bitnami/minio:2024.8.17
env:
MINIO_ROOT_USER: 123456
MINIO_ROOT_PASSWORD: 12345678
options: --tmpfs /bitnami/minio/data
ldap:
image: data.forgejo.org/oci/forgejo-test-openldap:1
options: --memory 500M
pgsql:
image: data.forgejo.org/oci/bitnami/postgresql:16
env:
POSTGRESQL_DATABASE: test
POSTGRESQL_PASSWORD: postgres
POSTGRESQL_FSYNC: off
POSTGRESQL_EXTRA_FLAGS: -c full_page_writes=off
options: --tmpfs /bitnami/postgresql
steps:
- uses: https://data.forgejo.org/actions/checkout@v6
- uses: ./.forgejo/workflows-composite/setup-env
- name: install dependencies & git >= 2.42
uses: ./.forgejo/workflows-composite/apt-install-from
with:
packages: git git-lfs
- uses: ./.forgejo/workflows-composite/build-backend
- run: |
su forgejo -c 'make test-pgsql-migration test-pgsql'
timeout-minutes: 120
env:
RACE_ENABLED: true
USE_REPO_TEST_DIR: 1
TEST_LDAP: 1
test-sqlite:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
runs-on: docker
needs: [backend-checks, frontend-checks]
container:
image: 'data.forgejo.org/oci/node:24-bookworm'
options: --tmpfs /tmp:exec,noatime
steps:
- uses: https://data.forgejo.org/actions/checkout@v6
- uses: ./.forgejo/workflows-composite/setup-env
- name: install dependencies & git >= 2.42
uses: ./.forgejo/workflows-composite/apt-install-from
with:
packages: git git-lfs
- uses: ./.forgejo/workflows-composite/build-backend
- run: |
su forgejo -c 'make test-sqlite-migration test-sqlite'
timeout-minutes: 120
env:
TAGS: sqlite sqlite_unlock_notify
RACE_ENABLED: true
TEST_TAGS: sqlite sqlite_unlock_notify
USE_REPO_TEST_DIR: 1
security-check:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
runs-on: docker
needs:
- test-sqlite
- test-pgsql
- test-mysql
- test-remote-cacher
- test-unit
container:
image: 'data.forgejo.org/oci/node:24-bookworm'
options: --tmpfs /tmp:exec,noatime
steps:
- uses: https://data.forgejo.org/actions/checkout@v6
- uses: ./.forgejo/workflows-composite/setup-env
- run: su forgejo -c 'make deps-backend deps-tools'
- run: su forgejo -c 'make security-check'
semgrep:
if: vars.ROLE == 'forgejo-coding' || vars.ROLE == 'forgejo-testing'
name: semgrep/ci
runs-on: docker
container:
image: 'data.forgejo.org/oci/semgrep:latest'
steps:
- run: apk add nodejs # required for actions/checkout
- uses: https://data.forgejo.org/actions/checkout@v6
- name: self-check semgrep rules
run: semgrep --test .semgrep/tests/ --config .semgrep/config/
- name: semgrep ci
run: semgrep ci --config .semgrep/config/ --metrics=off
Reference in a new issue View git blame Copy permalink