diff --git a/.claude/THIRD_PARTY_LICENSES.md b/.claude/THIRD_PARTY_LICENSES.md index 0fc5df629c..c81079e979 100644 --- a/.claude/THIRD_PARTY_LICENSES.md +++ b/.claude/THIRD_PARTY_LICENSES.md @@ -19,7 +19,7 @@ |---|---|---|---| | `skills/context-budget/SKILL.md` | `skills/context-budget/SKILL.md` | ECC | description を日本語化、Misskey 固有メモを追記 | | `commands/harness-audit.md` | `commands/harness-audit.md` | ECC | scripts 依存の自動採点を、Claude が `pnpm`/`git`/`grep` で手動採点する版に書き換え。Misskey 固有の評価軸 (SPDX / endpoint-list / migration / locales) を組み込み | -| `commands/quality-gate.md` | `commands/quality-gate.md` | ECC | 言語自動判定を排除し Misskey 固定 pipeline (`pnpm` + tsgo + ESLint + Vitest) に。Prettier/Biome フェーズを削除 | +| `commands/quality-gate.md` | `commands/quality-gate.md` | ECC | 言語自動判定を排除し Misskey 固定 pipeline (`pnpm` + tsc + ESLint + Vitest) に。Prettier/Biome フェーズを削除 | ### MIT License (full text) diff --git a/.claude/commands/harness-audit.md b/.claude/commands/harness-audit.md index ce1d62ba8c..7268503859 100644 --- a/.claude/commands/harness-audit.md +++ b/.claude/commands/harness-audit.md @@ -35,7 +35,7 @@ Misskey リポジトリの `.claude/` 構成を 7 カテゴリで採点し、改 | 2 | Context Efficiency | frontmatter description の冗長度、SKILL.md の長さ分布、重複情報、CLAUDE.md の肥大化 | | 3 | Quality Gates | Stop / PreToolUse / PostToolUse hook の整備、`/quality-gate` 等の完了前ゲートの有無、自動 lint/typecheck | | 4 | Memory Persistence | `.claude/skills/*/SKILL.md` と `references/` の同期状態を評価。プロジェクト側 `.claude/memory/` は未採用方針 (auto-memory はユーザーホーム側で自動運用) のため、ここを採点起点にせず既定 5/10 から開始する | -| 5 | Eval Coverage | `working-on-backend` / `working-on-frontend` の testing リファレンス (backend-testing.md / frontend-testing.md) の網羅、Misskey 固有の e2e/fed/Storybook/Cypress 適用ガイド | +| 5 | Eval Coverage | `working-on-backend` / `working-on-frontend` の testing リファレンス (backend-testing.md / frontend-testing.md) の網羅、Misskey 固有の e2e/fed/Storybook/Playwright 適用ガイド | | 6 | Security Guardrails | SPDX 規約適用、migration 不変性ルール、ja-JP.yml 限定編集ルール、secrets 検出 | | 7 | Cost Efficiency | enabledPlugins の重複・過剰、context-budget の整備、MCP 過剰登録なし | diff --git a/.claude/commands/quality-gate.md b/.claude/commands/quality-gate.md index 6844c2bcc0..c4477a7f28 100644 --- a/.claude/commands/quality-gate.md +++ b/.claude/commands/quality-gate.md @@ -12,16 +12,16 @@ upstream path: commands/quality-gate.md upstream license: MIT — https://github.com/affaan-m/everything-claude-code/blob/main/LICENSE project-level notice: see .claude/THIRD_PARTY_LICENSES.md (Misskey 内サードパーティ一覧 + MIT 全文) -Imported into Misskey .claude/ on 2026-05-10. Pipeline 概念 (lint → typecheck → test) は upstream ECC 版から借用 (MIT)。実コマンド層は Misskey の pnpm + tsgo + ESLint + Vitest に固定し、formatter (Prettier/Biome) フェーズは削除した。 +Imported into Misskey .claude/ on 2026-05-10. Pipeline 概念 (lint → typecheck → test) は upstream ECC 版から借用 (MIT)。実コマンド層は Misskey の pnpm + tsc + ESLint + Vitest に固定し、formatter (Prettier/Biome) フェーズは削除した。 -note: 元 ECC 版は言語自動判定 + format/lint/type のジェネリック版だったが、Misskey 専用に pnpm + tsgo + ESLint + Vitest の組み合わせに固定。重い test:e2e / test:fed は含まない (CI 側で実行される)。 +note: 元 ECC 版は言語自動判定 + format/lint/type のジェネリック版だったが、Misskey 専用に pnpm + tsc + ESLint + Vitest の組み合わせに固定。重い test:e2e / test:fed は含まない (CI 側で実行される)。 --> # /quality-gate — Misskey 軽量品質ゲート `/quality-gate [scope]` -完了前の **軽量** 品質チェック。重い E2E / 連合テスト (test:e2e / test:fed / Cypress) は CI 側で実行されるため、本コマンドには含めない。 +完了前の **軽量** 品質チェック。重い E2E / 連合テスト (test:e2e / test:fed / Playwright) は CI 側で実行されるため、本コマンドには含めない。 ## Scope @@ -50,7 +50,7 @@ pnpm --filter frontend test lint がまとめて失敗していて typecheck の結果だけ単独で見たい場合は、以下を個別に回す。**通常は不要** (lint の出力を読めば足りる): ```bash -pnpm --filter backend typecheck # tsgo 単体 +pnpm --filter backend typecheck # tsc 単体 pnpm --filter frontend typecheck # vue-tsc 単体 (Vue SFC の型を見るため) ``` @@ -63,7 +63,7 @@ pnpm --filter backend lint pnpm --filter backend test ``` -`tsgo` の出力を単独で見たい時のみ optional で `pnpm --filter backend typecheck` を別途回す。 +`tsc` の出力を単独で見たい時のみ optional で `pnpm --filter backend typecheck` を別途回す。 ### Frontend scope @@ -120,4 +120,4 @@ Frontend ut: PASS (87/87) - ジェネリックな言語自動判定を排除し、Misskey 固定 pipeline に。 - formatter フェーズなし (Misskey は ESLint --fix のみ採用)。 -- e2e / federation / Cypress は重いため除外し CI 側に委譲。 +- e2e / federation / Playwright は重いため除外し CI 側に委譲。 diff --git a/.claude/skills/working-on-backend/references/knowledge/backend-testing.md b/.claude/skills/working-on-backend/references/knowledge/backend-testing.md index 10add7d84c..8fd6dfd9c4 100644 --- a/.claude/skills/working-on-backend/references/knowledge/backend-testing.md +++ b/.claude/skills/working-on-backend/references/knowledge/backend-testing.md @@ -26,7 +26,7 @@ cp .github/misskey/test.yml .config/test.yml 補足: -- ルートの `pnpm start:test` (Cypress 用にテストサーバーを起動するコマンド) を使う経路では実行時に `ncp` で自動コピーされる ([package.json](../../../../../package.json))。それ以外で backend テストを直接走らせる時は上記の手動コピーが必要 +- ルートの `pnpm start:test` (Playwright 用にテストサーバーを起動するコマンド) を使う経路では実行時に `ncp` で自動コピーされる ([package.json](../../../../../package.json))。それ以外で backend テストを直接走らせる時は上記の手動コピーが必要 - すでに `.config/test.yml` があれば各テストスクリプトの内部 `compile-config` で十分なので、追加で `pnpm --filter backend compile-config` を叩く必要はない - `pnpm start:test` は backend e2e テスト (`pnpm --filter backend test:e2e`) の前提ではない (ポート競合の元になるため使わないこと) diff --git a/.claude/skills/working-on-backend/references/knowledge/nestjs-di.md b/.claude/skills/working-on-backend/references/knowledge/nestjs-di.md index 6c75e1fca1..548c685892 100644 --- a/.claude/skills/working-on-backend/references/knowledge/nestjs-di.md +++ b/.claude/skills/working-on-backend/references/knowledge/nestjs-di.md @@ -6,7 +6,7 @@ Misskey の backend は NestJS 11 + Fastify 5 + TypeORM 1 (PostgreSQL) + Redis - **DI コンテナ**: NestJS の `@Injectable()` サービス + Repository (TypeORM) パターン - **DI トークン**: [`@/di-symbols.js`](../../../../../packages/backend/src/di-symbols.ts) の `DI` から `@Inject(DI.xxx)` で注入 -- **ビルド**: `rolldown -c` で `built/` にバンドル。型チェックは `tsgo` +- **ビルド**: `rolldown -c` で `built/` にバンドル。型チェックは `tsc` ## エンドポイント内での DI diff --git a/.claude/skills/working-on-backend/references/tasks/adding-api-endpoint.md b/.claude/skills/working-on-backend/references/tasks/adding-api-endpoint.md index c1d6445338..927fd61fec 100644 --- a/.claude/skills/working-on-backend/references/tasks/adding-api-endpoint.md +++ b/.claude/skills/working-on-backend/references/tasks/adding-api-endpoint.md @@ -236,7 +236,7 @@ pnpm --filter backend test:e2e ```bash # 個別ファイルを高速にチェック pnpm exec eslint --fix packages/backend/src/server/api/endpoints//.ts -pnpm --filter backend typecheck # tsgo --noEmit (backend のみ) +pnpm --filter backend typecheck # tsc --noEmit (backend のみ) # 一括 (PR 提出前) pnpm --filter backend lint diff --git a/.claude/skills/working-on-frontend/SKILL.md b/.claude/skills/working-on-frontend/SKILL.md index a4bcab53b4..3fd04257e0 100644 --- a/.claude/skills/working-on-frontend/SKILL.md +++ b/.claude/skills/working-on-frontend/SKILL.md @@ -27,7 +27,7 @@ SKILL.md 本体は references への索引だけ。具体的な手順や規約 - SCSS Modules / `--MI_THEME-*` `--MI-*` CSS 変数 / グローバル utility class (`_button` 等) → [references/knowledge/scss-modules.md](references/knowledge/scss-modules.md) - `os.alert` / `os.confirm` / `os.popup` 等 UI ヘルパー (ブラウザ標準 `alert()` 直呼びは禁止) → [references/knowledge/os-api.md](references/knowledge/os-api.md) - `*.stories.impl.ts` 併設規則 + 複数 story / argTypes / layout / action パターン → [references/knowledge/storybook.md](references/knowledge/storybook.md) -- frontend Vitest / Cypress E2E の書き方と前提 → [references/knowledge/frontend-testing.md](references/knowledge/frontend-testing.md) +- frontend Vitest / Playwright E2E の書き方と前提 → [references/knowledge/frontend-testing.md](references/knowledge/frontend-testing.md) ## 必ず最後に通る場所 diff --git a/.claude/skills/working-on-frontend/references/knowledge/frontend-testing.md b/.claude/skills/working-on-frontend/references/knowledge/frontend-testing.md index cfd81b8f92..49fe0fa59c 100644 --- a/.claude/skills/working-on-frontend/references/knowledge/frontend-testing.md +++ b/.claude/skills/working-on-frontend/references/knowledge/frontend-testing.md @@ -1,4 +1,4 @@ -# Frontend テスト (Vitest / Cypress) +# Frontend テスト (Vitest / Playwright) Misskey frontend のテスト構成。 @@ -13,11 +13,11 @@ pnpm --filter frontend test-and-coverage # カバレッジ付き - 主な配置: `packages/frontend/test/*.test.ts` (例: `i18n.test.ts`, `theme.test.ts`, `is-birthday.test.ts`) - ビルドツール周りなど対象コードと隣接させた方が分かりやすいテストは、コードと同じディレクトリに `*.test.ts` として置く (例: [packages/frontend/lib/rollup-plugin-unwind-css-module-class-name.test.ts](../../../../../packages/frontend/lib/rollup-plugin-unwind-css-module-class-name.test.ts)) -- 共有コンポーネント (`MkX.vue`) のユニットテストは現状少なく、`*.spec.ts` / `__tests__/` 形式は採用していない (Storybook + Cypress でカバー) +- 共有コンポーネント (`MkX.vue`) のユニットテストは現状少なく、`*.spec.ts` / `__tests__/` 形式は採用していない (Storybook + Playwright でカバー) -## Cypress E2E +## Playwright E2E -Cypress は **起動済みのテストサーバー** に対して走るため、unit より前提が多い。[.github/workflows/test-frontend.yml](../../../../../.github/workflows/test-frontend.yml) の `e2e` ジョブと同じ手順をローカルで踏む: +Playwright は **起動済みのテストサーバー** に対して走るため、unit より前提が多い。[.github/workflows/test-frontend.yml](../../../../../.github/workflows/test-frontend.yml) の `e2e` ジョブと同じ手順をローカルで踏む: ```bash # 1. テスト用 DB / Redis を起動 (テスト用ポート。開発用の compose.local-db.yml ではない) @@ -29,15 +29,14 @@ cp .github/misskey/test.yml .config/test.yml # 3. 全体ビルド pnpm build -# 4. テストサーバー起動 + Cypress 実行 (いずれもルートから) -pnpm e2e # 内部で pnpm start:test を起動し http://localhost:61812 を待って Cypress run -pnpm cy:open # 対話的に開く (サーバーは別途 pnpm start:test で起動しておく) +# 4. テストサーバー起動 + Playwright 実行 (いずれもルートから) +pnpm e2e # 内部で pnpm start:test を起動し http://localhost:61812 を待って Playwright を実行 ``` -- 設定: ルート [cypress.config.ts](../../../../../cypress.config.ts) -- テスト本体は [cypress/](../../../../../cypress/) 配下 +- 設定: ルート [packages/frontend/playwright.config.ts](../../../../../packages/frontend/playwright.config.ts) で、`packages/frontend/test/e2e/` 配下の `*.spec.ts` を対象にする +- テスト本体は [packages/frontend/test/e2e/](../../../../../packages/frontend/test/e2e/) 配下に配置する。 -新規 frontend 機能の E2E は Cypress に書くのが基本。ただし対象は主要 UI フロー (login / post / drive etc) に限定し、細かい単位テストは Vitest または Storybook で代替する慣習。 +新規 frontend 機能の E2E は Playwright に書くのが基本。ただし対象は主要 UI フロー (login / post / drive etc) に限定し、細かい単位テストは Vitest または Storybook で代替する慣習。 ## Storybook (視覚確認 + Chromatic 視覚回帰) @@ -55,6 +54,6 @@ pnpm --filter frontend build-storybook # 静的ビルド frontend のテスト種別で DB / Redis の要否が違う: - **Vitest (unit)** — DB 不要。ロジック / コンポーネント単体のテストで backend に繋がない (CI の `vitest` ジョブにも `services:` は無い) -- **Cypress (E2E)** — テストサーバー (`pnpm start:test`) 経由で backend に繋ぐため DB / Redis が必要。**テスト用ポートの [packages/backend/test/compose.yml](../../../../../packages/backend/test/compose.yml)** を使う (上記 Cypress E2E の手順を参照) +- **Playwright (E2E)** — テストサーバー (`pnpm start:test`) 経由で backend に繋ぐため DB / Redis が必要。**テスト用ポートの [packages/backend/test/compose.yml](../../../../../packages/backend/test/compose.yml)** を使う (上記 Playwright E2E の手順を参照) 開発用の `compose.local-db.yml` (db `5432` / redis `6379`) は **テストには使わない**。テスト用の `packages/backend/test/compose.yml` (`54312` / `56312`) とはポートが異なり、混同すると接続できない。 diff --git a/.coderabbit.yaml b/.coderabbit.yaml new file mode 100644 index 0000000000..c41da777fc --- /dev/null +++ b/.coderabbit.yaml @@ -0,0 +1,34 @@ +language: "ja-JP" +tone_instructions: "常に丁寧語（です・ます調）を用い、敬意のある表現で応答してください。人格や能力への評価、皮肉、威圧的・高圧的・命令的な表現は避けてください。指摘では問題点と影響を明確にし、理由を簡潔に説明したうえで、具体的な改善案を提案してください。不確実な事項は推測と明示し、確認を促してください。重大なバグやセキュリティ上の懸念は重要度と根拠を明確に伝えてください。" +reviews: + profile: "chill" + high_level_summary: false + poem: true + auto_review: + enabled: true + drafts: false + base_branches: + - "develop" + ignore_title_keywords: + - "New Crowdin updates" + ignore_usernames: + - "dependabot[bot]" + - "renovate[bot]" + - "github-actions[bot]" + path_instructions: + - path: "**/*" + instructions: | + 【レビュー対象外】 + - フォーマット違反、型エラー、SPDXヘッダーの記載漏れ等、静的解析で検出可能な問題については、別Workflow（CI/CD）でチェックされるため、レビュー対象外として無視してください。 + + 【特別な対応を要するレビュー】 + - プルリクエストで変更されていない部分において、プルリクエストのタイトルや説明文に照らしてスコープ外であるが修正すべき点を見つけた場合は、このプルリクエストで修正させるのではなく、別のプルリクエストを開いてそれを修正するように指示してください。\ + なお、当該プルリクエストで行が追加/削除されたことによりスコープ外の問題が起こりうる場合は、例外的にこのプルリクエストで修正させてください。 + + 【注力してほしい観点】 + - ビジネスロジックの不備、セキュリティ、パフォーマンス、設計パターンの適切性などに焦点を当ててレビューしてください。 + - プルリクエストのタイトルや説明文に照らしてスコープ外の変更が含まれていないかをレビューしてください。なお、明らかにスコープ外であると確信できるもの（例: 全く関係ないファイルを編集している等）のみを対象とし、スコープ外かどうかの判断に困る場合は指摘しないでください。\ + スコープ外の変更が含まれていた場合は指摘し、その部分について、このプルリクエストに含めずに別のプルリクエストを開いて変更するように指示してください。 + path_filters: + - "!CHANGELOG.md" + - "!**/__snapshots__/**" diff --git a/.config/docker_example.yml b/.config/docker_example.yml index 60a314c31c..4d7ec5e1e2 100644 --- a/.config/docker_example.yml +++ b/.config/docker_example.yml @@ -233,15 +233,37 @@ proxyBypassHosts: # For security reasons, uploading attachments from the intranet is prohibited, # but exceptions can be made from the following settings. Default value is "undefined". # Read changelog to learn more (Improvements of 12.90.0 (2021/09/04)). -#allowedPrivateNetworks: [ -# '127.0.0.1/32' -#] +#allowedPrivateNetworks: +# - '127.0.0.1/32' # Upload or download file size limits (bytes) #maxFileSize: 262144000 # Log settings # logging: +# # Log output format. "json" outputs one-line JSON; defaults to "pretty". +# format: pretty +# # Minimum level for all loggers. Defaults to info in production and debug otherwise. +# level: info +# # The longest matching logger domain takes precedence over the global level. +# domains: +# core.boot: info +# queue: error +# queue.deliver: debug +# db.sql: off +# # HTTP status classes to record in the access log. No output when unspecified. +# access: +# statusClasses: +# - '2xx' +# - '3xx' +# - '4xx' +# - '5xx' +# # Enable body logging explicitly only during development. +# bodies: +# request: false +# response: false +# # 16 KiB by default, up to 128 KiB. +# maxBytes: 16384 # sql: # # Outputs query parameters during SQL execution to the log. # # default: false diff --git a/.config/example.yml b/.config/example.yml index 0aa8940677..94e2388835 100644 --- a/.config/example.yml +++ b/.config/example.yml @@ -303,8 +303,88 @@ id: 'aidx' #sentryForBackend: # enableNodeProfiling: true +# # Specify Sentry integration names to disable individual auto-instrumentation. +# # The names are integration .name values, not factory function names. +# # To check enabled names, set `options.debug: true` and see +# # "Integration installed: " in the Sentry logs. +# # +# # As of 2026-07-07 / @sentry/node 10.62.0, useful names for Misskey include: +# # Postgres ... DB queries (when pg is externalized) +# # Redis ... ioredis commands +# # Fastify ... inbound HTTP routes +# # Http ... inbound/outbound HTTP; disabling this can also affect request isolation +# # NodeFetch ... fetch/undici requests +# disabledIntegrations: ['Postgres'] # options: # dsn: 'https://examplePublicKey@o0.ingest.sentry.io/0' +# # By default, Misskey prevents Sentry trace headers (`sentry-trace` and +# # `baggage`) from being sent to remote ActivityPub/Webhook/etc. hosts. +# # To intentionally propagate distributed traces to trusted internal services, +# # list only those internal URL patterns here. +# # Avoid broad patterns that match remote servers unless you intentionally +# # want to restore Sentry's legacy behavior of propagating traces to all +# # outbound HTTP requests. +# #tracePropagationTargets: [] +# # Internal allowlist example: +# #tracePropagationTargets: +# # - 'internal-service.example' + +# ┌─────────┐ +#───┘ Tracing └──────────────────────────────────────────────── +# OpenTelemetry distributed tracing (Traces only, opt-in). +# Existing API and Queue spans are exported to an OTLP http/protobuf endpoint. +# +# Standard OTEL_* environment variables are honored for values omitted below: +# OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_ENDPOINT, +# OTEL_EXPORTER_OTLP_HEADERS, OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG, etc. +# +# When this is enabled together with sentryForBackend, Misskey shares Sentry's +# TracerProvider and adds an OTLP SpanProcessor to it. In that mode: +# - sampleRate below is ignored; sampling follows sentryForBackend.options +# tracesSampleRate / tracesSampler. +# - resourceAttributes below is ignored; set OTEL_SERVICE_NAME and +# OTEL_RESOURCE_ATTRIBUTES in the environment if you need to override them. +# - spans produced by Sentry integrations are exported to both Sentry and OTLP. +# - sentry-trace / baggage propagation to outbound remote requests is disabled +# by default while OTel is enabled. Set propagateTraceToRemote: true only if +# you intentionally want Sentry trace propagation for your deployment. + +#otelForBackend: +# endpoint: 'http://localhost:4318/v1/traces' +# #headers: +# # authorization: 'Bearer xxxxx' +# #sampleRate: 1.0 +# # Record PostgreSQL query spans. Disabled by default to avoid instrumentation +# # overhead when database-level diagnostics are not needed. +# #capturePgSpans: false +# # Include raw SQL text in PostgreSQL spans. Requires capturePgSpans and is +# # disabled by default because non-parameterized queries can expose values to +# # the OTLP Collector. +# #capturePgStatement: false +# # Include PostgreSQL connection-pool spans with an existing parent span. +# # Requires capturePgSpans and is disabled by default to avoid noisy spans +# # from connection churn. +# #capturePgConnectionSpans: false +# # Record Redis command spans. Disabled by default because subscribing to the +# # ioredis diagnostics channel adds work to every Redis command, including +# # commands without a parent span. Enable for development diagnostics or when +# # the added overhead is acceptable in production. +# #captureRedisCommandSpans: false +# # Include Redis startup/reconnect spans. Disabled by default to avoid noisy +# # connection churn; these spans are recorded even without an HTTP or job +# # parent span. +# #captureRedisConnectionSpans: false +# # Record Redis commands without an HTTP or job parent span. Disabled by +# # default because queue polling and background work can produce many root +# # traces; only applies when captureRedisCommandSpans is enabled. +# #captureRedisRootSpans: false +# #resourceAttributes: +# # deployment.environment: 'production' +# #propagateTraceToRemote: false +# # Queue worker spans are independent traces linked to their enqueuer by default. +# # Set to 'parent' to make them child spans instead. This can create very large +# # traces for high-fan-out queues such as federation delivery. +# #jobTraceContextMode: 'link' #sentryForFrontend: # vueIntegration: @@ -383,9 +463,8 @@ proxyBypassHosts: # For security reasons, uploading attachments from the intranet is prohibited, # but exceptions can be made from the following settings. Default value is "undefined". # Read changelog to learn more (Improvements of 12.90.0 (2021/09/04)). -#allowedPrivateNetworks: [ -# '127.0.0.1/32' -#] +#allowedPrivateNetworks: +# - '127.0.0.1/32' # Upload or download file size limits (bytes) #maxFileSize: 262144000 @@ -395,6 +474,29 @@ proxyBypassHosts: # Log settings # logging: +# # Log output format. "json" outputs one-line JSON; defaults to "pretty". +# format: pretty +# # Minimum level for all loggers. Defaults to info in production and debug otherwise. +# level: info +# # The longest matching logger domain takes precedence over the global level. +# domains: +# core.boot: info +# queue: error +# queue.deliver: debug +# db.sql: off +# # HTTP status classes to record in the access log. No output when unspecified. +# access: +# statusClasses: +# - '2xx' +# - '3xx' +# - '4xx' +# - '5xx' +# # Enable body logging explicitly only during development. +# bodies: +# request: false +# response: false +# # 16 KiB by default, up to 128 KiB. +# maxBytes: 16384 # sql: # # Outputs query parameters during SQL execution to the log. # # default: false diff --git a/.config/cypress-devcontainer.yml b/.config/playwright-devcontainer.yml similarity index 99% rename from .config/cypress-devcontainer.yml rename to .config/playwright-devcontainer.yml index 7656526019..144b893741 100644 --- a/.config/cypress-devcontainer.yml +++ b/.config/playwright-devcontainer.yml @@ -218,9 +218,8 @@ proxyBypassHosts: # Media Proxy #mediaProxy: https://example.com/proxy -allowedPrivateNetworks: [ - '127.0.0.1/32' -] +allowedPrivateNetworks: + - '127.0.0.1/32' # Upload or download file size limits (bytes) #maxFileSize: 262144000 diff --git a/.devcontainer/devcontainer.yml b/.devcontainer/devcontainer.yml index cd99d063d5..8bbe1f8599 100644 --- a/.devcontainer/devcontainer.yml +++ b/.devcontainer/devcontainer.yml @@ -205,9 +205,8 @@ proxyBypassHosts: # Media Proxy #mediaProxy: https://example.com/proxy -allowedPrivateNetworks: [ - '127.0.0.1/32' -] +allowedPrivateNetworks: + - '127.0.0.1/32' # Upload or download file size limits (bytes) #maxFileSize: 262144000 diff --git a/.devcontainer/init.sh b/.devcontainer/init.sh index 216292b082..12d622581b 100755 --- a/.devcontainer/init.sh +++ b/.devcontainer/init.sh @@ -12,4 +12,4 @@ pnpm install --frozen-lockfile cp .devcontainer/devcontainer.yml .config/default.yml pnpm build pnpm migrate -pnpm exec cypress install +pnpm --filter frontend exec playwright install --with-deps chromium diff --git a/.github/labeler.yml b/.github/labeler.yml index b64d726d65..78341fd6cb 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -16,7 +16,7 @@ 'packages/frontend:test': - any: - changed-files: - - any-glob-to-any-file: ['cypress/**/*'] + - any-glob-to-any-file: ['packages/frontend/test/**/*'] 'packages/sw': - any: diff --git a/.github/scripts/backend-js-footprint-loader.mjs b/.github/scripts/backend-js-footprint-loader.mjs deleted file mode 100644 index cd2c0af2e6..0000000000 --- a/.github/scripts/backend-js-footprint-loader.mjs +++ /dev/null @@ -1,46 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -import { appendFileSync, statSync } from 'node:fs'; -import { extname } from 'node:path'; -import { fileURLToPath } from 'node:url'; - -const traceFile = process.env.MK_BACKEND_JS_FOOTPRINT_TRACE; -const jsExtensions = new Set(['.js', '.mjs', '.cjs']); - -function recordLoadedFile(kind, url, format) { - if (traceFile == null || !url.startsWith('file:')) return; - - let filePath; - try { - filePath = fileURLToPath(url); - } catch { - return; - } - - const extension = extname(filePath); - if (!jsExtensions.has(extension)) return; - - let size = null; - try { - size = statSync(filePath).size; - } catch { - return; - } - - appendFileSync(traceFile, `${JSON.stringify({ - kind, - format, - path: filePath, - size, - timestamp: Date.now(), - })}\n`); -} - -export async function load(url, context, nextLoad) { - const result = await nextLoad(url, context); - recordLoadedFile('esm', url, result.format ?? context.format ?? null); - return result; -} diff --git a/.github/scripts/backend-js-footprint-require.cjs b/.github/scripts/backend-js-footprint-require.cjs deleted file mode 100644 index 1adab8bc05..0000000000 --- a/.github/scripts/backend-js-footprint-require.cjs +++ /dev/null @@ -1,46 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -'use strict'; - -const { appendFileSync, statSync } = require('node:fs'); -const Module = require('node:module'); -const { extname } = require('node:path'); - -const traceFile = process.env.MK_BACKEND_JS_FOOTPRINT_TRACE; -const jsExtensions = new Set(['.js', '.mjs', '.cjs']); - -function recordLoadedFile(kind, filePath, request) { - if (traceFile == null || typeof filePath !== 'string') return; - - const extension = extname(filePath); - if (!jsExtensions.has(extension) && extension !== '.node') return; - - let size = null; - try { - size = statSync(filePath).size; - } catch { - return; - } - - appendFileSync(traceFile, `${JSON.stringify({ - kind, - format: extension === '.node' ? 'native' : 'commonjs', - path: filePath, - request, - size, - timestamp: Date.now(), - })}\n`); -} - -const originalLoad = Module._load; -const originalResolveFilename = Module._resolveFilename; - -Module._load = function load(request, parent, isMain) { - const resolved = originalResolveFilename.call(this, request, parent, isMain); - const result = originalLoad.apply(this, arguments); - recordLoadedFile('cjs', resolved, request); - return result; -}; diff --git a/.github/scripts/backend-js-footprint.mjs b/.github/scripts/backend-js-footprint.mjs deleted file mode 100644 index 9c98357ad6..0000000000 --- a/.github/scripts/backend-js-footprint.mjs +++ /dev/null @@ -1,473 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -import { fork, spawn } from 'node:child_process'; -import { createRequire } from 'node:module'; -import { cpus, tmpdir } from 'node:os'; -import { dirname, extname, join, relative, resolve, sep } from 'node:path'; -import { setTimeout } from 'node:timers/promises'; -import { fileURLToPath, pathToFileURL } from 'node:url'; -import { gzipSync } from 'node:zlib'; -import * as fs from 'node:fs/promises'; -import * as fsSync from 'node:fs'; -import * as http from 'node:http'; -import * as util from './utility.mts'; - -const __filename = fileURLToPath(import.meta.url); -const __dirname = dirname(__filename); - -const [repoDirArg, outputFileArg] = process.argv.slice(2); - -const STARTUP_TIMEOUT = util.readIntegerEnv('MK_JS_FOOTPRINT_STARTUP_TIMEOUT_MS', 120000, 1); -const SETTLE_TIME = util.readIntegerEnv('MK_JS_FOOTPRINT_SETTLE_TIME_MS', 10000, 0); -const REQUEST_COUNT = util.readIntegerEnv('MK_JS_FOOTPRINT_REQUEST_COUNT', 10, 0); - -const repoDir = resolve(repoDirArg); -const outputFile = resolve(outputFileArg); -const backendDir = join(repoDir, 'packages/backend'); -const backendBuiltDir = join(backendDir, 'built'); -const traceFile = join(tmpdir(), `misskey-backend-js-footprint-${process.pid}-${Date.now()}.jsonl`); -const require = createRequire(join(repoDir, 'package.json')); -const ts = require('typescript'); -const jsExtensions = new Set(['.js', '.mjs', '.cjs']); -const fileMetricCache = new Map(); -const packageInfoCache = new Map(); -const nativePackageNames = new Set(); - -function isInside(parent, child) { - const rel = relative(parent, child); - return rel === '' || (!rel.startsWith('..') && !rel.includes(`..${sep}`)); -} - -function bytesToKiB(value) { - return Math.round(value / 1024); -} - -async function resetState() { - const backendRequire = createRequire(join(backendDir, 'package.json')); - const pg = backendRequire('pg'); - const Redis = backendRequire('ioredis'); - - const postgres = new pg.Client({ - host: '127.0.0.1', - port: 54312, - database: 'postgres', - user: 'postgres', - }); - - await postgres.connect(); - try { - await postgres.query('DROP DATABASE IF EXISTS "test-misskey" WITH (FORCE)'); - await postgres.query('CREATE DATABASE "test-misskey"'); - } finally { - await postgres.end(); - } - - const redis = new Redis({ host: '127.0.0.1', port: 56312 }); - try { - await redis.flushall(); - } finally { - redis.disconnect(); - } -} - -function createRequest() { - return new Promise((resolvePromise, reject) => { - const req = http.request({ - host: 'localhost', - port: 61812, - path: '/api/meta', - method: 'POST', - }, res => { - res.on('data', () => { }); - res.on('end', () => resolvePromise()); - }); - req.on('error', reject); - req.end(); - }); -} - -async function waitForServerReady(serverProcess) { - let serverReady = false; - serverProcess.on('message', message => { - if (message === 'ok') serverReady = true; - }); - - const startupStartTime = Date.now(); - while (!serverReady) { - if (Date.now() - startupStartTime > STARTUP_TIMEOUT) { - serverProcess.kill('SIGTERM'); - throw new Error('Server startup timeout'); - } - await setTimeout(100); - } -} - -async function stopServer(serverProcess) { - serverProcess.kill('SIGTERM'); - - let exited = false; - await new Promise(resolvePromise => { - serverProcess.on('exit', () => { - exited = true; - resolvePromise(undefined); - }); - - setTimeout(10000).then(() => { - if (!exited) serverProcess.kill('SIGKILL'); - resolvePromise(undefined); - }); - }); -} - -function getPackageNameFromPath(filePath) { - const normalized = util.normalizePath(filePath); - const marker = '/node_modules/'; - const index = normalized.lastIndexOf(marker); - if (index === -1) return null; - - const rest = normalized.slice(index + marker.length).split('/'); - if (rest[0] === '.pnpm') { - const nestedNodeModulesIndex = rest.indexOf('node_modules'); - if (nestedNodeModulesIndex === -1) return null; - const packageParts = rest.slice(nestedNodeModulesIndex + 1); - if (packageParts.length === 0) return null; - return packageParts[0].startsWith('@') ? packageParts.slice(0, 2).join('/') : packageParts[0]; - } - - return rest[0]?.startsWith('@') ? rest.slice(0, 2).join('/') : rest[0] ?? null; -} - -function findPackageDir(filePath, packageName) { - const normalizedPackageName = packageName.split('/').join(sep); - let current = dirname(filePath); - - while (current !== dirname(current)) { - if (current.endsWith(`${sep}${normalizedPackageName}`) && fsSync.existsSync(join(current, 'package.json'))) { - return current; - } - - const parent = dirname(current); - if (parent === current) break; - current = parent; - } - - return null; -} - -function readPackageInfo(filePath) { - const externalPackageName = getPackageNameFromPath(filePath); - if (externalPackageName != null) { - const packageDir = findPackageDir(filePath, externalPackageName); - const cacheKey = packageDir ?? externalPackageName; - if (packageInfoCache.has(cacheKey)) return packageInfoCache.get(cacheKey); - - let version = null; - if (packageDir != null) { - try { - const packageJson = JSON.parse(fsSync.readFileSync(join(packageDir, 'package.json'), 'utf8')); - version = typeof packageJson.version === 'string' ? packageJson.version : null; - } catch { } - } - - const info = { - category: 'external', - name: externalPackageName, - version, - dir: packageDir, - }; - packageInfoCache.set(cacheKey, info); - return info; - } - - if (isInside(backendBuiltDir, filePath)) { - return { - category: 'internal', - name: 'backend', - version: null, - dir: backendDir, - }; - } - - return { - category: 'internal', - name: 'workspace', - version: null, - dir: repoDir, - }; -} - -function analyzeSource(filePath, source) { - const sourceFile = ts.createSourceFile(filePath, source, ts.ScriptTarget.Latest, true, ts.ScriptKind.JS); - const metrics = { - astNodeCount: 0, - functionCount: 0, - classCount: 0, - stringLiteralBytes: 0, - }; - - function visit(node) { - metrics.astNodeCount += 1; - - if ( - ts.isFunctionDeclaration(node) || - ts.isFunctionExpression(node) || - ts.isArrowFunction(node) || - ts.isMethodDeclaration(node) || - ts.isConstructorDeclaration(node) || - ts.isGetAccessorDeclaration(node) || - ts.isSetAccessorDeclaration(node) - ) { - metrics.functionCount += 1; - } else if (ts.isClassDeclaration(node) || ts.isClassExpression(node)) { - metrics.classCount += 1; - } else if (ts.isStringLiteral(node) || ts.isNoSubstitutionTemplateLiteral(node)) { - metrics.stringLiteralBytes += Buffer.byteLength(node.text); - } - - ts.forEachChild(node, visit); - } - - visit(sourceFile); - return metrics; -} - -function readFileMetrics(filePath) { - if (fileMetricCache.has(filePath)) return fileMetricCache.get(filePath); - - const source = fsSync.readFileSync(filePath); - const sourceText = source.toString('utf8'); - const astMetrics = analyzeSource(filePath, sourceText); - const packageInfo = readPackageInfo(filePath); - const metric = { - path: filePath, - displayPath: util.normalizePath(relative(repoDir, filePath)), - sourceBytes: source.byteLength, - gzipBytes: gzipSync(source).byteLength, - ...astMetrics, - package: packageInfo, - }; - - fileMetricCache.set(filePath, metric); - return metric; -} - -async function readTraceRecords() { - let content = ''; - try { - content = await fs.readFile(traceFile, 'utf8'); - } catch (err) { - if (err.code === 'ENOENT') return []; - throw err; - } - - const records = []; - for (const line of content.split('\n')) { - if (line.trim() === '') continue; - try { - records.push(JSON.parse(line)); - } catch { } - } - return records; -} - -function emptyTotals() { - return { - loadedJsModules: 0, - loadedJsSourceBytes: 0, - loadedJsGzipBytes: 0, - astNodeCount: 0, - functionCount: 0, - classCount: 0, - stringLiteralBytes: 0, - externalPackageCount: 0, - nativeAddonPackageCount: 0, - }; -} - -function addFileMetrics(target, metric) { - target.loadedJsModules += 1; - target.loadedJsSourceBytes += metric.sourceBytes; - target.loadedJsGzipBytes += metric.gzipBytes; - target.astNodeCount += metric.astNodeCount; - target.functionCount += metric.functionCount; - target.classCount += metric.classCount; - target.stringLiteralBytes += metric.stringLiteralBytes; -} - -function summarizeRecords(records, phase) { - const jsPaths = new Set(); - const nativePaths = new Set(); - - for (const record of records) { - if (typeof record.path !== 'string') continue; - - const extension = extname(record.path); - if (jsExtensions.has(extension)) { - jsPaths.add(resolve(record.path)); - } else if (extension === '.node') { - nativePaths.add(resolve(record.path)); - } - } - - for (const nativePath of nativePaths) { - const packageInfo = readPackageInfo(nativePath); - if (packageInfo.category === 'external') nativePackageNames.add(packageInfo.name); - } - - const totals = emptyTotals(); - const packages = new Map(); - const modules = []; - - for (const filePath of [...jsPaths].toSorted()) { - let metric; - try { - metric = readFileMetrics(filePath); - } catch (err) { - process.stderr.write(`Failed to analyze ${filePath}: ${err.message}\n`); - continue; - } - - addFileMetrics(totals, metric); - - const packageKey = metric.package.name; - if (!packages.has(packageKey)) { - packages.set(packageKey, { - name: metric.package.name, - version: metric.package.version, - category: metric.package.category, - sourceBytes: 0, - gzipBytes: 0, - modules: 0, - astNodeCount: 0, - functionCount: 0, - classCount: 0, - stringLiteralBytes: 0, - nativeAddon: false, - }); - } - - const packageSummary = packages.get(packageKey); - packageSummary.sourceBytes += metric.sourceBytes; - packageSummary.gzipBytes += metric.gzipBytes; - packageSummary.modules += 1; - packageSummary.astNodeCount += metric.astNodeCount; - packageSummary.functionCount += metric.functionCount; - packageSummary.classCount += metric.classCount; - packageSummary.stringLiteralBytes += metric.stringLiteralBytes; - - modules.push({ - path: metric.displayPath, - package: metric.package.name, - category: metric.package.category, - sourceBytes: metric.sourceBytes, - gzipBytes: metric.gzipBytes, - astNodeCount: metric.astNodeCount, - functionCount: metric.functionCount, - classCount: metric.classCount, - stringLiteralBytes: metric.stringLiteralBytes, - }); - } - - for (const packageName of nativePackageNames) { - const packageSummary = packages.get(packageName); - if (packageSummary != null) packageSummary.nativeAddon = true; - } - - const externalPackages = [...packages.values()].filter(packageSummary => packageSummary.category === 'external'); - totals.externalPackageCount = externalPackages.length; - totals.nativeAddonPackageCount = externalPackages.filter(packageSummary => packageSummary.nativeAddon).length; - - return { - totals: { - ...totals, - loadedJsSourceKiB: bytesToKiB(totals.loadedJsSourceBytes), - loadedJsGzipKiB: bytesToKiB(totals.loadedJsGzipBytes), - stringLiteralKiB: bytesToKiB(totals.stringLiteralBytes), - }, - packages: [...packages.values()].toSorted((a, b) => b.sourceBytes - a.sourceBytes), - modules: modules.toSorted((a, b) => b.sourceBytes - a.sourceBytes), - }; -} - -async function measureFootprint() { - await fs.writeFile(traceFile, ''); - - process.stderr.write('Resetting database and Redis\n'); - await resetState(); - - process.stderr.write('Running migrations\n'); - await util.run('pnpm', ['--filter', 'backend', 'migrate'], { - cwd: repoDir, - env: process.env, - logStdout: true, - }); - - const serverProcess = fork(join(backendBuiltDir, 'entry.js'), [], { - cwd: backendDir, - env: { - ...process.env, - NODE_ENV: 'production', - MK_DISABLE_CLUSTERING: '1', - MK_ONLY_SERVER: '1', - MK_NO_DAEMONS: '1', - MK_BACKEND_JS_FOOTPRINT_TRACE: traceFile, - }, - stdio: ['pipe', 'pipe', 'pipe', 'ipc'], - execArgv: [ - '--require', - join(__dirname, 'backend-js-footprint-require.cjs'), - '--experimental-loader', - pathToFileURL(join(__dirname, 'backend-js-footprint-loader.mjs')).href, - ], - }); - - serverProcess.stdout?.on('data', data => { - process.stderr.write(`[server stdout] ${data}`); - }); - - serverProcess.stderr?.on('data', data => { - process.stderr.write(`[server stderr] ${data}`); - }); - - serverProcess.on('error', err => { - process.stderr.write(`[server error] ${err}\n`); - }); - - try { - await waitForServerReady(serverProcess); - await setTimeout(SETTLE_TIME); - - //const startup = summarizeRecords(await readTraceRecords(), 'startup'); - - await Promise.all( - Array.from({ length: REQUEST_COUNT }).map(() => createRequest()), - ); - await setTimeout(1000); - - const afterRequest = summarizeRecords(await readTraceRecords(), 'afterRequest'); - - return { - timestamp: new Date().toISOString(), - measurement: { - strategy: 'runtime-loader-trace', - startupTimeoutMs: STARTUP_TIMEOUT, - settleTimeMs: SETTLE_TIME, - requestCount: REQUEST_COUNT, - cpus: cpus().length, - }, - phases: { - //startup, - afterRequest, - }, - }; - } finally { - await stopServer(serverProcess); - await fs.rm(traceFile, { force: true }); - } -} - -const result = await measureFootprint(); -await fs.writeFile(outputFile, `${JSON.stringify(result, null, 2)}\n`); diff --git a/.github/scripts/backend-memory-report.mts b/.github/scripts/backend-memory-report.mts deleted file mode 100644 index 9aae10d887..0000000000 --- a/.github/scripts/backend-memory-report.mts +++ /dev/null @@ -1,429 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -import { readFile, writeFile } from 'node:fs/promises'; -import * as util from './utility.mts'; -import * as heapSnapshotUtil from './heap-snapshot-util.mts'; -import type { MemoryReport } from './measure-backend-memory-comparison.mts'; - -const [baseFile, headFile, outputFile, baseJsFootprintFile, headJsFootprintFile] = process.argv.slice(2); - -type RuntimeLoadedJsFootprintReport = { - phases: Record<'afterRequest', { - totals: { - loadedJsModules: number; - loadedJsSourceBytes: number; - loadedJsGzipBytes: number; - astNodeCount: number; - functionCount: number; - classCount: number; - stringLiteralBytes: number; - externalPackageCount: number; - nativeAddonPackageCount: number; - }; - modules: { - path: string; - package: string; - category: string; - sourceBytes: number; - gzipBytes: number; - astNodeCount: number; - functionCount: number; - classCount: number; - stringLiteralBytes: number; - }[]; - }>; -}; - -const memoryReportPhases = [ - { - key: 'afterGc', - title: 'After GC', - }, -] as const; - -const metrics = [ - 'HeapUsed', - 'Pss', - 'Private_Dirty', - 'VmRSS', - 'External', -] as const; - -function formatMemoryMb(valueKiB: number | null | undefined) { - if (valueKiB == null) return '-'; - return `${util.formatNumber(valueKiB / 1000)} MB`; -} - -function getMemoryValue(report: MemoryReport, phase: typeof memoryReportPhases[number]['key'], metric: typeof metrics[number]) { - return report.summary[phase].memoryUsage[metric]; -} - -function getMemoryValueFromSample(sample: MemoryReport['samples'][number], phase: typeof memoryReportPhases[number]['key'], metric: typeof metrics[number]) { - return sample.phases[phase].memoryUsage[metric]; -} - -function getSampleSpread(report: MemoryReport, phase: typeof memoryReportPhases[number]['key'], metric: typeof metrics[number]) { - const values = report.samples.map(sample => getMemoryValueFromSample(sample, phase, metric)); - if (values.length < 2) return null; - - const center = util.median(values); - return util.median(values.map(value => Math.abs(value - center))); -} - -function renderMainTableForPhase(base: MemoryReport, head: MemoryReport, phase: typeof memoryReportPhases[number]['key']) { - const lines = [ - '| Metric | Base | Head | Δ median | Δ MAD | Δ min | Δ max |', - '| --- | ---: | ---: | ---: | ---: | ---: | ---: |', - ]; - - function formatDeltaMemory(deltaKiB: number) { - return util.formatColoredDelta(deltaKiB, v => formatMemoryMb(v), 100); // 0.1 MB threshold - } - - for (const metric of metrics) { - const baseValue = getMemoryValue(base, phase, metric); - const headValue = getMemoryValue(head, phase, metric); - - const baseSpread = getSampleSpread(base, phase, metric); - const headSpread = getSampleSpread(head, phase, metric); - const summary = util.pairedDeltaSummary(base.samples, head.samples, (sample) => getMemoryValueFromSample(sample, phase, metric)); - const percent = summary.median * 100 / baseValue; - const deltaMedian = summary == null ? '-' : `${formatDeltaMemory(summary.median)}
${util.formatDeltaPercent(percent, 0.1).replaceAll('\\%', '\\\\%')}`; - - lines.push(`| **${metric}** | ${formatMemoryMb(baseValue)}
± ${formatMemoryMb(baseSpread)} | ${formatMemoryMb(headValue)}
± ${formatMemoryMb(headSpread)} | ${deltaMedian} | ${summary?.mad == null ? '-' : formatMemoryMb(summary.mad)} | ${summary == null ? '-' : formatDeltaMemory(summary.min)} | ${summary == null ? '-' : formatDeltaMemory(summary.max)} |`); - } - - return lines.join('\n'); -} - -/* -function measurementSummary(base, head) { - const baseCount = base?.sampleCount; - const headCount = head?.sampleCount; - const strategy = base?.comparison?.strategy; - if (baseCount == null || headCount == null) return null; - - if (strategy === 'interleaved-pairs') { - const rounds = base?.comparison?.rounds ?? baseCount; - const warmupRounds = base?.comparison?.warmupRounds ?? 0; - return `_Measured as ${rounds} interleaved base/head pairs after ${warmupRounds} warmup pair(s). Values are medians; ± is median absolute deviation._`; - } - - return `_Sample count: base ${baseCount}, head ${headCount}. Values are medians; ± is median absolute deviation._`; -} -*/ - -function renderHeapSnapshotSection(base: MemoryReport, head: MemoryReport) { - const baseHeapSnapshotReport = { - summary: base.summary.afterGc.heapSnapshot!, - samples: base.samples.map(sample => ({ - round: sample.round, - data: sample.phases.afterGc.heapSnapshot!, - })), - }; - - const headHeapSnapshotReport = { - summary: head.summary.afterGc.heapSnapshot!, - samples: head.samples.map(sample => ({ - round: sample.round, - data: sample.phases.afterGc.heapSnapshot!, - })), - }; - - const table = heapSnapshotUtil.renderHeapSnapshotTable(baseHeapSnapshotReport, headHeapSnapshotReport); - if (table == null) return null; - - const lines = [ - '### V8 Heap Snapshot Statistics', - '', - table, - '', - ]; - - for (const graph of [ - //heapSnapshotUtil.renderHeapSnapshotSankey(baseHeapSnapshotReport, 'Base'), - heapSnapshotUtil.renderHeapSnapshotSankey(headHeapSnapshotReport, 'Head'), - ]) { - if (graph == null) continue; - lines.push(graph); - lines.push(''); - } - - return lines.join('\n'); -} - -function getJsFootprintValue(report: RuntimeLoadedJsFootprintReport, phase: 'afterRequest', key: keyof RuntimeLoadedJsFootprintReport['phases'][typeof phase]['totals']) { - const value = report.phases[phase].totals[key]; - return Number.isFinite(value) ? value : null; -} - -function renderJsFootprintMetricTable(base: RuntimeLoadedJsFootprintReport, head: RuntimeLoadedJsFootprintReport) { - const metricRows = [ - ['Loaded JS modules', 'loadedJsModules', util.formatNumber], - ['Loaded JS source', 'loadedJsSourceBytes', util.formatBytes], - //['Loaded JS gzip estimate', 'loadedJsGzipBytes', util.formatBytes], - //['AST nodes', 'astNodeCount', util.formatNumber], - //['Functions', 'functionCount', util.formatNumber], - //['Classes', 'classCount', util.formatNumber], - //['String literals', 'stringLiteralBytes', util.formatBytes], - ['External packages loaded', 'externalPackageCount', util.formatNumber], - ['Native addon packages', 'nativeAddonPackageCount', util.formatNumber], - ] as const; - - const lines = [ - '| Metric | Base | Head | Δ | Δ (%) |', - '| --- | ---: | ---: | ---: | ---: |', - ]; - - for (const [title, key, formatter] of metricRows) { - const baseValue = getJsFootprintValue(base, 'afterRequest', key); - const headValue = getJsFootprintValue(head, 'afterRequest', key); - if (baseValue == null || headValue == null) continue; - - lines.push(`| **${title}** | ${formatter(baseValue)} | ${formatter(headValue)} | ${util.formatColoredDelta(headValue - baseValue, v => formatter(v))} | ${util.calcAndFormatDeltaPercent(baseValue, headValue).replaceAll('\\%', '\\\\%')} |`); - } - - return lines.join('\n'); -} - -/* -function renderJsFootprintPhaseTable(base, head) { - const lines = [ - '| Phase | Base modules | Head modules | Δ modules | Base source | Head source | Δ source |', - '| --- | ---: | ---: | ---: | ---: | ---: | ---: |', - ]; - - for (const [phase, title] of [['startup', 'Startup'], ['afterRequest', 'After warmup requests']]) { - const baseModules = getJsFootprintValue(base, phase, 'loadedJsModules'); - const headModules = getJsFootprintValue(head, phase, 'loadedJsModules'); - const baseSource = getJsFootprintValue(base, phase, 'loadedJsSourceBytes'); - const headSource = getJsFootprintValue(head, phase, 'loadedJsSourceBytes'); - if (baseModules == null || headModules == null || baseSource == null || headSource == null) continue; - - lines.push(`| ${title} | ${util.formatNumber(baseModules)} | ${util.formatNumber(headModules)} | ${formatPlainDelta(baseModules, headModules)} | ${util.formatBytes(baseSource)} | ${util.formatBytes(headSource)} | ${formatPlainDelta(baseSource, headSource, util.formatBytes)} |`); - } - - return lines.join('\n'); -} -*/ - -function packageMap(report: RuntimeLoadedJsFootprintReport) { - const map = new Map(); - for (const packageSummary of report.phases.afterRequest.packages) { - if (packageSummary?.category !== 'external' || typeof packageSummary.name !== 'string') continue; - map.set(packageSummary.name, packageSummary); - } - return map; -} - -function packageDisplayName(packageSummary: { name: string; version?: string | null }) { - if (packageSummary.version == null) return packageSummary.name; - return `${packageSummary.name} ${packageSummary.version}`; -} - -function renderNewExternalPackages(base: RuntimeLoadedJsFootprintReport, head: RuntimeLoadedJsFootprintReport) { - const basePackages = packageMap(base); - const headPackages = packageMap(head); - const newPackages = [...headPackages.values()] - .filter(packageSummary => !basePackages.has(packageSummary.name)) - .toSorted((a, b) => b.sourceBytes - a.sourceBytes) - .slice(0, 10); - - if (newPackages.length === 0) return null; - - const lines = [ - '#### Newly Loaded External Packages', - '', - '| Package | Loaded JS | Modules | Notes |', - '| --- | ---: | ---: | --- |', - ]; - - for (const packageSummary of newPackages) { - lines.push(`| ${packageDisplayName(packageSummary)} | ${util.formatBytes(packageSummary.sourceBytes)} | ${util.formatNumber(packageSummary.modules)} | ${packageSummary.nativeAddon ? 'native addon' : ''} |`); - } - - return lines.join('\n'); -} - -function renderLargestPackageIncreases(base: RuntimeLoadedJsFootprintReport, head: RuntimeLoadedJsFootprintReport) { - const basePackages = packageMap(base); - const headPackages = packageMap(head); - const increases = [...headPackages.values()] - .map(headPackage => { - const basePackage = basePackages.get(headPackage.name); - const baseSourceBytes = basePackage?.sourceBytes ?? 0; - const baseModules = basePackage?.modules ?? 0; - return { - ...headPackage, - baseSourceBytes, - baseModules, - sourceDiff: headPackage.sourceBytes - baseSourceBytes, - moduleDiff: headPackage.modules - baseModules, - }; - }) - .filter(packageSummary => packageSummary.sourceDiff > 0) - .toSorted((a, b) => b.sourceDiff - a.sourceDiff) - .slice(0, 10); - - if (increases.length === 0) return null; - - const lines = [ - '#### Largest Package Increases', - '', - '| Package | Base | Head | Δ | Modules Δ |', - '| --- | ---: | ---: | ---: | ---: |', - ]; - - for (const packageSummary of increases) { - lines.push(`| ${packageDisplayName(packageSummary)} | ${util.formatBytes(packageSummary.baseSourceBytes)} | ${util.formatBytes(packageSummary.sourceBytes)} | ${util.formatColoredDelta(packageSummary.sourceBytes - packageSummary.baseSourceBytes, v => util.formatBytes(v))} | ${util.formatColoredDelta(packageSummary.modules - packageSummary.baseModules, v => util.formatNumber(v))} |`); - } - - return lines.join('\n'); -} - -function renderNewLoadedModules(base: RuntimeLoadedJsFootprintReport, head: RuntimeLoadedJsFootprintReport) { - function moduleMap(report: RuntimeLoadedJsFootprintReport) { - const map = new Map(); - for (const moduleSummary of report.phases.afterRequest.modules) { - if (typeof moduleSummary.path !== 'string') continue; - map.set(moduleSummary.path, moduleSummary); - } - return map; - } - - const baseModules = moduleMap(base); - const headModules = moduleMap(head); - const newModules = [...headModules.values()] - .filter(moduleSummary => !baseModules.has(moduleSummary.path)) - .toSorted((a, b) => b.sourceBytes - a.sourceBytes) - .slice(0, 10); - - if (newModules.length === 0) return null; - - const lines = [ - '#### Largest Newly Loaded Modules', - '', - '| Module | Package | Loaded JS |', - '| --- | --- | ---: |', - ]; - - for (const moduleSummary of newModules) { - lines.push(`| \`${moduleSummary.path}\` | ${moduleSummary.package} | ${util.formatBytes(moduleSummary.sourceBytes)} |`); - } - - return lines.join('\n'); -} - -function renderJsFootprintSection(base: RuntimeLoadedJsFootprintReport, head: RuntimeLoadedJsFootprintReport) { - const lines = [ - '### Runtime Loaded JS Footprint', - '', - '
Click to show', - '', - renderJsFootprintMetricTable(base, head), - '', - //'#### Load Phase Breakdown', - //'', - //renderJsFootprintPhaseTable(base, head), - //'', - ]; - - for (const block of [ - renderNewExternalPackages(base, head), - renderLargestPackageIncreases(base, head), - renderNewLoadedModules(base, head), - ]) { - if (block == null) continue; - lines.push(block); - lines.push(''); - } - - lines.push('
'); - lines.push(''); - - return lines.join('\n'); -} - -const base = JSON.parse(await readFile(baseFile, 'utf8')) as MemoryReport; -const head = JSON.parse(await readFile(headFile, 'utf8')) as MemoryReport; -const baseJsFootprint = JSON.parse(await readFile(baseJsFootprintFile, 'utf8')) as RuntimeLoadedJsFootprintReport; -const headJsFootprint = JSON.parse(await readFile(headJsFootprintFile, 'utf8')) as RuntimeLoadedJsFootprintReport; -const lines = [ - '## ⚙️ Backend Memory Usage Report', - '', -]; - -//const summary = measurementSummary(base, head); -//if (summary != null) { -// lines.push(summary); -// lines.push(''); -//} - -for (const phase of memoryReportPhases) { - lines.push(`### ${phase.title}`); - lines.push(renderMainTableForPhase(base, head, phase.key)); - lines.push(''); -} - -const heapSnapshotSection = renderHeapSnapshotSection(base, head); -if (heapSnapshotSection != null) { - lines.push(heapSnapshotSection); - lines.push(''); -} - -const artifactUrl = process.env.MK_MEMORY_HEAP_SNAPSHOT_ARTIFACT_URL_HEAD!.trim(); -lines.push(`[Download representative V8 heap snapshot (head)](${artifactUrl})`); -lines.push(''); - -const jsFootprintSection = renderJsFootprintSection(baseJsFootprint, headJsFootprint); -if (jsFootprintSection != null) { - lines.push(jsFootprintSection); - lines.push(''); -} - -function getWarningMetric(base: MemoryReport, head: MemoryReport) { - for (const metric of ['Pss', 'Private_Dirty', 'VmRSS'] as const) { - if (getMemoryValue(base, 'afterGc', metric) != null && getMemoryValue(head, 'afterGc', metric) != null) { - return metric; - } - } - return null; -} - -function getDiffPercent(base: MemoryReport, head: MemoryReport, phase: typeof memoryReportPhases[number]['key'], metric: typeof metrics[number]) { - const baseValue = getMemoryValue(base, phase, metric); - const headValue = getMemoryValue(head, phase, metric); - if (baseValue == null || headValue == null || baseValue <= 0) return null; - - return ((headValue - baseValue) * 100) / baseValue; -} - -function isBeyondSampleNoise(base: MemoryReport, head: MemoryReport, phase: typeof memoryReportPhases[number]['key'], metric: typeof metrics[number]) { - const baseValue = getMemoryValue(base, phase, metric); - const headValue = getMemoryValue(head, phase, metric); - if (baseValue == null || headValue == null) return false; - - const delta = headValue - baseValue; - if (delta <= 0) return false; - - const baseSpread = getSampleSpread(base, phase, metric); - const headSpread = getSampleSpread(head, phase, metric); - if (baseSpread == null || headSpread == null) return true; - - const combinedSpread = Math.hypot(baseSpread, headSpread); - return delta > combinedSpread * 3; -} - -const warningMetric = getWarningMetric(base, head); -const warningDiffPercent = warningMetric == null ? null : getDiffPercent(base, head, 'afterGc', warningMetric); -if (warningMetric != null && warningDiffPercent != null && warningDiffPercent > 5 && isBeyondSampleNoise(base, head, 'afterGc', warningMetric)) { - lines.push(`⚠️ **Warning**: Memory usage (${warningMetric}) has increased by more than 5% and exceeds the observed sample noise. Please verify this is not an unintended change.`); - lines.push(''); -} - -//lines.push(`[See workflow logs for details](https://github.com/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID})`); - -await writeFile(outputFile, `${lines.join('\n')}\n`); diff --git a/.github/scripts/frontend-js-size.mts b/.github/scripts/frontend-js-size.mts deleted file mode 100644 index 4d6dd0a0bc..0000000000 --- a/.github/scripts/frontend-js-size.mts +++ /dev/null @@ -1,503 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -import { promises as fs } from 'node:fs'; -import path from 'node:path'; -import * as util from './utility.mts'; - -const marker = ''; - -const locale = process.env.FRONTEND_JS_SIZE_LOCALE ?? 'ja-JP'; - -//function sharePercent(value, total) { -// if (total === 0) return '0%'; -// return Math.round((value / total) * 100) + '%'; -//} - -function escapeCell(value: string) { - return String(value).replaceAll('|', '\\|').replaceAll('\n', '
'); -} - -//function tableCell(value) { -// return String(value).replaceAll('|', '\\|').replaceAll('\r', ' ').replaceAll('\n', ' '); -//} - -//function code(value) { -// const sanitized = String(value).replaceAll('\r', ' ').replaceAll('\n', ' '); -// const backtickRuns = sanitized.match(/`+/g) ?? []; -// const fenceLength = Math.max(1, ...backtickRuns.map((run) => run.length + 1)); -// const fence = '`'.repeat(fenceLength); -// const padding = sanitized.startsWith('`') || sanitized.endsWith('`') ? ' ' : ''; -// -// return `${fence}${padding}${sanitized}${padding}${fence}`; -//} - -//function tableCode(value) { -// return tableCell(code(value)); -//} - -type Manifest = Record; - -type FileEntry = { - key: string; - displayName: string; - file: string; - size: number; -}; - -function entryDisplayName(entry: FileEntry) { - if (!entry) return ''; - return entry.displayName || entry.file; -} - -function findEntryKey(manifest: Manifest) { - const entries = Object.entries(manifest); - return entries.find(([key, chunk]) => key === 'src/_boot_.ts' || chunk.src === 'src/_boot_.ts')?.[0] - ?? entries.find(([, chunk]) => chunk.name === 'entry' && chunk.isEntry)?.[0] - ?? entries.find(([, chunk]) => chunk.isEntry)?.[0] - ?? null; -} - -function stableChunkKey(manifestKey: string, chunk: Manifest[string]) { - return chunk.src ?? (chunk.name ? `chunk:${chunk.name}` : manifestKey); -} - -function collectStartupKeys(manifest: Manifest) { - const entryKey = findEntryKey(manifest); - const keys = new Set(); - if (entryKey == null) return keys; - - function visit(key: string) { - if (keys.has(key)) return; - const chunk = manifest[key]; - if (!chunk || !chunk.file?.endsWith('.js')) return; - keys.add(stableChunkKey(key, chunk)); - for (const importKey of chunk.imports ?? []) { - visit(importKey); - } - } - - visit(entryKey); - return keys; -} - -async function resolveBuiltFile(outDir: string, file: string) { - if (file.startsWith('scripts/')) { - const localizedFile = file.slice('scripts/'.length); - const localizedPath = path.join(outDir, locale, localizedFile); - if (await util.fileExists(localizedPath)) { - return { - absolutePath: localizedPath, - relativePath: `${locale}/${localizedFile}`, - }; - } - - throw new Error(`Expected ${locale} localized chunk for ${file}, but ${localizedPath} was not found.`); - } - return { - absolutePath: path.join(outDir, file), - relativePath: file, - }; -} - -async function collectReport(repoDir: string) { - const outDir = path.join(repoDir, 'built/_frontend_vite_'); - const manifestPath = path.join(outDir, 'manifest.json'); - const manifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as Manifest; - const byKey = new Map(); - const byFile = new Set(); - - for (const [key, chunk] of Object.entries(manifest)) { - if (!chunk.file?.endsWith('.js')) continue; - const builtFile = await resolveBuiltFile(outDir, chunk.file); - const size = await util.fileSize(builtFile.absolutePath); - const stableKey = stableChunkKey(key, chunk); - const displayName = chunk.src ?? chunk.name ?? key; - byKey.set(stableKey, { - key: stableKey, - displayName, - file: builtFile.relativePath, - size, - }); - byFile.add(builtFile.relativePath); - } - - const localeDir = path.join(outDir, locale); - if (await util.fileExists(localeDir)) { - for await (const fullPath of util.traverseDirectory(localeDir)) { - if (!fullPath.endsWith('.js')) continue; - const relativePath = util.normalizePath(path.relative(outDir, fullPath)); - if (byFile.has(relativePath)) continue; - const size = await util.fileSize(fullPath); - byKey.set(relativePath, { - key: relativePath, - displayName: relativePath, - file: relativePath, - size, - }); - } - } - - return { - manifest, - chunks: Object.fromEntries(byKey), - startupKeys: [...collectStartupKeys(manifest)], - }; -} - -type VisualizerReport = { - nodeParts?: Record; - nodeMetas?: Record; - renderedLength: number; - gzipLength: number; - brotliLength: number; - }>; - options?: Record; -}; - - -function collectVisualizerReport(data: VisualizerReport) { - const nodeParts = data.nodeParts ?? {}; - const nodeMetas = Object.values(data.nodeMetas ?? {}); - const moduleRows = []; - const bundleMap = new Map(); - - for (const meta of nodeMetas) { - const row = { - id: meta.id, - bundles: 0, - renderedLength: 0, - gzipLength: 0, - brotliLength: 0, - importedByCount: meta.importedBy?.length ?? 0, - importedCount: meta.imported?.length ?? 0, - }; - - for (const [bundleId, partUid] of Object.entries(meta.moduleParts ?? {})) { - const part = nodeParts[partUid]; - if (part == null) continue; - - row.bundles += 1; - row.renderedLength += part.renderedLength; - row.gzipLength += part.gzipLength; - row.brotliLength += part.brotliLength; - - const bundle = bundleMap.get(bundleId) ?? { - id: bundleId, - modules: 0, - renderedLength: 0, - gzipLength: 0, - brotliLength: 0, - }; - bundle.modules += 1; - bundle.renderedLength += part.renderedLength; - bundle.gzipLength += part.gzipLength; - bundle.brotliLength += part.brotliLength; - bundleMap.set(bundleId, bundle); - } - - if (row.bundles > 0) { - moduleRows.push(row); - } - } - - let staticImports = 0; - let dynamicImports = 0; - for (const meta of nodeMetas) { - for (const imported of meta.imported ?? []) { - if (imported.dynamic) { - dynamicImports += 1; - } else { - staticImports += 1; - } - } - } - - const bundleRows = [...bundleMap.values()].sort((a, b) => b.renderedLength - a.renderedLength); - const hotModules = [...moduleRows].sort((a, b) => b.renderedLength - a.renderedLength); - const totalRendered = moduleRows.reduce((sum, row) => sum + row.renderedLength, 0); - const totalGzip = moduleRows.reduce((sum, row) => sum + row.gzipLength, 0); - const totalBrotli = moduleRows.reduce((sum, row) => sum + row.brotliLength, 0); - - return { - options: data.options ?? {}, - summary: { - bundles: bundleRows.length, - modules: moduleRows.length, - entries: nodeMetas.filter((meta) => meta.isEntry).length, - externals: nodeMetas.filter((meta) => meta.isExternal).length, - staticImports, - dynamicImports, - }, - metrics: { - renderedLength: totalRendered, - gzipLength: totalGzip, - brotliLength: totalBrotli, - }, - hotModules, - }; -} - -function renderVisualizerSummaryTable(before: ReturnType, after: ReturnType) { - const summary = [ - 'bundles', - 'modules', - 'entries', - //'externals', - 'staticImports', - 'dynamicImports', - ] as const; - - const metrics = [ - 'renderedLength', - 'gzipLength', - 'brotliLength', - ] as const; - - return [ - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ``, - ...summary.map((key) => ``), - ...metrics.map((key) => ``), - ``, - ``, - ``, - ...summary.map((key) => ``), - ...metrics.map((key) => ``), - ``, - ``, - ``, - ``, - ...summary.map((key) => ``), - ...metrics.map((key) => ``), - ``, - ``, - ``, - ...summary.map((key) => ``), - ...metrics.map((key) => ``), - ``, - ``, - `
BundlesModulesEntriesImportsSize
StaticDynamicRenderedGzipBrotli
Before${util.formatNumber(before.summary[key])}${util.formatBytes(before.metrics[key])}
After${util.formatNumber(after.summary[key])}${util.formatBytes(after.metrics[key])}
Δ${util.calcAndFormatDeltaNumber(before.summary[key], after.summary[key], 0)}${util.calcAndFormatDeltaBytes(before.metrics[key], after.metrics[key], 1000)}
Δ (%)${util.calcAndFormatDeltaPercent(before.summary[key], after.summary[key], 0.1)}${util.calcAndFormatDeltaPercent(before.metrics[key], after.metrics[key], 0.1)}
`, - ]; -} - -function getChunkComparisonRows(keys: string[], before: Awaited>, after: Awaited>) { - return keys.map((key) => { - const beforeEntry = before.chunks[key]; - const afterEntry = after.chunks[key]; - const beforeSize = beforeEntry?.size ?? 0; - const afterSize = afterEntry?.size ?? 0; - return { - key, - name: entryDisplayName(beforeEntry ?? afterEntry), - chunkFile: beforeEntry?.file ?? afterEntry?.file, - beforeSize, - afterSize, - changeType: beforeEntry == null ? 'added' : afterEntry == null ? 'removed' : beforeSize !== afterSize ? 'updated' : 'unchanged', - sortSize: Math.max(beforeSize, afterSize), - }; - }); -} - -function summarizeChunkChanges(rows: ReturnType) { - return { - updated: rows.filter((row) => row.changeType === 'updated').length, - added: rows.filter((row) => row.changeType === 'added').length, - removed: rows.filter((row) => row.changeType === 'removed').length, - }; -} - -function formatChunkChangeSummary(label: string, summary: ReturnType) { - return `${label} (${summary.updated} updated, ${summary.added} added, ${summary.removed} removed)`; -} - -function compareChunkComparisonRows(a: ReturnType[number], b: ReturnType[number]) { - return Math.abs(b.afterSize - b.beforeSize) - Math.abs(a.afterSize - a.beforeSize) - || (b.afterSize - b.beforeSize) - (a.afterSize - a.beforeSize) - || b.sortSize - a.sortSize - || a.name.localeCompare(b.name); -} - -function chunkMarkdownTable(rows: ReturnType, total?: { beforeSize: number; afterSize: number }) { - if (rows.length === 0) return '_No data_'; - - const lines = [ - '| Chunk | Before | After | Δ | Δ (%) |', - '| --- | ---: | ---: | ---: | ---: |', - ]; - if (total != null) { - lines.push(`| (total) | ${util.formatBytes(total.beforeSize)} | ${util.formatBytes(total.afterSize)} | ${util.calcAndFormatDeltaBytes(total.beforeSize, total.afterSize, 1000)} | ${util.calcAndFormatDeltaPercent(total.beforeSize, total.afterSize, 0.1).replaceAll('\\%', '\\\\%')} |`); - lines.push('| | | | | |'); - } - for (const row of rows) { - if (row.changeType === 'added') { - lines.push(`|
\`${escapeCell(row.name)}\` \`${escapeCell(row.chunkFile)}\`
| ${util.formatBytes(row.beforeSize)} | ${util.formatBytes(row.afterSize)} | ${util.calcAndFormatDeltaBytes(row.beforeSize, row.afterSize, 1000)} | $\\color{orange}{\\text{( + )}}$ |`); - } else if (row.changeType === 'removed') { - lines.push(`|
\`${escapeCell(row.name)}\` \`${escapeCell(row.chunkFile)}\`
| ${util.formatBytes(row.beforeSize)} | ${util.formatBytes(row.afterSize)} | ${util.calcAndFormatDeltaBytes(row.beforeSize, row.afterSize, 1000)} | $\\color{green}{\\text{( - )}}$ |`); - } else { - lines.push(`|
\`${escapeCell(row.name)}\` \`${escapeCell(row.chunkFile)}\`
| ${util.formatBytes(row.beforeSize)} | ${util.formatBytes(row.afterSize)} | ${util.calcAndFormatDeltaBytes(row.beforeSize, row.afterSize, 1000)} | ${util.calcAndFormatDeltaPercent(row.beforeSize, row.afterSize, 0.1).replaceAll('\\%', '\\\\%')} |`); - } - } - return lines.join('\n'); -} - -function renderFrontendChunkReport(before: Awaited>, after: Awaited>) { - const commonChunkKeys = Object.keys(before.chunks).filter((key) => after.chunks[key] != null); - const addedChunkKeys = Object.keys(after.chunks).filter((key) => before.chunks[key] == null); - const removedChunkKeys = Object.keys(before.chunks).filter((key) => after.chunks[key] == null); - const allChunkKeys = [ - ...commonChunkKeys, - ...addedChunkKeys, - ...removedChunkKeys, - ]; - //const comparisonRows = getChunkComparisonRows(commonChunkKeys, before, after); - const allComparisonRows = getChunkComparisonRows(allChunkKeys, before, after); - - const changedRows = allComparisonRows.filter((row) => row.changeType !== 'unchanged'); - const diffSummary = summarizeChunkChanges(changedRows); - const diffTotal = { - beforeSize: allComparisonRows.reduce((sum, row) => sum + row.beforeSize, 0), - afterSize: allComparisonRows.reduce((sum, row) => sum + row.afterSize, 0), - }; - const diffRows = changedRows.sort(compareChunkComparisonRows).slice(0, 30); // TODO: 実際に30を超えて切り捨てられたrowがあった場合はその旨をmarkdown内に表示するようにする - - const startupKeys = new Set([ - ...before.startupKeys, - ...after.startupKeys, - ]); - const startupComparisonRows = getChunkComparisonRows([...startupKeys], before, after); - const startupRows = startupComparisonRows.sort(compareChunkComparisonRows); - const startupSummary = summarizeChunkChanges(startupComparisonRows); - const startupTotal = { - beforeSize: startupComparisonRows.reduce((sum, row) => sum + row.beforeSize, 0), - afterSize: startupComparisonRows.reduce((sum, row) => sum + row.afterSize, 0), - }; - - //const largeRows = comparisonRows - // .sort((a, b) => b.sortSize - a.sortSize || a.name.localeCompare(b.name)) - // .slice(0, 30); - - return [ - '
', - `${formatChunkChangeSummary('Chunk size diff', diffSummary)}`, - '', - chunkMarkdownTable(diffRows, diffTotal), - '', - '
', - '', - '
', - `${formatChunkChangeSummary('Startup chunk size', startupSummary)}`, - '', - chunkMarkdownTable(startupRows, startupTotal), - '', - `_Startup chunks are the Vite entry for \`src/_boot_.ts\` and its static imports._`, - '', - '
', - '', - //'
', - //`Largest`, - //'', - //markdownTable(largeRows), - //'', - //'
', - //'', - ].join('\n'); -} - -function renderFrontendBundleReport(before: ReturnType, after: ReturnType) { - const lines = [ - ...renderVisualizerSummaryTable(before, after), - '', - //'
', - //'Top 10', - //'', - ]; - - /* - for (const row of after.hotModules.slice(0, 10)) { - lines.push(`- ${code(row.id)}: ${sharePercent(row.renderedLength, after.metrics.renderedLength)} (${formatBytes(row.renderedLength)})`); - } - - lines.push( - '', - '
', - ); - - lines.push( - '', - '
', - 'Hot Modules (Self Size)', - '', - '| Module | Bundles | Rendered | Share | Gzip | Brotli | Imports | Imported By |', - '|---|---:|---:|---:|---:|---:|---:|---:|', - ); - - for (const row of after.hotModules.slice(0, 15)) { - lines.push(`| ${tableCode(row.id)} | ${row.bundles} | ${formatBytes(row.renderedLength)} | ${sharePercent(row.renderedLength, after.metrics.renderedLength)} | ${formatBytes(row.gzipLength)} | ${formatBytes(row.brotliLength)} | ${row.importedCount} | ${row.importedByCount} |`); - } - - lines.push( - '', - '
', - ); - */ - - return lines.join('\n'); -} - -const args = process.argv.slice(2); -const [beforeDir, afterDir, beforeStatsFile, afterStatsFile, outFile] = args; -const before = await collectReport(beforeDir); -const after = await collectReport(afterDir); -const beforeStats = JSON.parse(await fs.readFile(beforeStatsFile, 'utf8')) as VisualizerReport; -const afterStats = JSON.parse(await fs.readFile(afterStatsFile, 'utf8')) as VisualizerReport; -const beforeVisualizerReport = collectVisualizerReport(beforeStats); -const afterVisualizerReport = collectVisualizerReport(afterStats); -const visualizerArtifactLink = `[Open treemap HTML](${process.env.FRONTEND_BUNDLE_REPORT_ARTIFACT_URL})`; - -const body = [ - marker, - '', - `## 📦 Frontend Bundle Report`, - '', - renderFrontendChunkReport(before, after), - '', - '## Bundle Stats', - '', - renderFrontendBundleReport(beforeVisualizerReport, afterVisualizerReport), - '', - visualizerArtifactLink, -].join('\n'); - -await fs.writeFile(outFile, body); diff --git a/.github/scripts/heap-snapshot-util.mts b/.github/scripts/heap-snapshot-util.mts deleted file mode 100644 index c99ce5f441..0000000000 --- a/.github/scripts/heap-snapshot-util.mts +++ /dev/null @@ -1,200 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -// NOTE: このファイルはworkflow上でバックエンドからも参照されるため、side effectがあってはならない - -import * as util from './utility.mts'; - -export const heapSnapshotCategory = { - total: { label: 'Total', color: 'gray', colorHex: '#888888' }, - code: { label: 'Code', color: 'orange', colorHex: '#f28e2c' }, - strings: { label: 'Strings', color: 'red', colorHex: '#e15759' }, - jsArrays: { label: 'JS arrays', color: 'cyan', colorHex: '#76b7b2' }, - typedArrays: { label: 'Typed arrays', color: 'green', colorHex: '#59a14f' }, - systemObjects: { label: 'System objects', color: 'yellow', colorHex: '#edc949' }, - otherJsObjects: { label: 'Other JS objs', color: 'violet', colorHex: '#af7aa1' }, - otherNonJsObjects: { label: 'Other non-JS objs', color: 'pink', colorHex: '#ff9da7' }, -} as const satisfies Record; - -export type HeapSnapshotData = { - categories: Record; - nodeCounts: Record; - breakdowns?: Record>; -}; - -export type HeapSnapshotReport = { - summary: HeapSnapshotData; - samples: { - round: number; - data: HeapSnapshotData; - }[]; -}; - -function getHeapSnapshotCategoryValue(report: HeapSnapshotReport, category: keyof typeof heapSnapshotCategory) { - return report.summary.categories[category]; -} - -export function renderHeapSnapshotTable(base: HeapSnapshotReport, head: HeapSnapshotReport) { - const lines = [ - '| Metric | Base | Head | Δ median | Δ MAD | Δ min | Δ max |', - '| --- | ---: | ---: | ---: | ---: | ---: | ---: |', - ]; - const baseTotal = getHeapSnapshotCategoryValue(base, 'total'); - const headTotal = getHeapSnapshotCategoryValue(head, 'total'); - - function getHeapSnapshotSampleSpread(report: HeapSnapshotReport, category: keyof typeof heapSnapshotCategory) { - const values = report.samples - .map(sample => sample.data.categories[category]) - .filter(value => Number.isFinite(value)) as number[]; - if (values.length < 2) throw new Error(`Not enough samples for category ${category}`); - - const center = util.median(values); - return util.median(values.map(value => Math.abs(value - center))); - } - - for (const category of Object.keys(heapSnapshotCategory) as (keyof typeof heapSnapshotCategory)[]) { - const baseValue = getHeapSnapshotCategoryValue(base, category); - const headValue = getHeapSnapshotCategoryValue(head, category); - const baseSpread = getHeapSnapshotSampleSpread(base, category); - const headSpread = getHeapSnapshotSampleSpread(head, category); - const summary = util.pairedDeltaSummary(base.samples, head.samples, (sample) => sample.data.categories[category]); - const percent = summary.median * 100 / baseValue; - - if (category === 'total') { - const deltaMedian = `${util.formatDeltaBytes(summary.median, 100000)}
${util.formatDeltaPercent(percent, 0.1).replaceAll('\\%', '\\\\%')}`; - const baseText = `${util.formatBytes(baseValue)}
± ${util.formatBytes(baseSpread)}`; - const headText = `${util.formatBytes(headValue)}
± ${util.formatBytes(headSpread)}`; - const metricText = `$\\color{${heapSnapshotCategory[category].color}}{\\rule{8pt}{8pt}}$ **${heapSnapshotCategory[category].label}**`; - lines.push(`| ${metricText} | ${baseText} | ${headText} | ${deltaMedian} | ${util.formatBytes(summary.mad)} | ${util.formatDeltaBytes(summary.min, 100000)} | ${util.formatDeltaBytes(summary.max, 100000)} |`); - lines.push('| | | | | | | |'); - } else { - const deltaMedian = util.formatDeltaBytes(summary.median, 100000); - const baseText = util.formatBytes(baseValue); - const headText = util.formatBytes(headValue); - const basePercent = util.formatPercent((baseValue * 100) / baseTotal); - const headPercent = util.formatPercent((headValue * 100) / headTotal); - const metricText = `
$\\color{${heapSnapshotCategory[category].color}}{\\rule{8pt}{8pt}}$ **${heapSnapshotCategory[category].label}**${basePercent} → ${headPercent}
`; - lines.push(`| ${metricText} | ${baseText} | ${headText} | ${deltaMedian} | ${util.formatBytes(summary.mad)} | ${util.formatDeltaBytes(summary.min, 100000)} | ${util.formatDeltaBytes(summary.max, 100000)} |`); - } - } - - if (lines.length === 2) return null; - return lines.join('\n'); -} - -const heapSnapshotSankeyChildMinRatio = 0.3; -const heapSnapshotSankeyParentMinPercent = 10; - -function escapeCsvValue(value: string) { - return `"${String(value).replaceAll('"', '""')}"`; -} - -export function renderHeapSnapshotSankey(report: HeapSnapshotReport, title: string) { - const total = getHeapSnapshotCategoryValue(report, 'total'); - if (total == null || total <= 0) return null; - - function getHeapSnapshotBreakdownEntries(category: keyof typeof heapSnapshotCategory) { - const breakdown = report.summary.breakdowns?.[category]; - if (breakdown == null || typeof breakdown !== 'object') return []; - - return Object.entries(breakdown) - .filter(([, value]) => Number.isFinite(value) && value > 0) - .toSorted((a, b) => b[1] - a[1]); - } - - function formatHeapSnapshotSankeyChildLabel(label: string) { - return String(label).replace(/^[^:]+:\s*/, ''); - } - - function formatSankeyPercentValue(value: number) { - const rounded = Math.round(value * 100) / 100; - if (rounded === 0 && value > 0) return '0.01'; - if (Number.isInteger(rounded)) return String(rounded); - return rounded.toFixed(2).replace(/0+$/, '').replace(/\.$/, ''); - } - - const categories = (Object.keys(heapSnapshotCategory) as (keyof typeof heapSnapshotCategory)[]) - .filter(category => category !== 'total') - .map(category => { - const value = getHeapSnapshotCategoryValue(report, category); - if (value == null || value <= 0) return null; - const breakdownEntries = getHeapSnapshotBreakdownEntries(category); - const breakdownTotal = breakdownEntries.reduce((sum, [, childValue]) => sum + childValue, 0); - const percent = (value * 100) / total; - const childEntries = []; - let otherPercent = 0; - - if (breakdownTotal > 0 && percent > heapSnapshotSankeyParentMinPercent) { - for (const [childName, childValue] of breakdownEntries) { - const childRatio = childValue / breakdownTotal; - const childPercent = percent * childRatio; - if (childRatio >= heapSnapshotSankeyChildMinRatio) { - childEntries.push([formatHeapSnapshotSankeyChildLabel(childName), childPercent]); - } else { - otherPercent += childPercent; - } - } - - if (childEntries.length > 0 && otherPercent > 0) { - childEntries.push(['Other', otherPercent]); - } - } - - return { - category, - percent, - childEntries, - }; - }) - .filter(value => value != null); - - if (categories.length === 0) return null; - - const nodeColors = { - [title]: heapSnapshotCategory.total.colorHex, - } as Record; - for (const { category, childEntries } of categories) { - const categoryColor = heapSnapshotCategory[category].colorHex; - nodeColors[category] = categoryColor; - - for (const [childName] of childEntries) { - nodeColors[childName] = categoryColor; - } - } - - const lines = [ - `
${title} heap snapshot composition`, - '', - '```mermaid', - `%%{init: ${JSON.stringify({ - sankey: { - showValues: false, - linkColor: 'target', - labelStyle: 'outlined', - nodeAlignment: 'center', - nodePadding: 10, - nodeColors: { - ...nodeColors, - 'Other': '#888888', - }, - }, - })}}%%`, - 'sankey-beta', - ]; - - for (const { category, percent, childEntries } of categories) { - lines.push(`${escapeCsvValue(title)},${escapeCsvValue(heapSnapshotCategory[category].label)},${formatSankeyPercentValue(percent)}`); - - for (const [childName, childPercent] of childEntries) { - lines.push(`${escapeCsvValue(heapSnapshotCategory[category].label)},${escapeCsvValue(childName)},${formatSankeyPercentValue(childPercent)}`); - } - } - - lines.push('```'); - lines.push(''); - lines.push('
'); - - return lines.join('\n'); -} diff --git a/.github/scripts/measure-backend-memory-comparison.mts b/.github/scripts/measure-backend-memory-comparison.mts deleted file mode 100644 index 3dce74dc0a..0000000000 --- a/.github/scripts/measure-backend-memory-comparison.mts +++ /dev/null @@ -1,309 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -import { createRequire } from 'node:module'; -import { copyFile, rm, writeFile } from 'node:fs/promises'; -import { join, resolve } from 'node:path'; -import * as util from './utility.mts'; -import * as heapSnapshotUtil from './heap-snapshot-util.mts'; -import type { MemoryReportRaw } from '../../packages/backend/scripts/measure-memory.mts'; - -const phases = ['afterGc'] as const; - -export type MemoryReport = { - timestamp: string; - sampleCount: any; - aggregation: string; - measurement: { - startupTimeoutMs: any; - memorySettleTimeMs: any; - ipcTimeoutMs: any; - requestCount: any; - heapSnapshot: { - enabled: any; - timeoutMs: any; - breakdownTopN: any; - }; - }; - summary: Record; - heapSnapshot?: heapSnapshotUtil.HeapSnapshotData; - }>; - samples: (MemoryReportRaw['samples'][number] & { - round: number; - })[]; -}; - -const [baseDirArg, headDirArg, baseOutputArg, headOutputArg] = process.argv.slice(2); - -const HEAP_SNAPSHOT_BREAKDOWN_TOP_N = util.readIntegerEnv('MK_MEMORY_HEAP_SNAPSHOT_BREAKDOWN_TOP_N', 6, 1); -const HEAD_HEAP_SNAPSHOT_WORK_DIR = resolve('head-heap-snapshots'); -const HEAD_HEAP_SNAPSHOT_OUTPUT_PATH = resolve('head-heap-snapshot.heapsnapshot'); - -async function resetState(repoDir: string) { - const require = createRequire(join(repoDir, 'packages/backend/package.json')); - const pg = require('pg'); - const Redis = require('ioredis'); - - const postgres = new pg.Client({ - host: '127.0.0.1', - port: 54312, - database: 'postgres', - user: 'postgres', - }); - - await postgres.connect(); - try { - await postgres.query('DROP DATABASE IF EXISTS "test-misskey" WITH (FORCE)'); - await postgres.query('CREATE DATABASE "test-misskey"'); - } finally { - await postgres.end(); - } - - const redis = new Redis({ host: '127.0.0.1', port: 56312 }); - try { - await redis.flushall(); - } finally { - redis.disconnect(); - } -} - -function summarizeHeapSnapshotBreakdowns(samples: MemoryReport['samples'], phase: typeof phases[number]) { - const breakdowns = {} as Record>; - - for (const category of Object.keys(heapSnapshotUtil.heapSnapshotCategory) as (keyof typeof heapSnapshotUtil.heapSnapshotCategory)[]) { - if (category === 'total') continue; - - const childKeys = new Set(); - for (const sample of samples) { - for (const childKey of Object.keys(sample.phases[phase].heapSnapshot?.breakdowns?.[category] ?? {})) { - childKeys.add(childKey); - } - } - - const categoryBreakdown = {} as Record; - for (const childKey of childKeys) { - const values = samples - .map(sample => sample.phases[phase].heapSnapshot?.breakdowns?.[category]?.[childKey]) - .filter(value => Number.isFinite(value)) as number[]; - - if (values.length > 0) categoryBreakdown[childKey] = util.median(values); - } - - if (Object.keys(categoryBreakdown).length > 0) { - breakdowns[category] = collapseHeapSnapshotBreakdown(categoryBreakdown); - } - } - - return breakdowns; -} - -function collapseHeapSnapshotBreakdown(breakdown: Record) { - const entries = Object.entries(breakdown) - .filter(([, value]) => value > 0) - .toSorted((a, b) => b[1] - a[1]); - - const topEntries = entries.slice(0, HEAP_SNAPSHOT_BREAKDOWN_TOP_N); - const otherValue = entries - .slice(HEAP_SNAPSHOT_BREAKDOWN_TOP_N) - .reduce((sum, [, value]) => sum + value, 0); - - const collapsed = Object.fromEntries(topEntries); - if (otherValue > 0) collapsed.Other = otherValue; - return collapsed; -} - -function summarizeSamples(samples: MemoryReport['samples']) { - const summary = {} as MemoryReport['summary']; - - for (const phase of phases) { - summary[phase] = { - memoryUsage: {}, - }; - - const metricKeys = new Set(); - for (const sample of samples) { - for (const key of Object.keys(sample.phases[phase].memoryUsage)) { - metricKeys.add(key); - } - } - - for (const key of metricKeys) { - const values = samples.map(sample => sample.phases[phase].memoryUsage[key]); - summary[phase].memoryUsage[key] = util.median(values); - } - - const heapSnapshotCategoryValues = {} as Record; - for (const category of Object.keys(heapSnapshotUtil.heapSnapshotCategory) as (keyof typeof heapSnapshotUtil.heapSnapshotCategory)[]) { - const values = samples - .map(sample => sample.phases[phase].heapSnapshot?.categories?.[category]) - .filter(value => Number.isFinite(value)) as number[]; - - if (values.length > 0) heapSnapshotCategoryValues[category] = util.median(values); - } - - const heapSnapshotNodeCountValues = {} as Record; - for (const category of Object.keys(heapSnapshotUtil.heapSnapshotCategory) as (keyof typeof heapSnapshotUtil.heapSnapshotCategory)[]) { - const values = samples - .map(sample => sample.phases[phase].heapSnapshot?.nodeCounts?.[category]) - .filter(value => Number.isFinite(value)) as number[]; - - if (values.length > 0) heapSnapshotNodeCountValues[category] = util.median(values); - } - - if (Object.keys(heapSnapshotCategoryValues).length > 0) { - const heapSnapshotBreakdowns = summarizeHeapSnapshotBreakdowns(samples, phase); - - summary[phase].heapSnapshot = { - categories: heapSnapshotCategoryValues, - nodeCounts: heapSnapshotNodeCountValues, - ...(Object.keys(heapSnapshotBreakdowns).length > 0 ? { breakdowns: heapSnapshotBreakdowns } : {}), - }; - } - } - - return summary; -} - -async function measureRepo(label: string, repoDir: string, round: number, options: { heapSnapshotSavePath?: string } = {}) { - process.stderr.write(`[${label}] Resetting database and Redis\n`); - await resetState(repoDir); - - process.stderr.write(`[${label}] Running migrations\n`); - await util.run('pnpm', ['--filter', 'backend', 'migrate'], { - cwd: repoDir, - env: process.env, - logStdout: true, - }); - - process.stderr.write(`[${label}] Measuring memory\n`); - const measureEnv = { - ...process.env, - MK_MEMORY_SAMPLE_COUNT: '1', - } as NodeJS.ProcessEnv; - if (round <= 0) measureEnv.MK_MEMORY_HEAP_SNAPSHOT = '0'; - if (options.heapSnapshotSavePath != null) measureEnv.MK_MEMORY_HEAP_SNAPSHOT_SAVE_PATH = options.heapSnapshotSavePath; - - const stdout = await util.run('node', ['packages/backend/scripts/measure-memory.mts'], { - cwd: repoDir, - env: measureEnv, - }); - - const report = JSON.parse(stdout) as MemoryReportRaw; - const sample = report.samples[0]; - - return sample; -} - -function headHeapSnapshotPath(round: number) { - return join(HEAD_HEAP_SNAPSHOT_WORK_DIR, `round-${round}.heapsnapshot`); -} - -function selectRepresentativeHeadHeapSnapshotRound(samples: MemoryReport['samples'], summary: MemoryReport['summary']) { - const medianTotal = summary.afterGc.heapSnapshot?.categories?.total; - if (medianTotal == null || !Number.isFinite(medianTotal)) return null; - - let selected: { round: number; distance: number } | null = null; - for (const sample of samples) { - const total = sample.phases.afterGc.heapSnapshot?.categories?.total; - if (total == null || !Number.isFinite(total)) continue; - - const distance = Math.abs(total - medianTotal); - if (selected == null || distance < selected.distance || (distance === selected.distance && sample.round < selected.round)) { - selected = { - round: sample.round, - distance, - }; - } - } - - return selected?.round ?? null; -} - -async function saveRepresentativeHeadHeapSnapshot(samples: MemoryReport['samples'], summary: MemoryReport['summary']) { - const round = selectRepresentativeHeadHeapSnapshotRound(samples, summary); - if (round == null) return; - - await copyFile(headHeapSnapshotPath(round), HEAD_HEAP_SNAPSHOT_OUTPUT_PATH); - process.stderr.write(`Selected head heap snapshot round ${round} for artifact\n`); - await rm(HEAD_HEAP_SNAPSHOT_WORK_DIR, { recursive: true, force: true }); -} - -async function main() { - const baseDir = resolve(baseDirArg); - const headDir = resolve(headDirArg); - const baseOutput = resolve(baseOutputArg); - const headOutput = resolve(headOutputArg); - const rounds = util.readIntegerEnv('MK_MEMORY_COMPARE_ROUNDS', 5, 1); - const warmupRounds = util.readIntegerEnv('MK_MEMORY_COMPARE_WARMUP_ROUNDS', 1, 0); - const startedAt = new Date().toISOString(); - - await rm(HEAD_HEAP_SNAPSHOT_WORK_DIR, { recursive: true, force: true }); - await rm(HEAD_HEAP_SNAPSHOT_OUTPUT_PATH, { force: true }); - - const reports = { - base: { - dir: baseDir, - samples: [] as MemoryReport['samples'], - }, - head: { - dir: headDir, - samples: [] as MemoryReport['samples'], - }, - }; - - for (let round = 1; round <= warmupRounds; round++) { - process.stderr.write(`Starting warmup round ${round}/${warmupRounds}\n`); - for (const label of ['base', 'head'] as const) { - await measureRepo(label, reports[label].dir, -round); - } - } - - for (let round = 1; round <= rounds; round++) { - const order = round % 2 === 1 ? ['base', 'head'] as const : ['head', 'base'] as const; - process.stderr.write(`Starting measurement round ${round}/${rounds}: ${order.join(' -> ')}\n`); - - for (const label of order) { - const shouldSaveHeadHeapSnapshot = label === 'head'; - const options = shouldSaveHeadHeapSnapshot - ? { heapSnapshotSavePath: headHeapSnapshotPath(round) } - : {}; - const sample = await measureRepo(label, reports[label].dir, round, options); - reports[label].samples.push({ - ...sample, - round, - }); - } - } - - const summaries = { - base: summarizeSamples(reports.base.samples), - head: summarizeSamples(reports.head.samples), - }; - await saveRepresentativeHeadHeapSnapshot(reports.head.samples, summaries.head); - - for (const label of ['base', 'head'] as const) { - const report = { - timestamp: new Date().toISOString(), - sampleCount: reports[label].samples.length, - aggregation: 'median', - comparison: { - strategy: 'interleaved-pairs', - rounds, - warmupRounds, - startedAt, - }, - summary: summaries[label], - samples: reports[label].samples, - }; - - await writeFile(label === 'base' ? baseOutput : headOutput, `${JSON.stringify(report, null, 2)}\n`); - } -} - -main().catch(err => { - console.error(err); - process.exit(1); -}); diff --git a/.github/scripts/utility.mts b/.github/scripts/utility.mts deleted file mode 100644 index fb65d9191d..0000000000 --- a/.github/scripts/utility.mts +++ /dev/null @@ -1,204 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -// NOTE: このファイルはworkflow上でバックエンドからも参照されるため、side effectがあってはならない - -import { spawn } from 'node:child_process'; -import { promises as fs } from 'node:fs'; -import path from 'node:path'; - -export function median(values: number[]) { - const sorted = values.toSorted((a, b) => a - b); - const center = Math.floor(sorted.length / 2); - if (sorted.length % 2 === 1) return sorted[center]; - return Math.round((sorted[center - 1] + sorted[center]) / 2); -} - -export function mad(values: number[]) { - if (values.length < 2) throw new Error('Not enough samples to calculate MAD'); - - const center = median(values); - return median(values.map(value => Math.abs(value - center))); -} - -function getSamplesByRound(samples: T) { - const samplesByRound = new Map(); - for (const sample of samples) { - if (sample.round <= 0) continue; - samplesByRound.set(sample.round, sample); - } - return samplesByRound; -} - -export function pairedDeltaSummary(baseSamples: T, headSamples: T, getValue: (sample: T[number]) => number | null) { - const baseSamplesByRound = getSamplesByRound(baseSamples); - const headSamplesByRound = getSamplesByRound(headSamples); - const values = []; - - for (const [round, baseSample] of baseSamplesByRound) { - const headSample = headSamplesByRound.get(round); - if (headSample == null) continue; - - const baseValue = getValue(baseSample); - const headValue = getValue(headSample); - if (baseValue == null || headValue == null) continue; - - values.push(headValue - baseValue); - } - - return { - median: median(values), - mad: mad(values), - min: Math.min(...values), - max: Math.max(...values), - samples: values.length, - }; -} - -export function normalizePath(filePath: string) { - return filePath.split(path.sep).join('/'); -} - -export async function fileExists(filePath: string) { - try { - await fs.access(filePath); - return true; - } catch { - return false; - } -} - -export async function fileSize(filePath: string) { - const stat = await fs.stat(filePath); - return stat.size; -} - -export async function* traverseDirectory(dir: string): AsyncGenerator { - for (const entry of await fs.readdir(dir, { withFileTypes: true })) { - const fullPath = path.join(dir, entry.name); - if (entry.isDirectory()) { - yield* traverseDirectory(fullPath); - } else if (entry.isFile()) { - yield fullPath; - } - } -} - -export function escapeLatex(text: string) { - return text - .replaceAll('\\', '\\\\') - .replaceAll('{', '\\{') - .replaceAll('}', '\\}') - .replaceAll('%', '\\%'); -} - -export function formatColoredDelta(delta: number, text: (value: number) => string, colorThreshold = 0) { - if (delta === 0) return text(0); - const sign = delta > 0 ? '+' : '-'; - if (Math.abs(delta) < colorThreshold) return `$\\text{${sign}${escapeLatex(text(Math.abs(delta)))}}$`; - const color = delta > 0 ? 'orange' : 'green'; - return `$\\color{${color}}{\\text{${sign}${escapeLatex(text(Math.abs(delta)))}}}$`; -} - -const numberFormatter = new Intl.NumberFormat('en-US', { - maximumFractionDigits: 1, -}); - -export function formatNumber(value: number) { - return numberFormatter.format(value); -} - -export function formatBytes(value: number) { - if (value === 0) return '0 B'; - const units = ['B', 'KB', 'MB', 'GB']; - let unitIndex = 0; - let size = value; - while (size >= 1000 && unitIndex < units.length - 1) { - size /= 1000; - unitIndex += 1; - } - - const maximumFractionDigits = size >= 10 || unitIndex === 0 ? 0 : 1; - return `${numberFormatter.format(Number(size.toFixed(maximumFractionDigits)))} ${units[unitIndex]}`; -} - -export function calcAndFormatDeltaNumber(before: number, after: number, colorThreshold = 0) { - if (before == null || after == null) return '-'; - const delta = after - before; - return formatColoredDelta(delta, v => formatNumber(v), colorThreshold); -} - -export function formatDeltaBytes(deltaBytes: number, colorThreshold = 0) { - return formatColoredDelta(deltaBytes, v => formatBytes(v), colorThreshold); -} - -export function calcAndFormatDeltaBytes(before: number, after: number, colorThreshold = 0) { - if (before == null || after == null) return '-'; - const delta = after - before; - return formatDeltaBytes(delta, colorThreshold); -} - -export function formatPercent(value: number) { - return `${formatNumber(value)}%`; -} - -export function formatDeltaPercent(deltaPercent: number, colorThreshold = 0) { - return formatColoredDelta(deltaPercent, v => formatPercent(v), colorThreshold); -} - -export function calcAndFormatDeltaPercent(before: number, after: number, colorThreshold = 0) { - if (before == null || before === 0 || after == null || after === 0) return '-'; - const delta = after - before; - return formatDeltaPercent(delta / before * 100, colorThreshold); -} - -export function commandName(command: string) { - if (process.platform !== 'win32') return command; - if (command === 'pnpm') return 'pnpm.cmd'; - return command; -} - -export function readIntegerEnv(name: string, defaultValue: number, min: number) { - const rawValue = process.env[name]; - if (rawValue == null || rawValue === '') return defaultValue; - if (!/^\d+$/.test(rawValue)) throw new Error(`${name} must be an integer`); - - const value = Number(rawValue); - if (!Number.isSafeInteger(value) || value < min) throw new Error(`${name} must be >= ${min}`); - return value; -} - -export function run(command: string, args: string[], options: { cwd?: string; env?: NodeJS.ProcessEnv; logStdout?: boolean } = {}) { - return new Promise((resolvePromise, reject) => { - const child = spawn(commandName(command), args, { - cwd: options.cwd, - env: options.env, - stdio: ['ignore', 'pipe', 'pipe'], - }); - - let stdout = ''; - let stderr = ''; - - child.stdout.on('data', data => { - stdout += data; - if (options.logStdout) process.stderr.write(data); - }); - - child.stderr.on('data', data => { - stderr += data; - process.stderr.write(data); - }); - - child.on('error', reject); - - child.on('close', code => { - if (code === 0) { - resolvePromise(stdout); - } else { - reject(new Error(`${command} ${args.join(' ')} failed with exit code ${code}\n${stderr}`)); - } - }); - }); -} diff --git a/.github/workflows/api-misskey-js.yml b/.github/workflows/api-misskey-js.yml index 19987d2af4..09e9c8ab6e 100644 --- a/.github/workflows/api-misskey-js.yml +++ b/.github/workflows/api-misskey-js.yml @@ -16,13 +16,13 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Setup Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' diff --git a/.github/workflows/report-backend-memory.yml b/.github/workflows/backend-diagnostics.comment.yml similarity index 54% rename from .github/workflows/report-backend-memory.yml rename to .github/workflows/backend-diagnostics.comment.yml index 3343ec745e..51ac04e86a 100644 --- a/.github/workflows/report-backend-memory.yml +++ b/.github/workflows/backend-diagnostics.comment.yml @@ -1,10 +1,10 @@ -name: Report backend memory +name: Backend diagnostics (comment) on: workflow_run: types: [completed] workflows: - - Get backend memory usage # get-backend-memory.yml + - Backend diagnostics (inspect) # backend-diagnostics.inspect.yml jobs: compare-memory: @@ -17,7 +17,17 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 + + - name: Setup pnpm + uses: pnpm/action-setup@v6.0.9 + - name: Use Node.js + uses: actions/setup-node@v7.0.0 + with: + node-version-file: '.node-version' + cache: 'pnpm' + - name: Install dependencies + run: pnpm --filter diagnostics-backend... install --frozen-lockfile - name: Download artifacts uses: actions/download-artifact@v8 @@ -33,9 +43,9 @@ jobs: - name: Load PR Number id: load-pr-num run: echo "pr-number=$(cat artifacts/pr_number)" >> "$GITHUB_OUTPUT" - - name: Find head heap snapshot artifact - id: find-heap-snapshot-artifact - uses: actions/github-script@v8 + - name: Find heap snapshot artifacts + id: find-heap-snapshot-artifacts + uses: actions/github-script@v9 with: script: | const { owner, repo } = context.repo; @@ -45,15 +55,26 @@ jobs: repo, run_id, }); - const artifact = artifacts.find(artifact => artifact.name === 'backend-memory-head-heap-snapshot'); - if (artifact == null) return; - core.setOutput('url', `https://github.com/${owner}/${repo}/actions/runs/${run_id}/artifacts/${artifact.id}`); + const artifactNames = { + base: 'base-heap-snapshot.heapsnapshot', + head: 'head-heap-snapshot.heapsnapshot', + }; + for (const [label, artifactName] of Object.entries(artifactNames)) { + const artifact = artifacts.find(artifact => artifact.name === artifactName); + if (artifact == null) throw new Error(`Artifact not found: ${artifactName}`); + core.setOutput(`${label}-url`, `https://github.com/${owner}/${repo}/actions/runs/${run_id}/artifacts/${artifact.id}`); + } - id: build-comment name: Build memory comment env: - MK_MEMORY_HEAP_SNAPSHOT_ARTIFACT_URL_HEAD: ${{ steps.find-heap-snapshot-artifact.outputs.url }} - run: node .github/scripts/backend-memory-report.mts ./artifacts/memory-base.json ./artifacts/memory-head.json ./output.md ./artifacts/js-footprint-base.json ./artifacts/js-footprint-head.json + MK_MEMORY_HEAP_SNAPSHOT_ARTIFACT_URL_BASE: ${{ steps.find-heap-snapshot-artifacts.outputs.base-url }} + MK_MEMORY_HEAP_SNAPSHOT_ARTIFACT_URL_HEAD: ${{ steps.find-heap-snapshot-artifacts.outputs.head-url }} + run: >- + pnpm --filter diagnostics-backend run render-md + "$GITHUB_WORKSPACE/artifacts/memory-base.json" + "$GITHUB_WORKSPACE/artifacts/memory-head.json" + "$GITHUB_WORKSPACE/output.md" - uses: thollander/actions-comment-pull-request@v3 with: pr-number: ${{ steps.load-pr-num.outputs.pr-number }} @@ -64,6 +85,6 @@ jobs: if: failure() && steps.load-pr-num.outputs.pr-number with: pr-number: ${{ steps.load-pr-num.outputs.pr-number }} - comment-tag: show_memory_diff_error + comment-tag: show_memory_diff message: | An error occurred while comparing backend memory usage. See [workflow logs](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details. diff --git a/.github/workflows/get-backend-memory.yml b/.github/workflows/backend-diagnostics.inspect.yml similarity index 70% rename from .github/workflows/get-backend-memory.yml rename to .github/workflows/backend-diagnostics.inspect.yml index 400a81a7d9..c3b2e9e2b7 100644 --- a/.github/workflows/get-backend-memory.yml +++ b/.github/workflows/backend-diagnostics.inspect.yml @@ -1,5 +1,5 @@ -# this name is used in report-backend-memory.yml so be careful when change name -name: Get backend memory usage +# this name is used in backend-diagnostics.comment.yml so be careful when change name +name: Backend diagnostics (inspect) on: pull_request: @@ -9,14 +9,10 @@ on: paths: - packages/backend/** - packages/misskey-js/** - - .github/scripts/utility.mts - - .github/scripts/backend-memory-report.mts - - .github/scripts/measure-backend-memory-comparison.mts - - .github/scripts/backend-js-footprint.mjs - - .github/scripts/backend-js-footprint-loader.mjs - - .github/scripts/backend-js-footprint-require.cjs - - .github/workflows/get-backend-memory.yml - - .github/workflows/report-backend-memory.yml + - packages-private/diagnostics-shared/** + - packages-private/diagnostics-backend/** + - .github/workflows/backend-diagnostics.inspect.yml + - .github/workflows/backend-diagnostics.comment.yml jobs: get-memory-usage: @@ -39,13 +35,13 @@ jobs: steps: - name: Checkout base - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 with: ref: ${{ github.base_ref }} path: base submodules: true - name: Checkout head - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 with: ref: refs/pull/${{ github.event.number }}/merge path: head @@ -55,7 +51,7 @@ jobs: with: package_json_file: head/package.json - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: 'head/.node-version' cache: 'pnpm' @@ -91,22 +87,32 @@ jobs: working-directory: head run: pnpm build - name: Measure backend memory usage + working-directory: head env: - MK_MEMORY_COMPARE_ROUNDS: 5 + MK_MEMORY_COMPARE_ROUNDS: 15 MK_MEMORY_COMPARE_WARMUP_ROUNDS: 1 MK_MEMORY_HEAP_SNAPSHOT: 1 - run: node head/.github/scripts/measure-backend-memory-comparison.mts base head memory-base.json memory-head.json + MK_MEMORY_HEAP_SNAPSHOT_ROUNDS: 3 + # 計測ハーネスはhead側のものを両方に使うが、成果物はrunnerのworkspace直下に出す + MK_MEMORY_HEAP_SNAPSHOT_OUTPUT_DIR: ${{ github.workspace }} + run: >- + pnpm --filter diagnostics-backend run inspect + "$GITHUB_WORKSPACE/base" "$GITHUB_WORKSPACE/head" + "$GITHUB_WORKSPACE/memory-base.json" "$GITHUB_WORKSPACE/memory-head.json" + - name: Upload base heap snapshot + uses: actions/upload-artifact@v7 + with: + path: base-heap-snapshot.heapsnapshot + archive: false + if-no-files-found: error + retention-days: 7 - name: Upload head heap snapshot uses: actions/upload-artifact@v7 with: - name: backend-memory-head-heap-snapshot path: head-heap-snapshot.heapsnapshot + archive: false if-no-files-found: error retention-days: 7 - - name: Measure backend loaded JS footprint - run: | - node head/.github/scripts/backend-js-footprint.mjs base js-footprint-base.json - node head/.github/scripts/backend-js-footprint.mjs head js-footprint-head.json - name: Upload Artifact uses: actions/upload-artifact@v7 with: @@ -114,8 +120,6 @@ jobs: path: | memory-base.json memory-head.json - js-footprint-base.json - js-footprint-head.json save-pr-number: runs-on: ubuntu-latest diff --git a/.github/workflows/changelog-check.yml b/.github/workflows/changelog-check.yml index 318999f596..95fe48ab4c 100644 --- a/.github/workflows/changelog-check.yml +++ b/.github/workflows/changelog-check.yml @@ -12,11 +12,16 @@ jobs: steps: - name: Checkout head - uses: actions/checkout@v6.0.3 - - name: Setup Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/checkout@v7.0.0 + + - name: setup pnpm + uses: pnpm/action-setup@v6 + + - name: setup node + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' + cache: pnpm - name: Checkout base run: | @@ -27,17 +32,16 @@ jobs: git checkout origin/${{ github.base_ref }} CHANGELOG.md - name: Copy to Checker directory for CHANGELOG-base.md - run: cp _base/CHANGELOG.md scripts/changelog-checker/CHANGELOG-base.md + run: cp _base/CHANGELOG.md packages-private/changelog-checker/CHANGELOG-base.md - name: Copy to Checker directory for CHANGELOG-head.md - run: cp CHANGELOG.md scripts/changelog-checker/CHANGELOG-head.md + run: cp CHANGELOG.md packages-private/changelog-checker/CHANGELOG-head.md - name: diff continue-on-error: true run: diff -u CHANGELOG-base.md CHANGELOG-head.md - working-directory: scripts/changelog-checker + working-directory: packages-private/changelog-checker + + - name: Install dependencies + run: pnpm --filter changelog-checker... install --frozen-lockfile - - name: Setup Checker - run: npm install - working-directory: scripts/changelog-checker - name: Run Checker - run: npm run run - working-directory: scripts/changelog-checker + run: pnpm --filter changelog-checker check diff --git a/.github/workflows/check-misskey-js-autogen.yml b/.github/workflows/check-misskey-js-autogen.yml index c5350b62d2..631daa79c8 100644 --- a/.github/workflows/check-misskey-js-autogen.yml +++ b/.github/workflows/check-misskey-js-autogen.yml @@ -18,7 +18,7 @@ jobs: if: ${{ github.event.pull_request.mergeable == null || github.event.pull_request.mergeable == true }} steps: - name: checkout - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 with: submodules: true persist-credentials: false @@ -29,7 +29,7 @@ jobs: - name: setup node id: setup-node - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: pnpm @@ -66,7 +66,7 @@ jobs: if: ${{ github.event.pull_request.mergeable == null || github.event.pull_request.mergeable == true }} steps: - name: checkout - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 with: submodules: true persist-credentials: false diff --git a/.github/workflows/check-misskey-js-version.yml b/.github/workflows/check-misskey-js-version.yml index 71351972f4..1d703ad210 100644 --- a/.github/workflows/check-misskey-js-version.yml +++ b/.github/workflows/check-misskey-js-version.yml @@ -20,7 +20,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 - name: Check version run: | if [ "$(jq -r '.version' package.json)" != "$(jq -r '.version' packages/misskey-js/package.json)" ]; then diff --git a/.github/workflows/check-spdx-license-id.yml b/.github/workflows/check-spdx-license-id.yml index f75db131f0..ae0102b92c 100644 --- a/.github/workflows/check-spdx-license-id.yml +++ b/.github/workflows/check-spdx-license-id.yml @@ -12,7 +12,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 - name: Check run: | counter=0 @@ -44,7 +44,6 @@ jobs: } directories=( - "cypress/e2e" "packages/backend/migration" "packages/backend/src" "packages/backend/test" diff --git a/.github/workflows/check_copyright_year.yml b/.github/workflows/check_copyright_year.yml index 4b5e71c43a..a6ed53a0f8 100644 --- a/.github/workflows/check_copyright_year.yml +++ b/.github/workflows/check_copyright_year.yml @@ -10,7 +10,7 @@ jobs: check_copyright_year: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 - run: | if [ "$(grep Copyright COPYING | sed -e 's/.*2014-\([0-9]*\) .*/\1/g')" -ne "$(date +%Y)" ]; then echo "Please change copyright year!" diff --git a/.github/workflows/docker-develop.yml b/.github/workflows/docker-develop.yml index 983f664917..f7dd869e02 100644 --- a/.github/workflows/docker-develop.yml +++ b/.github/workflows/docker-develop.yml @@ -27,7 +27,7 @@ jobs: platform=${{ matrix.platform }} echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV - name: Check out the repo - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v4 - name: Log in to Docker Hub diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index 277c88894e..8cf0c2b666 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -32,7 +32,7 @@ jobs: platform=${{ matrix.platform }} echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV - name: Check out the repo - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v4 - name: Docker meta diff --git a/.github/workflows/dockle.yml b/.github/workflows/dockle.yml index 87256c000e..385cbede66 100644 --- a/.github/workflows/dockle.yml +++ b/.github/workflows/dockle.yml @@ -17,7 +17,7 @@ jobs: DOCKLE_VERSION: 0.4.15 steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 - name: Download and install dockle v${{ env.DOCKLE_VERSION }} run: | diff --git a/.github/workflows/frontend-bundle-report-comment.yml b/.github/workflows/frontend-bundle-report-comment.yml deleted file mode 100644 index b93f8a320f..0000000000 --- a/.github/workflows/frontend-bundle-report-comment.yml +++ /dev/null @@ -1,318 +0,0 @@ -name: frontend-bundle-report-comment - -on: - workflow_run: - workflows: - - frontend-bundle-report - types: - - completed - pull_request_target: - types: - - opened - - synchronize - - reopened - - ready_for_review - paths: - - packages/frontend/** - - packages/frontend-shared/** - - packages/frontend-builder/** - - packages/i18n/** - - packages/icons-subsetter/** - - packages/misskey-js/** - - packages/misskey-reversi/** - - packages/misskey-bubble-game/** - - package.json - - pnpm-lock.yaml - - pnpm-workspace.yaml - - .node-version - - .github/scripts/utility.mts - - .github/scripts/frontend-js-size.mts - - .github/workflows/frontend-bundle-report.yml - - .github/workflows/frontend-bundle-report-comment.yml - -permissions: - actions: read - contents: read - issues: write - pull-requests: write - -jobs: - comment: - name: Comment frontend bundle report - if: github.event_name == 'pull_request_target' || (github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success') - runs-on: ubuntu-latest - concurrency: - group: frontend-bundle-report-comment-${{ github.event.pull_request.number || github.event.workflow_run.id }} - cancel-in-progress: true - steps: - - name: Find bundle report run - if: github.event_name == 'pull_request_target' - id: find-report-run - uses: actions/github-script@v9 - with: - script: | - const workflow_id = 'frontend-bundle-report.yml'; - const artifactName = 'frontend-bundle-report'; - const headSha = context.payload.pull_request.head.sha; - const prNumber = context.payload.pull_request.number; - const pollIntervalMs = 30_000; - const timeoutMs = 90 * 60_000; - const startedAt = Date.now(); - const { owner, repo } = context.repo; - - async function listReportWorkflowRuns() { - const runsForHead = await github.paginate(github.rest.actions.listWorkflowRuns, { - owner, - repo, - workflow_id, - event: 'pull_request', - head_sha: headSha, - per_page: 100, - }); - - if (runsForHead.length > 0) { - return runsForHead; - } - - const recentRuns = await github.paginate(github.rest.actions.listWorkflowRuns, { - owner, - repo, - workflow_id, - event: 'pull_request', - per_page: 100, - }); - return recentRuns.filter((run) => - run.pull_requests?.some((pullRequest) => pullRequest.number === prNumber)); - } - - async function findReportRun() { - const runs = (await listReportWorkflowRuns()) - .sort((a, b) => new Date(b.updated_at) - new Date(a.updated_at)); - - for (const run of runs) { - if (run.status !== 'completed') continue; - if (run.conclusion !== 'success') { - core.warning(`Frontend bundle report run ${run.id} completed with conclusion: ${run.conclusion}`); - return { done: true, run: null }; - } - - const artifacts = await github.paginate(github.rest.actions.listWorkflowRunArtifacts, { - owner, - repo, - run_id: run.id, - per_page: 100, - }); - const report = artifacts.find((artifact) => artifact.name === artifactName && !artifact.expired); - if (report) return { done: true, run }; - - core.info(`Frontend bundle report run ${run.id} did not produce ${artifactName}.`); - return { done: true, run: null }; - } - - return { done: false, run: null }; - } - - while (Date.now() - startedAt < timeoutMs) { - const { done, run } = await findReportRun(); - if (run) { - core.info(`Found frontend bundle report on workflow run ${run.id}.`); - core.setOutput('run-id', String(run.id)); - return; - } - if (done) { - return; - } - - core.info('Waiting for frontend bundle report artifact...'); - await new Promise((resolve) => setTimeout(resolve, pollIntervalMs)); - } - - core.warning(`Timed out waiting for ${artifactName} from ${workflow_id} for ${headSha}.`); - - - name: Find bundle report artifact - if: github.event_name == 'workflow_run' - id: find-report-artifact - uses: actions/github-script@v9 - with: - script: | - const artifactName = 'frontend-bundle-report'; - const { owner, repo } = context.repo; - const artifacts = await github.paginate(github.rest.actions.listWorkflowRunArtifacts, { - owner, - repo, - run_id: context.payload.workflow_run.id, - per_page: 100, - }); - const report = artifacts.find((artifact) => artifact.name === artifactName && !artifact.expired); - if (report) { - core.setOutput('exists', 'true'); - } else { - core.info(`Workflow run ${context.payload.workflow_run.id} did not produce ${artifactName}.`); - core.setOutput('exists', 'false'); - } - - - name: Download bundle report from workflow_run - if: github.event_name == 'workflow_run' && steps.find-report-artifact.outputs.exists == 'true' - uses: actions/download-artifact@v8 - with: - name: frontend-bundle-report - path: ${{ runner.temp }}/frontend-bundle-report - github-token: ${{ github.token }} - repository: ${{ github.repository }} - run-id: ${{ github.event.workflow_run.id }} - - - name: Download bundle report from pull_request_target - if: github.event_name == 'pull_request_target' && steps.find-report-run.outputs.run-id != '' - uses: actions/download-artifact@v8 - with: - name: frontend-bundle-report - path: ${{ runner.temp }}/frontend-bundle-report - github-token: ${{ github.token }} - repository: ${{ github.repository }} - run-id: ${{ steps.find-report-run.outputs.run-id }} - - - name: Comment on pull request - if: (github.event_name == 'workflow_run' && steps.find-report-artifact.outputs.exists == 'true') || steps.find-report-run.outputs.run-id != '' - uses: actions/github-script@v9 - with: - github-token: ${{ secrets.FRONTEND_BUNDLE_REPORT_COMMENT_TOKEN || secrets.FRONTEND_JS_SIZE_COMMENT_TOKEN || secrets.FRONTEND_BUNDLE_VISUALIZER_COMMENT_TOKEN || github.token }} - script: | - const fs = require('node:fs'); - const path = require('node:path'); - - const jsSizeMarker = ''; - const visualizerMarker = ''; - const reportMarkers = [jsSizeMarker, visualizerMarker]; - const reportDir = path.join(process.env.RUNNER_TEMP, 'frontend-bundle-report'); - const jsSizeReportPath = path.join(reportDir, 'frontend-js-size-report.md'); - const prNumberPath = path.join(reportDir, 'pr-number.txt'); - const headShaPath = path.join(reportDir, 'head-sha.txt'); - const workflowRun = context.payload.workflow_run; - const pullRequest = context.payload.pull_request; - const eventHeadSha = workflowRun?.head_sha ?? pullRequest?.head?.sha ?? null; - const { owner, repo } = context.repo; - - if (!fs.existsSync(jsSizeReportPath)) { - core.setFailed('The frontend bundle report artifact does not contain frontend-js-size-report.md.'); - return; - } - - const artifactHeadSha = fs.existsSync(headShaPath) - ? fs.readFileSync(headShaPath, 'utf8').trim() - : null; - if (eventHeadSha != null && artifactHeadSha != null && artifactHeadSha !== eventHeadSha) { - core.info(`The artifact head SHA (${artifactHeadSha}) differs from the event head SHA (${eventHeadSha}). Using artifact metadata for PR validation.`); - } - const reportHeadSha = artifactHeadSha ?? eventHeadSha; - - const artifactPrNumber = fs.existsSync(prNumberPath) - ? Number(fs.readFileSync(prNumberPath, 'utf8').trim()) - : null; - let issue_number = null; - if (pullRequest != null) { - issue_number = pullRequest.number; - if (Number.isInteger(artifactPrNumber) && artifactPrNumber !== issue_number) { - core.setFailed(`The artifact pull request number (${artifactPrNumber}) does not match the event pull request number (${issue_number}).`); - return; - } - } else if (workflowRun != null) { - const associatedPullRequests = new Map(); - for (const pullRequest of workflowRun.pull_requests ?? []) { - if (Number.isInteger(pullRequest.number)) { - associatedPullRequests.set(pullRequest.number, pullRequest); - } - } - - if (reportHeadSha != null) { - const pullRequestsForCommit = await github.paginate(github.rest.repos.listPullRequestsAssociatedWithCommit, { - owner, - repo, - commit_sha: reportHeadSha, - per_page: 100, - }); - for (const pullRequest of pullRequestsForCommit) { - associatedPullRequests.set(pullRequest.number, pullRequest); - } - } - - if (Number.isInteger(artifactPrNumber) && associatedPullRequests.has(artifactPrNumber)) { - issue_number = artifactPrNumber; - } else if (Number.isInteger(artifactPrNumber) && associatedPullRequests.size === 0) { - issue_number = artifactPrNumber; - } else if (!Number.isInteger(artifactPrNumber) && associatedPullRequests.size === 1) { - issue_number = [...associatedPullRequests.keys()][0]; - } else if (Number.isInteger(artifactPrNumber)) { - core.setFailed(`The artifact pull request number (${artifactPrNumber}) is not associated with ${reportHeadSha}.`); - return; - } else { - core.setFailed(`Could not determine the pull request associated with ${reportHeadSha}.`); - return; - } - } else { - core.setFailed('Could not determine the pull request event for this report.'); - return; - } - - const currentPullRequest = await github.rest.pulls.get({ - owner, - repo, - pull_number: issue_number, - }); - const currentHeadSha = currentPullRequest.data.head?.sha; - if (reportHeadSha != null && currentHeadSha != null && reportHeadSha !== currentHeadSha) { - core.info(`The report head SHA (${reportHeadSha}) is not the current pull request head SHA (${currentHeadSha}). Skipping stale frontend bundle report.`); - return; - } - - const jsSizeReport = fs.readFileSync(jsSizeReportPath, 'utf8').trim(); - if (!jsSizeReport.includes(jsSizeMarker)) { - core.setFailed('The frontend JS size report is missing the expected marker.'); - return; - } - let body = `${jsSizeReport}\n`; - - const maxCommentLength = 65_000; - if (body.length > maxCommentLength) { - const reportLocation = workflowRun?.html_url != null - ? `[workflow run](${workflowRun.html_url})` - : 'workflow artifact'; - const footer = [ - '', - '', - `_Report truncated because it exceeded ${maxCommentLength.toLocaleString('en-US')} characters. See the ${reportLocation} for the full report._`, - ].join('\n'); - body = `${body.slice(0, maxCommentLength - footer.length)}${footer}`; - } - - const comments = await github.paginate(github.rest.issues.listComments, { - owner, - repo, - issue_number, - per_page: 100, - }); - const previousReports = comments.filter((comment) => - comment.user?.type === 'Bot' && reportMarkers.some((reportMarker) => comment.body?.includes(reportMarker))); - - if (previousReports.length > 0) { - const [previous, ...duplicates] = previousReports; - await github.rest.issues.updateComment({ - owner, - repo, - comment_id: previous.id, - body, - }); - for (const duplicate of duplicates) { - await github.rest.issues.deleteComment({ - owner, - repo, - comment_id: duplicate.id, - }); - } - } else { - await github.rest.issues.createComment({ - owner, - repo, - issue_number, - body, - }); - } diff --git a/.github/workflows/frontend-bundle-report.yml b/.github/workflows/frontend-bundle-report.yml deleted file mode 100644 index b5af3bb0e1..0000000000 --- a/.github/workflows/frontend-bundle-report.yml +++ /dev/null @@ -1,170 +0,0 @@ -name: frontend-bundle-report - -on: - pull_request: - types: - - opened - - synchronize - - reopened - - ready_for_review - paths: - - packages/frontend/** - - packages/frontend-shared/** - - packages/frontend-builder/** - - packages/i18n/** - - packages/icons-subsetter/** - - packages/misskey-js/** - - packages/misskey-reversi/** - - packages/misskey-bubble-game/** - - package.json - - pnpm-lock.yaml - - pnpm-workspace.yaml - - .node-version - - .github/scripts/utility.mts - - .github/scripts/frontend-js-size.mts - - .github/workflows/frontend-bundle-report.yml - - .github/workflows/frontend-bundle-report-comment.yml - -permissions: - contents: read - pull-requests: read - -concurrency: - group: frontend-bundle-report-${{ github.event.pull_request.number }} - cancel-in-progress: true - -jobs: - report: - name: Build frontend bundle report - runs-on: ubuntu-latest - env: - FRONTEND_JS_SIZE_LOCALE: ja-JP - steps: - - name: Checkout base - uses: actions/checkout@v6.0.3 - with: - repository: ${{ github.event.pull_request.base.repo.full_name }} - ref: ${{ github.event.pull_request.base.sha }} - path: before - submodules: true - - - name: Checkout pull request - uses: actions/checkout@v6.0.3 - with: - repository: ${{ github.event.pull_request.head.repo.full_name }} - ref: ${{ github.event.pull_request.head.sha }} - path: after - submodules: true - - - name: Check base visualizer support - id: check-base-visualizer - shell: bash - run: | - if grep -q 'FRONTEND_BUNDLE_VISUALIZER' before/packages/frontend/vite.config.ts; then - echo 'supported=true' >> "$GITHUB_OUTPUT" - else - echo 'supported=false' >> "$GITHUB_OUTPUT" - echo 'Base commit does not support frontend bundle visualizer. Skipping frontend bundle report.' >> "$GITHUB_STEP_SUMMARY" - fi - - - name: Setup pnpm - if: steps.check-base-visualizer.outputs.supported == 'true' - uses: pnpm/action-setup@v6.0.9 - with: - package_json_file: after/package.json - - - name: Setup Node.js - if: steps.check-base-visualizer.outputs.supported == 'true' - uses: actions/setup-node@v6.4.0 - with: - node-version-file: after/.node-version - cache: pnpm - cache-dependency-path: | - before/pnpm-lock.yaml - after/pnpm-lock.yaml - - - name: Install dependencies for base - if: steps.check-base-visualizer.outputs.supported == 'true' - working-directory: before - run: pnpm i --frozen-lockfile - - - name: Build frontend dependencies for base - if: steps.check-base-visualizer.outputs.supported == 'true' - working-directory: before - run: pnpm --filter "frontend^..." run build - - - name: Prepare report output - if: steps.check-base-visualizer.outputs.supported == 'true' - run: mkdir -p "$RUNNER_TEMP/frontend-bundle-report" - - - name: Build frontend report for base - if: steps.check-base-visualizer.outputs.supported == 'true' - working-directory: before - env: - FRONTEND_BUNDLE_VISUALIZER: 'true' - FRONTEND_BUNDLE_VISUALIZER_FILE: ${{ runner.temp }}/frontend-bundle-report/before-stats.json - run: pnpm --filter frontend run build - - - name: Install dependencies for pull request - if: steps.check-base-visualizer.outputs.supported == 'true' - working-directory: after - run: pnpm i --frozen-lockfile - - - name: Build frontend dependencies for pull request - if: steps.check-base-visualizer.outputs.supported == 'true' - working-directory: after - run: pnpm --filter "frontend^..." run build - - - name: Build frontend report for pull request - if: steps.check-base-visualizer.outputs.supported == 'true' - working-directory: after - env: - FRONTEND_BUNDLE_VISUALIZER: 'true' - FRONTEND_BUNDLE_VISUALIZER_FILE: ${{ runner.temp }}/frontend-bundle-report/after-stats.json - FRONTEND_BUNDLE_VISUALIZER_HTML_FILE: ${{ runner.temp }}/frontend-bundle-report/frontend-bundle-visualizer.html - run: pnpm --filter frontend run build - - - name: Upload bundle visualizer - if: steps.check-base-visualizer.outputs.supported == 'true' - id: upload-bundle-visualizer - uses: actions/upload-artifact@v7 - with: - name: frontend-bundle-visualizer - path: ${{ runner.temp }}/frontend-bundle-report/frontend-bundle-visualizer.html - if-no-files-found: error - archive: false - retention-days: 7 - - - name: Generate report markdown - if: steps.check-base-visualizer.outputs.supported == 'true' - shell: bash - env: - BASE_SHA: ${{ github.event.pull_request.base.sha }} - HEAD_SHA: ${{ github.event.pull_request.head.sha }} - PR_NUMBER: ${{ github.event.pull_request.number }} - FRONTEND_BUNDLE_REPORT_ARTIFACT_URL: ${{ steps.upload-bundle-visualizer.outputs.artifact-url }} - run: | - REPORT_DIR="$RUNNER_TEMP/frontend-bundle-report" - node after/.github/scripts/frontend-js-size.mts before after "$REPORT_DIR/before-stats.json" "$REPORT_DIR/after-stats.json" "$REPORT_DIR/frontend-js-size-report.md" - printf '%s\n' "$PR_NUMBER" > "$REPORT_DIR/pr-number.txt" - printf '%s\n' "$BASE_SHA" > "$REPORT_DIR/base-sha.txt" - printf '%s\n' "$HEAD_SHA" > "$REPORT_DIR/head-sha.txt" - printf '%s\n' "${{ github.event.pull_request.html_url }}" > "$REPORT_DIR/pr-url.txt" - - - name: Check report - if: steps.check-base-visualizer.outputs.supported == 'true' - run: | - REPORT_DIR="$RUNNER_TEMP/frontend-bundle-report" - test -s "$REPORT_DIR/before-stats.json" - test -s "$REPORT_DIR/after-stats.json" - test -s "$REPORT_DIR/frontend-js-size-report.md" - cat "$REPORT_DIR/frontend-js-size-report.md" >> "$GITHUB_STEP_SUMMARY" - - - name: Upload bundle report - if: steps.check-base-visualizer.outputs.supported == 'true' - uses: actions/upload-artifact@v7 - with: - name: frontend-bundle-report - path: ${{ runner.temp }}/frontend-bundle-report/ - if-no-files-found: error - retention-days: 7 diff --git a/.github/workflows/frontend-diagnostics.comment.yml b/.github/workflows/frontend-diagnostics.comment.yml new file mode 100644 index 0000000000..96deddcc58 --- /dev/null +++ b/.github/workflows/frontend-diagnostics.comment.yml @@ -0,0 +1,75 @@ +name: Frontend diagnostics (comment) + +on: + workflow_run: + workflows: + - Frontend diagnostics (inspect) + types: + - completed + +permissions: + actions: read + contents: read + issues: write + pull-requests: write + +jobs: + comment: + name: Comment frontend diagnostics report + if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success' + runs-on: ubuntu-latest + concurrency: + group: frontend-diagnostics-report-${{ github.event.workflow_run.id }} + cancel-in-progress: true + steps: + - name: Download frontend diagnostics report + uses: actions/download-artifact@v8 + with: + name: frontend-diagnostics + path: ${{ runner.temp }}/frontend-diagnostics + github-token: ${{ github.token }} + repository: ${{ github.repository }} + run-id: ${{ github.event.workflow_run.id }} + + - name: Resolve current pull request + id: resolve-pr + uses: actions/github-script@v9 + env: + PR_NUMBER_FILE: ${{ runner.temp }}/frontend-diagnostics/pr-number.txt + with: + script: | + const fs = require('fs/promises'); + const { owner, repo } = context.repo; + const workflowRun = context.payload.workflow_run; + const pullRequestNumberText = (await fs.readFile(process.env.PR_NUMBER_FILE, 'utf8')).trim(); + if (!/^[1-9]\d*$/.test(pullRequestNumberText)) { + throw new Error('Invalid pull request number in frontend diagnostics artifact.'); + } + + const pullRequestNumber = Number(pullRequestNumberText); + if (!Number.isSafeInteger(pullRequestNumber)) { + throw new Error('Pull request number in frontend diagnostics artifact is not a safe integer.'); + } + + const { data: currentPullRequest } = await github.rest.pulls.get({ + owner, + repo, + pull_number: pullRequestNumber, + }); + + if (currentPullRequest.state !== 'open' || currentPullRequest.head.sha !== workflowRun.head_sha) { + core.notice(`Skipping stale frontend diagnostics report for pull request #${pullRequestNumber}.`); + core.setOutput('is-current', 'false'); + return; + } + + core.setOutput('is-current', 'true'); + core.setOutput('pr-number', String(pullRequestNumber)); + + - name: Comment on pull request + if: steps.resolve-pr.outputs.is-current == 'true' + uses: thollander/actions-comment-pull-request@v3 + with: + pr-number: ${{ steps.resolve-pr.outputs.pr-number }} + comment-tag: frontend-diagnostics + file-path: ${{ runner.temp }}/frontend-diagnostics/frontend-diagnostics.md diff --git a/.github/workflows/frontend-diagnostics.inspect.yml b/.github/workflows/frontend-diagnostics.inspect.yml new file mode 100644 index 0000000000..acbc8ea77e --- /dev/null +++ b/.github/workflows/frontend-diagnostics.inspect.yml @@ -0,0 +1,246 @@ +name: Frontend diagnostics (inspect) + +on: + pull_request: + types: + - opened + - synchronize + - reopened + - ready_for_review + paths: + - packages/frontend/** + - packages/frontend-shared/** + - packages/frontend-builder/** + - packages/backend/** + - packages/i18n/** + - packages/icons-subsetter/** + - packages/misskey-js/** + - packages/misskey-reversi/** + - packages/misskey-bubble-game/** + - package.json + - pnpm-lock.yaml + - pnpm-workspace.yaml + - .node-version + - .github/misskey/test.yml + - packages-private/diagnostics-shared/** + - packages-private/diagnostics-frontend/** + - .github/workflows/frontend-diagnostics.inspect.yml + - .github/workflows/frontend-diagnostics.comment.yml + +permissions: + contents: read + pull-requests: read + +concurrency: + group: frontend-diagnostics-inspect-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + report: + name: Measure frontend diagnostics + runs-on: ubuntu-latest + timeout-minutes: 90 + + services: + postgres: + image: postgres:18 + ports: + - 54312:5432 + env: + POSTGRES_DB: test-misskey + POSTGRES_HOST_AUTH_METHOD: trust + redis: + image: redis:8 + ports: + - 56312:6379 + + steps: + - name: Checkout base + uses: actions/checkout@v7.0.0 + with: + persist-credentials: false + repository: ${{ github.event.pull_request.base.repo.full_name }} + ref: ${{ github.event.pull_request.base.sha }} + path: base + submodules: true + + - name: Checkout head + uses: actions/checkout@v7.0.0 + with: + persist-credentials: false + repository: ${{ github.event.pull_request.head.repo.full_name }} + ref: ${{ github.event.pull_request.head.sha }} + path: head + submodules: true + + - name: Setup pnpm + uses: pnpm/action-setup@v6.0.9 + with: + package_json_file: head/package.json + + - name: Setup Node.js + uses: actions/setup-node@v7.0.0 + with: + node-version-file: head/.node-version + cache: pnpm + cache-dependency-path: | + base/pnpm-lock.yaml + head/pnpm-lock.yaml + + - name: Prepare report output + shell: bash + run: mkdir -p "$RUNNER_TEMP/frontend-diagnostics" + + - name: Install dependencies for base + working-directory: base + run: pnpm i --frozen-lockfile + + - name: Configure base + working-directory: base + run: cp .github/misskey/test.yml .config + + - name: Build base + working-directory: base + env: + FRONTEND_BUNDLE_VISUALIZER: 'true' + FRONTEND_BUNDLE_VISUALIZER_FILE: ${{ runner.temp }}/frontend-diagnostics/base-bundle-stats.json + run: pnpm build + + - name: Install dependencies for head + working-directory: head + run: pnpm i --frozen-lockfile + + - name: Configure head + working-directory: head + run: cp .github/misskey/test.yml .config + + - name: Build head + working-directory: head + env: + FRONTEND_BUNDLE_VISUALIZER: 'true' + FRONTEND_BUNDLE_VISUALIZER_FILE: ${{ runner.temp }}/frontend-diagnostics/head-bundle-stats.json + FRONTEND_BUNDLE_VISUALIZER_HTML_FILE: ${{ runner.temp }}/frontend-diagnostics/frontend-bundle-visualizer.html + run: pnpm build + + - name: Upload bundle visualizer + id: upload-bundle-visualizer + uses: actions/upload-artifact@v7 + with: + name: frontend-bundle-visualizer + path: ${{ runner.temp }}/frontend-diagnostics/frontend-bundle-visualizer.html + if-no-files-found: error + archive: false + retention-days: 7 + + - name: Install Playwright browsers + working-directory: head/packages-private/diagnostics-frontend + run: pnpm exec playwright install --with-deps --no-shell chromium + + - name: Measure frontend browser metrics + shell: bash + working-directory: head + env: + FRONTEND_BROWSER_METRICS_SAMPLE_COUNT: 5 + MK_ENABLE_CROSS_ORIGIN_ISOLATION: "true" + # 計測ハーネスはhead側のものを両方に使うが、成果物はrunnerのworkspace直下に出す + FRONTEND_BROWSER_HEAP_SNAPSHOT_OUTPUT_DIR: ${{ github.workspace }} + run: | + REPORT_DIR="$RUNNER_TEMP/frontend-diagnostics" + pnpm --filter diagnostics-frontend run inspect-browser \ + "$GITHUB_WORKSPACE/base" "$GITHUB_WORKSPACE/head" \ + "$REPORT_DIR/base-browser.json" "$REPORT_DIR/head-browser.json" + + - name: Upload browser base heap snapshot + id: upload-browser-base-heap-snapshot + uses: actions/upload-artifact@v7 + with: + name: frontend-browser-base-heap-snapshot + path: base-heap-snapshot.heapsnapshot + archive: false + if-no-files-found: error + retention-days: 7 + + - name: Upload browser head heap snapshot + id: upload-browser-head-heap-snapshot + uses: actions/upload-artifact@v7 + with: + name: frontend-browser-head-heap-snapshot + path: head-heap-snapshot.heapsnapshot + archive: false + if-no-files-found: error + retention-days: 7 + + - name: Generate browser detailed html + shell: bash + working-directory: head + run: | + REPORT_DIR="$RUNNER_TEMP/frontend-diagnostics" + test -s "$REPORT_DIR/base-browser.json" + test -s "$REPORT_DIR/head-browser.json" + pnpm --filter diagnostics-frontend run render-browser-html \ + "$REPORT_DIR/base-browser.json" "$REPORT_DIR/head-browser.json" \ + "$REPORT_DIR/frontend-browser-detailed-html.html" + + - name: Upload browser detailed html + id: upload-browser-detailed-html + uses: actions/upload-artifact@v7 + with: + name: frontend-browser-metrics-detailed-html + path: ${{ runner.temp }}/frontend-diagnostics/frontend-browser-detailed-html.html + if-no-files-found: error + archive: false + retention-days: 7 + + - name: Generate frontend diagnostics report + shell: bash + working-directory: head + env: + BASE_SHA: ${{ github.event.pull_request.base.sha }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + PR_NUMBER: ${{ github.event.pull_request.number }} + FRONTEND_BUNDLE_REPORT_ARTIFACT_URL: ${{ steps.upload-bundle-visualizer.outputs.artifact-url }} + FRONTEND_BROWSER_BASE_HEAP_SNAPSHOT_ARTIFACT_URL: ${{ steps.upload-browser-base-heap-snapshot.outputs.artifact-url }} + FRONTEND_BROWSER_HEAD_HEAP_SNAPSHOT_ARTIFACT_URL: ${{ steps.upload-browser-head-heap-snapshot.outputs.artifact-url }} + FRONTEND_BROWSER_DETAILED_HTML_ARTIFACT_URL: ${{ steps.upload-browser-detailed-html.outputs.artifact-url }} + run: | + REPORT_DIR="$RUNNER_TEMP/frontend-diagnostics" + test -s "$REPORT_DIR/base-bundle-stats.json" + test -s "$REPORT_DIR/head-bundle-stats.json" + test -s "$REPORT_DIR/base-browser.json" + test -s "$REPORT_DIR/head-browser.json" + pnpm --filter diagnostics-frontend run render-md \ + "$GITHUB_WORKSPACE/base" "$GITHUB_WORKSPACE/head" \ + "$REPORT_DIR/base-bundle-stats.json" "$REPORT_DIR/head-bundle-stats.json" \ + "$REPORT_DIR/base-browser.json" "$REPORT_DIR/head-browser.json" \ + "$REPORT_DIR/frontend-diagnostics.md" + printf '%s\n' "$PR_NUMBER" > "$REPORT_DIR/pr-number.txt" + printf '%s\n' "$BASE_SHA" > "$REPORT_DIR/base-sha.txt" + printf '%s\n' "$HEAD_SHA" > "$REPORT_DIR/head-sha.txt" + printf '%s\n' "${{ github.event.pull_request.html_url }}" > "$REPORT_DIR/pr-url.txt" + + - name: Check frontend diagnostics report + shell: bash + run: | + REPORT_DIR="$RUNNER_TEMP/frontend-diagnostics" + test -s "$REPORT_DIR/frontend-diagnostics.md" + test -s "$REPORT_DIR/frontend-browser-detailed-html.html" + test -s "$REPORT_DIR/pr-number.txt" + test -s "$REPORT_DIR/head-sha.txt" + cat "$REPORT_DIR/frontend-diagnostics.md" >> "$GITHUB_STEP_SUMMARY" + + - name: Upload frontend diagnostics report + uses: actions/upload-artifact@v7 + with: + name: frontend-diagnostics + path: | + ${{ runner.temp }}/frontend-diagnostics/base-bundle-stats.json + ${{ runner.temp }}/frontend-diagnostics/head-bundle-stats.json + ${{ runner.temp }}/frontend-diagnostics/base-browser.json + ${{ runner.temp }}/frontend-diagnostics/head-browser.json + ${{ runner.temp }}/frontend-diagnostics/frontend-diagnostics.md + ${{ runner.temp }}/frontend-diagnostics/pr-number.txt + ${{ runner.temp }}/frontend-diagnostics/base-sha.txt + ${{ runner.temp }}/frontend-diagnostics/head-sha.txt + ${{ runner.temp }}/frontend-diagnostics/pr-url.txt + if-no-files-found: error + retention-days: 7 diff --git a/.github/workflows/get-api-diff.yml b/.github/workflows/get-api-diff.yml index ef34d19e68..77da323634 100644 --- a/.github/workflows/get-api-diff.yml +++ b/.github/workflows/get-api-diff.yml @@ -25,14 +25,14 @@ jobs: ref: refs/pull/${{ github.event.number }}/merge steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: ref: ${{ matrix.ref }} submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' diff --git a/.github/workflows/labeler.yml b/.github/workflows/labeler.yml index 5787572dd5..0cfbb22c65 100644 --- a/.github/workflows/labeler.yml +++ b/.github/workflows/labeler.yml @@ -11,6 +11,6 @@ jobs: pull-requests: write runs-on: ubuntu-latest steps: - - uses: actions/labeler@v6 + - uses: actions/labeler@v7 with: repo-token: "${{ secrets.GITHUB_TOKEN }}" diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 7585d51888..c1ef3cf64a 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -19,7 +19,9 @@ on: - packages/misskey-world/** - packages/frontend-misskey-world-engine/** - packages/shared/eslint.config.js + - scripts/check-dts*.mjs - .github/workflows/lint.yml + - package.json pull_request: paths: - packages/backend/** @@ -35,18 +37,20 @@ on: - packages/misskey-world/** - packages/frontend-misskey-world-engine/** - packages/shared/eslint.config.js + - scripts/check-dts*.mjs - .github/workflows/lint.yml + - package.json jobs: pnpm_install: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: fetch-depth: 0 submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - - uses: actions/setup-node@v6.4.0 + - uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' @@ -75,19 +79,19 @@ jobs: eslint-cache-version: v1 eslint-cache-path: ${{ github.workspace }}/node_modules/.cache/eslint-${{ matrix.workspace }} steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: fetch-depth: 0 submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - - uses: actions/setup-node@v6.4.0 + - uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' - run: pnpm i --frozen-lockfile - name: Restore eslint cache - uses: actions/cache@v5.0.5 + uses: actions/cache@v6.1.0 with: path: ${{ env.eslint-cache-path }} key: eslint-${{ env.eslint-cache-version }}-${{ matrix.workspace }}-${{ hashFiles('**/pnpm-lock.yaml') }}-${{ github.ref_name }}-${{ github.sha }} @@ -106,16 +110,35 @@ jobs: - sw - misskey-js steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: fetch-depth: 0 submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - - uses: actions/setup-node@v6.4.0 + - uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' - run: pnpm i --frozen-lockfile - run: pnpm --filter "${{ matrix.workspace }}^..." run build - run: pnpm --filter ${{ matrix.workspace }} run typecheck + + check-dts: + needs: [pnpm_install] + runs-on: ubuntu-latest + continue-on-error: true + steps: + - uses: actions/checkout@v7.0.0 + with: + fetch-depth: 0 + submodules: true + - name: Setup pnpm + uses: pnpm/action-setup@v6.0.9 + - uses: actions/setup-node@v7.0.0 + with: + node-version-file: '.node-version' + cache: 'pnpm' + - run: pnpm i --frozen-lockfile + - run: node --test scripts/check-dts.test.mjs + - run: pnpm check-dts diff --git a/.github/workflows/locale.yml b/.github/workflows/locale.yml index 000cfed91d..1eb2f56056 100644 --- a/.github/workflows/locale.yml +++ b/.github/workflows/locale.yml @@ -1,4 +1,4 @@ -name: Lint +name: Locale on: push: @@ -16,13 +16,13 @@ jobs: runs-on: ubuntu-latest continue-on-error: true steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: fetch-depth: 0 submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - - uses: actions/setup-node@v6.4.0 + - uses: actions/setup-node@v7.0.0 with: node-version-file: ".node-version" cache: "pnpm" diff --git a/.github/workflows/on-release-created.yml b/.github/workflows/on-release-created.yml index 76fd5548f2..37e976f25f 100644 --- a/.github/workflows/on-release-created.yml +++ b/.github/workflows/on-release-created.yml @@ -16,13 +16,13 @@ jobs: id-token: write steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' diff --git a/.github/workflows/packages-private.yml b/.github/workflows/packages-private.yml new file mode 100644 index 0000000000..2a93455c25 --- /dev/null +++ b/.github/workflows/packages-private.yml @@ -0,0 +1,52 @@ +name: Lint and test packages-private + +on: + push: + branches: + - master + - develop + paths: + - packages-private/** + - packages/shared/eslint.config.js + - package.json + - pnpm-workspace.yaml + - pnpm-lock.yaml + - .github/workflows/packages-private.yml + pull_request: + paths: + - packages-private/** + - packages/shared/eslint.config.js + - package.json + - pnpm-workspace.yaml + - pnpm-lock.yaml + - .github/workflows/packages-private.yml + +permissions: + contents: read + +jobs: + check: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + workspace: + - diagnostics-shared + - diagnostics-backend + - diagnostics-frontend + - changelog-checker + steps: + - uses: actions/checkout@v7.0.0 + with: + persist-credentials: false + - name: Setup pnpm + uses: pnpm/action-setup@v6.0.9 + - uses: actions/setup-node@v7.0.0 + with: + node-version-file: '.node-version' + cache: 'pnpm' + - run: pnpm --filter ${{ matrix.workspace }}... install --frozen-lockfile + # lint = typecheck + eslint + - run: pnpm --filter ${{ matrix.workspace }} run lint + # 各パッケージは必ず test スクリプトを持たせる (無いと ERR_PNPM_RECURSIVE_RUN_NO_SCRIPT で落ちる) + - run: pnpm --filter ${{ matrix.workspace }} run test diff --git a/.github/workflows/release-edit-with-push.yml b/.github/workflows/release-edit-with-push.yml index bc16dbcef2..176dcde1af 100644 --- a/.github/workflows/release-edit-with-push.yml +++ b/.github/workflows/release-edit-with-push.yml @@ -19,7 +19,7 @@ jobs: edit: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 # headが$GITHUB_REF_NAME, baseが$STABLE_BRANCHかつopenのPRを1つ取得 - name: Get PR run: | diff --git a/.github/workflows/release-with-dispatch.yml b/.github/workflows/release-with-dispatch.yml index f318200584..051a5e5419 100644 --- a/.github/workflows/release-with-dispatch.yml +++ b/.github/workflows/release-with-dispatch.yml @@ -36,7 +36,7 @@ jobs: outputs: pr_number: ${{ steps.get_pr.outputs.pr_number }} steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 # headが$GITHUB_REF_NAME, baseが$STABLE_BRANCHかつopenのPRを1つ取得 - name: Get PRs run: | diff --git a/.github/workflows/report-api-diff.yml b/.github/workflows/report-api-diff.yml index 67d163d697..4dc65fb1a5 100644 --- a/.github/workflows/report-api-diff.yml +++ b/.github/workflows/report-api-diff.yml @@ -65,7 +65,7 @@ jobs: echo '```diff' >> ./output.md cat ./api.json.diff >> ./output.md echo '```' >> ./output.md - echo '' >> ./output.md + echo '' >> .output.md fi echo "$FOOTER" >> ./output.md diff --git a/.github/workflows/storybook.yml b/.github/workflows/storybook.yml index 70bfa329a1..670ac3549e 100644 --- a/.github/workflows/storybook.yml +++ b/.github/workflows/storybook.yml @@ -22,12 +22,12 @@ jobs: NODE_OPTIONS: "--max_old_space_size=7168" steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 if: github.event_name != 'pull_request_target' with: fetch-depth: 0 submodules: true - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 if: github.event_name == 'pull_request_target' with: fetch-depth: 0 @@ -39,7 +39,7 @@ jobs: - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' diff --git a/.github/workflows/test-backend.yml b/.github/workflows/test-backend.yml index 3bf3b9eff5..7992d5d83b 100644 --- a/.github/workflows/test-backend.yml +++ b/.github/workflows/test-backend.yml @@ -43,7 +43,7 @@ jobs: ports: - 56312:6379 meilisearch: - image: getmeili/meilisearch:v1.48.1 + image: getmeili/meilisearch:v1.49.0 ports: - 57712:7700 env: @@ -51,7 +51,7 @@ jobs: MEILI_ENV: development steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: submodules: true - name: Setup pnpm @@ -60,7 +60,7 @@ jobs: run: | sudo apt install -y ffmpeg - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: ${{ matrix.node-version-file }} cache: 'pnpm' @@ -74,7 +74,7 @@ jobs: - name: Test run: pnpm --filter backend test-and-coverage - name: Upload to Codecov - uses: codecov/codecov-action@v6 + uses: codecov/codecov-action@v7 with: token: ${{ secrets.CODECOV_TOKEN }} files: ./packages/backend/coverage/coverage-final.json @@ -103,13 +103,13 @@ jobs: - 56312:6379 steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: ${{ matrix.node-version-file }} cache: 'pnpm' @@ -123,7 +123,7 @@ jobs: - name: Test run: pnpm --filter backend test-and-coverage:e2e - name: Upload to Codecov - uses: codecov/codecov-action@v6 + uses: codecov/codecov-action@v7 with: token: ${{ secrets.CODECOV_TOKEN }} files: ./packages/backend/coverage/coverage-final.json @@ -147,7 +147,7 @@ jobs: POSTGRES_HOST_AUTH_METHOD: trust steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: submodules: true - name: Setup pnpm @@ -156,7 +156,7 @@ jobs: id: current-date run: echo "today=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: ${{ matrix.node-version-file }} cache: 'pnpm' diff --git a/.github/workflows/test-federation.yml b/.github/workflows/test-federation.yml index 14dc33f626..1005cfa058 100644 --- a/.github/workflows/test-federation.yml +++ b/.github/workflows/test-federation.yml @@ -26,7 +26,7 @@ jobs: - .node-version - .github/min.node-version steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 with: submodules: true - name: Setup pnpm @@ -35,7 +35,7 @@ jobs: run: | sudo apt install -y ffmpeg - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: ${{ matrix.node-version-file }} cache: 'pnpm' diff --git a/.github/workflows/test-frontend.yml b/.github/workflows/test-frontend.yml index 6e55760462..774c552672 100644 --- a/.github/workflows/test-frontend.yml +++ b/.github/workflows/test-frontend.yml @@ -28,13 +28,13 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' @@ -48,7 +48,7 @@ jobs: - name: Test run: pnpm --filter frontend test-and-coverage - name: Upload Coverage - uses: codecov/codecov-action@v6 + uses: codecov/codecov-action@v7 with: token: ${{ secrets.CODECOV_TOKEN }} files: ./packages/frontend/coverage/coverage-final.json @@ -57,11 +57,6 @@ jobs: name: E2E tests (frontend) runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - browser: [chrome] - services: postgres: image: postgres:18 @@ -76,19 +71,13 @@ jobs: - 56312:6379 steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: submodules: true - # https://github.com/cypress-io/cypress-docker-images/issues/150 - #- name: Install mplayer for FireFox - # run: sudo apt install mplayer -y - # if: ${{ matrix.browser == 'firefox' }} - #- uses: browser-actions/setup-firefox@latest - # if: ${{ matrix.browser == 'firefox' }} - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' @@ -97,29 +86,14 @@ jobs: run: cp .github/misskey/test.yml .config - name: Build run: pnpm build - # https://github.com/cypress-io/cypress/issues/4351#issuecomment-559489091 - - name: ALSA Env - run: echo -e 'pcm.!default {\n type hw\n card 0\n}\n\nctl.!default {\n type hw\n card 0\n}' > ~/.asoundrc - # XXX: This tries reinstalling Cypress if the binary is not cached - # Remove this when the cache issue is fixed - - name: Cypress install - run: pnpm exec cypress install - - name: Cypress run - uses: cypress-io/github-action@v7.4.0 + - name: Playwright install + working-directory: packages/frontend + run: pnpm exec playwright install --with-deps chromium + - name: Test + run: pnpm start-server-and-test start:test http://localhost:61812 "pnpm --filter frontend test:e2e" timeout-minutes: 15 - with: - install: false - start: pnpm start:test - wait-on: 'http://localhost:61812' - headed: true - browser: ${{ matrix.browser }} - uses: actions/upload-artifact@v7 if: failure() with: - name: ${{ matrix.browser }}-cypress-screenshots - path: cypress/screenshots - - uses: actions/upload-artifact@v7 - if: always() - with: - name: ${{ matrix.browser }}-cypress-videos - path: cypress/videos + name: playwright-e2e-artifacts + path: packages/frontend/test/e2e/artifacts diff --git a/.github/workflows/test-misskey-js.yml b/.github/workflows/test-misskey-js.yml index f3c6f0a516..e3198051aa 100644 --- a/.github/workflows/test-misskey-js.yml +++ b/.github/workflows/test-misskey-js.yml @@ -22,13 +22,13 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v6.0.3 + uses: actions/checkout@v7.0.0 - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Setup Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' @@ -48,7 +48,7 @@ jobs: CI: true - name: Upload Coverage - uses: codecov/codecov-action@v6 + uses: codecov/codecov-action@v7 with: token: ${{ secrets.CODECOV_TOKEN }} files: ./packages/misskey-js/coverage/coverage-final.json diff --git a/.github/workflows/test-production.yml b/.github/workflows/test-production.yml index 0d9b2293dc..8192d21c81 100644 --- a/.github/workflows/test-production.yml +++ b/.github/workflows/test-production.yml @@ -16,13 +16,13 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' diff --git a/.github/workflows/validate-api-json.yml b/.github/workflows/validate-api-json.yml index ada1b3b2d8..b355f4e52b 100644 --- a/.github/workflows/validate-api-json.yml +++ b/.github/workflows/validate-api-json.yml @@ -17,13 +17,13 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6.0.3 + - uses: actions/checkout@v7.0.0 with: submodules: true - name: Setup pnpm uses: pnpm/action-setup@v6.0.9 - name: Use Node.js - uses: actions/setup-node@v6.4.0 + uses: actions/setup-node@v7.0.0 with: node-version-file: '.node-version' cache: 'pnpm' diff --git a/.gitignore b/.gitignore index ed1daaf489..e975691b93 100644 --- a/.gitignore +++ b/.gitignore @@ -23,9 +23,8 @@ packages/sw/.yarn/cache # pnpm .pnpm-store -# Cypress -cypress/screenshots -cypress/videos +# Playwright +packages/frontend/test/e2e/artifacts # Coverage coverage @@ -35,7 +34,7 @@ coverage !/.config/example.yml !/.config/docker_example.yml !/.config/docker_example.env -!/.config/cypress-devcontainer.yml +!/.config/playwright-devcontainer.yml docker-compose.yml ./compose.yml .devcontainer/compose.yml diff --git a/.okteto/okteto-pipeline.yml b/.okteto/okteto-pipeline.yml deleted file mode 100644 index e2996fbbc9..0000000000 --- a/.okteto/okteto-pipeline.yml +++ /dev/null @@ -1,6 +0,0 @@ -build: - misskey: - args: - - NODE_ENV=development -deploy: - - helm upgrade --install misskey chart --set image=${OKTETO_BUILD_MISSKEY_IMAGE} --set url="https://misskey-$(kubectl config view --minify -o jsonpath='{..namespace}').cloud.okteto.net" --set environment=development diff --git a/CHANGELOG.md b/CHANGELOG.md index bad2db5ce2..229e00d2b3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -## 2026.6.1 +## 2026.7.0 ### Note @@ -12,22 +12,71 @@ - Node.js のセキュリティアップデートに伴い、最低動作バージョンを 22.22.2 / 24.17.0 / 26.4.0 に引き上げました。 - Docker Image は Node.js 26.4.0-trixie に更新されています。 - バックエンドで画像処理に用いているライブラリ sharp のシステム要件の変更により、**SSE4.2 命令セットをサポートしていない x86_64 CPU では Misskey が正しく動作しなくなります**。仮想マシンに Misskey をデプロイしている場合や、古いハードウェアをお使いの場合は、アップデート前にお使いの環境をご確認ください。なお、ARM64 など x86_64 ではない環境においてはこの変更による影響はありません。 +- YAMLパーサーをアップデートし、より厳格なチェックが行われるようになりました。起動時にconfigの読み取りで構文エラーが発生する可能性があります。\ + 例えば `allowPrivateNetworks` を 2026.6.0 までの `example.yml` の構文を元に記述している場合は、配列の閉じ括弧のインデントを上げるか、リスト表示に書き換える必要があります。詳しくは https://github.com/misskey-dev/misskey/pull/17701 をご覧ください。 ### General - Feat: コントロールパネルから二要素認証を解除できるように +- Feat: 条件に一致したURLプレビューのサムネイルを隠すことができるように + (Based on https://github.com/MisskeyIO/misskey/pull/214) +- Enhance: 依存関係の更新 ### Client -- Fix: チャットでIMEの変換を確定するEnterでメッセージが送信されてしまうことがある問題を修正 - 2025.4.0 以前の設定情報の移行処理が削除されました - 2025.4.0 から直接 2026.6.0 以上にアップデートする場合は設定が移行されませんので注意してください。移行したい場合は一度 2026.5.1 を経由してください。 +- Enhance: 画像ビューワーを刷新・動画プレイヤーを統合 + - 操作性の改善 + - ビューワーとMisskeyの各種機能との統合を強化 + - パフォーマンスの向上 + - Enhance: マウスホイールで拡大縮小できるように + - Fix: 幅が狭い画面で動画の再生が困難な問題を修正 + - Fix: 一部の画像のみセンシティブなとき、ビューワー内で画像を切り替えるとセンシティブな画像がそのまま表示される問題を修正 + - Fix: 一部の画像をビューワーで読み込んだ際に正しく表示されない問題を修正 +- Enhance: ファイルアップロード前にプレビューできるように +- Enhance: タブがバックグラウンドの間は必要ない定期更新処理を停止するように +- Enhance: 翻訳の更新 +- Fix: 「画像を新しいタブで開く」が機能しなくなっていた問題を修正 - Fix: デバイスタイプをスマートフォンに固定している状態で画面幅が広いとき、画面左上のアイコンが表示されない問題を修正 +- Fix: チャットでIMEの変換を確定するEnterでメッセージが送信されてしまうことがある問題を修正 +- Fix: 自分へのメンションに対する色分けで、判定が大文字/小文字を区別していた問題を修正 +- Fix: いくつかのイベントリスナーが正しく解除されない問題を修正（メモリ使用量の改善） +- Fix: 非ログイン時トップページをスクロール操作できないことがある問題を修正 +- Fix: ローカルユーザーへのホスト付きメンションが本文に含まれる指名ノートの作成時、投稿フォームにて、当該ユーザーが宛先に含まれていても正しく認識されない問題を修正 +- Fix: チャートの描画終了後にリソースが解放されない問題を修正 +- Fix: ドライブの「このファイルからノートを作成」やギャラリーの「ノートで共有」、誕生日ウィジェットからのノート作成において、通常投稿の下書きが表示される問題を修正 +- Fix: QRコードリーダーがページを離れても停止しない問題を修正 +- Fix: ノートの詳細表示で削除された引用元が表示されない問題を修正 ### Server +- Feat: OpenTelemetryサポート + - 詳細な設定はconfigファイルを参照してください。 + - Sentryとの併用も可能です。Sentry併用時は、PostgreSQL Query と Redis command は Sentry で計装されます。 + - 以下の自動計装をサポートしています。（計装対象にする項目は設定可能） + - PostgreSQL query + - Redis command + - 全ての受信HTTPリクエスト + - 全ての送信HTTPリクエスト + - ジョブキュー（エンキュー元のトレースを含む） +- Feat: ログ基盤の刷新 + - API内部エラーのログに構造化属性と正規化したエラー情報を付与し、認証情報を自動的に秘匿するように（従来形式の表示は維持） + - ログ全体の既定出力レベルとドメインごとの出力レベルを設定できるように + - バックエンドのログを1行JSON形式で出力できるように + - OpenTelemetryのTrace ContextをJSON形式のログへ関連付けられるように + - HTTPのAccess logをstatus class単位で出力できるように（開発時のリクエスト・レスポンス本文、OpenTelemetryのTrace Contextにも対応） +- Enhance: Sentry バックエンドの自動計装を `sentryForBackend.disabledIntegrations` で個別に無効化できるように - Enhance: センシティブメディアの判定を外部サービス ([sensitive-detector](https://github.com/misskey-dev/sensitive-detector)) に分離し、`nsfwjs` / `@tensorflow/tfjs(-node)` の同梱と NSFW 判定モデルを廃止 (#16804) -- Enhance: Node.js 22.23.0以降、24.17.0以降、26.4.0以降をサポートするように +- Enhance: Node.js 22.22.2以降、24.17.0以降、26.4.0以降をサポートするように - Enhance: Docker Image の Node.js を 26.4.0 に、Debian を trixie (v13) に更新 +- Enhance: URLプレビューの結果を内部でキャッシュするように - Fix: `/stats` API のレスポンス型が正しくない問題を修正 - Fix: ハッシュタグに関連するデータを更新する際のエラーハンドリングを修正 +- Fix: Sentry 使用環境下にて、Misskey が発行した SQL クエリが span に含まれない問題を修正 +- Fix: Sentry 使用環境下にて、外部送信リクエストへ `sentry-trace` / `baggage` ヘッダーが既定で付与されないように +- Fix: フォロワー限定投稿へのリプライをホーム投稿に出来る問題を修正 +- Fix: ファイルをアップロードするAPIにて、処理終了後に一時ファイルが削除されないことがある問題を修正 +- Fix: 初期設定で作成したアカウント以外でアカウント作成APIが使用できない問題を修正 +- Fix: フォロー中のチャンネルを再度フォローした際にALREADY_FOLLOWINGエラーを返すように +- Fix: セキュリティに関する修正 ## 2026.6.0 diff --git a/chart/Chart.yaml b/chart/Chart.yaml deleted file mode 100644 index d151f1dd17..0000000000 --- a/chart/Chart.yaml +++ /dev/null @@ -1,4 +0,0 @@ -apiVersion: v2 -name: misskey -version: 0.0.0 -description: This chart is created for the purpose of previewing Pull Requests. Do not use this for production use. diff --git a/chart/files/default.yml b/chart/files/default.yml deleted file mode 100644 index c148cade9f..0000000000 --- a/chart/files/default.yml +++ /dev/null @@ -1,232 +0,0 @@ -#━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -# Misskey configuration -#━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -# ┌─────┐ -#───┘ URL └───────────────────────────────────────────────────── - -# Final accessible URL seen by a user. -# url: https://example.tld/ - -# ONCE YOU HAVE STARTED THE INSTANCE, DO NOT CHANGE THE -# URL SETTINGS AFTER THAT! - -# ┌───────────────────────┐ -#───┘ Port and TLS settings └─────────────────────────────────── - -# -# Misskey supports two deployment options for public. -# - -# Option 1: With Reverse Proxy -# -# +----- https://example.tld/ ------------+ -# +------+ |+-------------+ +----------------+| -# | User | ---> || Proxy (443) | ---> | Misskey (3000) || -# +------+ |+-------------+ +----------------+| -# +---------------------------------------+ -# -# You need to setup reverse proxy. (eg. nginx) -# You do not define 'https' section. - -# Option 2: Standalone -# -# +- https://example.tld/ -+ -# +------+ | +---------------+ | -# | User | ---> | | Misskey (443) | | -# +------+ | +---------------+ | -# +------------------------+ -# -# You need to run Misskey as root. -# You need to set Certificate in 'https' section. - -# To use option 1, uncomment below line. -port: 3000 # A port that your Misskey server should listen. - -# To use option 2, uncomment below lines. -#port: 443 - -#https: -# # path for certification -# key: /etc/letsencrypt/live/example.tld/privkey.pem -# cert: /etc/letsencrypt/live/example.tld/fullchain.pem - -# ┌──────────────────────────┐ -#───┘ PostgreSQL configuration └──────────────────────────────── - -db: - host: localhost - port: 5432 - - # Database name - db: misskey - - # Auth - user: example-misskey-user - pass: example-misskey-pass - - # Whether disable Caching queries - #disableCache: true - - # Extra Connection options - #extra: - # ssl: true - -dbReplications: false - -# You can configure any number of replicas here -#dbSlaves: -# - -# host: -# port: -# db: -# user: -# pass: -# - -# host: -# port: -# db: -# user: -# pass: - -# ┌─────────────────────┐ -#───┘ Redis configuration └───────────────────────────────────── - -redis: - host: localhost - port: 6379 - #family: 0 # 0=Both, 4=IPv4, 6=IPv6 - #pass: example-pass - #prefix: example-prefix - #db: 1 - -#redisForPubsub: -# host: localhost -# port: 6379 -# #family: 0 # 0=Both, 4=IPv4, 6=IPv6 -# #pass: example-pass -# #prefix: example-prefix -# #db: 1 - -#redisForJobQueue: -# host: localhost -# port: 6379 -# #family: 0 # 0=Both, 4=IPv4, 6=IPv6 -# #pass: example-pass -# #prefix: example-prefix -# #db: 1 - -#redisForTimelines: -# host: redis -# port: 6379 -# #family: 0 # 0=Both, 4=IPv4, 6=IPv6 -# #pass: example-pass -# #prefix: example-prefix -# #db: 1 - -#redisForReactions: -# host: redis -# port: 6379 -# #family: 0 # 0=Both, 4=IPv4, 6=IPv6 -# #pass: example-pass -# #prefix: example-prefix -# #db: 1 - -# ┌───────────────────────────┐ -#───┘ MeiliSearch configuration └───────────────────────────── - -#meilisearch: -# host: localhost -# port: 7700 -# apiKey: '' -# ssl: true -# index: '' - -# ┌───────────────┐ -#───┘ ID generation └─────────────────────────────────────────── - -# You can select the ID generation method. -# You don't usually need to change this setting, but you can -# change it according to your preferences. - -# Available methods: -# aid ... Short, Millisecond accuracy -# aidx ... Millisecond accuracy -# meid ... Similar to ObjectID, Millisecond accuracy -# ulid ... Millisecond accuracy -# objectid ... This is left for backward compatibility - -# ONCE YOU HAVE STARTED THE INSTANCE, DO NOT CHANGE THE -# ID SETTINGS AFTER THAT! - -id: "aidx" - -# ┌────────────────┐ -#───┘ Error tracking └────────────────────────────────────────── - -# Sentry is available for error tracking. -# See the Sentry documentation for more details on options. - -#sentryForBackend: -# enableNodeProfiling: true -# options: -# dsn: 'https://examplePublicKey@o0.ingest.sentry.io/0' - -#sentryForFrontend: -# vueIntegration: -# tracingOptions: -# trackComponents: true -# browserTracingIntegration: -# replayIntegration: -# options: -# dsn: 'https://examplePublicKey@o0.ingest.sentry.io/0' - -# ┌─────────────────────┐ -#───┘ Other configuration └───────────────────────────────────── - -# Whether disable HSTS -#disableHsts: true - -# Number of worker processes -#clusterLimit: 1 - -# Number of threads of extra thread pool for CPU-intensive tasks (per worker) -#threadPoolSize: 1 - -# Job concurrency per worker -# deliverJobConcurrency: 128 -# inboxJobConcurrency: 16 - -# Job rate limiter -# deliverJobPerSec: 128 -# inboxJobPerSec: 32 - -# Job attempts -# deliverJobMaxAttempts: 12 -# inboxJobMaxAttempts: 8 - -# IP address family used for outgoing request (ipv4, ipv6 or dual) -#outgoingAddressFamily: ipv4 - -# Proxy for HTTP/HTTPS -#proxy: http://127.0.0.1:3128 - -#proxyBypassHosts: [ -# 'example.com', -# '192.0.2.8' -#] - -# Proxy for SMTP/SMTPS -#proxySmtp: http://127.0.0.1:3128 # use HTTP/1.1 CONNECT -#proxySmtp: socks4://127.0.0.1:1080 # use SOCKS4 -#proxySmtp: socks5://127.0.0.1:1080 # use SOCKS5 - -# Media Proxy -#mediaProxy: https://example.com/proxy - -#allowedPrivateNetworks: [ -# '127.0.0.1/32' -#] - -# Upload or download file size limits (bytes) -#maxFileSize: 262144000 diff --git a/chart/templates/ConfigMap.yml b/chart/templates/ConfigMap.yml deleted file mode 100644 index 37c25e0864..0000000000 --- a/chart/templates/ConfigMap.yml +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ include "misskey.fullname" . }}-configuration -data: - default.yml: |- - {{ .Files.Get "files/default.yml"|nindent 4 }} - url: {{ .Values.url }} diff --git a/chart/templates/Deployment.yml b/chart/templates/Deployment.yml deleted file mode 100644 index 280bf9a597..0000000000 --- a/chart/templates/Deployment.yml +++ /dev/null @@ -1,47 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ include "misskey.fullname" . }} - labels: - {{- include "misskey.labels" . | nindent 4 }} -spec: - selector: - matchLabels: - {{- include "misskey.selectorLabels" . | nindent 6 }} - replicas: 1 - template: - metadata: - labels: - {{- include "misskey.selectorLabels" . | nindent 8 }} - spec: - containers: - - name: misskey - image: {{ .Values.image }} - env: - - name: NODE_ENV - value: {{ .Values.environment }} - volumeMounts: - - name: {{ include "misskey.fullname" . }}-configuration - mountPath: /misskey/.config - readOnly: true - ports: - - containerPort: 3000 - - name: postgres - image: postgres:18-alpine - env: - - name: POSTGRES_USER - value: "example-misskey-user" - - name: POSTGRES_PASSWORD - value: "example-misskey-pass" - - name: POSTGRES_DB - value: "misskey" - ports: - - containerPort: 5432 - - name: redis - image: redis:7-alpine - ports: - - containerPort: 6379 - volumes: - - name: {{ include "misskey.fullname" . }}-configuration - configMap: - name: {{ include "misskey.fullname" . }}-configuration diff --git a/chart/templates/Service.yml b/chart/templates/Service.yml deleted file mode 100644 index afd851a9f1..0000000000 --- a/chart/templates/Service.yml +++ /dev/null @@ -1,14 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: {{ include "misskey.fullname" . }} - annotations: - dev.okteto.com/auto-ingress: "true" -spec: - type: ClusterIP - ports: - - port: 3000 - protocol: TCP - name: http - selector: - {{- include "misskey.selectorLabels" . | nindent 4 }} diff --git a/chart/templates/_helpers.tpl b/chart/templates/_helpers.tpl deleted file mode 100644 index a5a2499f3f..0000000000 --- a/chart/templates/_helpers.tpl +++ /dev/null @@ -1,62 +0,0 @@ -{{/* -Expand the name of the chart. -*/}} -{{- define "misskey.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} -{{- end }} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "misskey.fullname" -}} -{{- if .Values.fullnameOverride }} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- $name := default .Chart.Name .Values.nameOverride }} -{{- if contains $name .Release.Name }} -{{- .Release.Name | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} -{{- end }} -{{- end }} -{{- end }} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "misskey.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} -{{- end }} - -{{/* -Common labels -*/}} -{{- define "misskey.labels" -}} -helm.sh/chart: {{ include "misskey.chart" . }} -{{ include "misskey.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end }} - -{{/* -Selector labels -*/}} -{{- define "misskey.selectorLabels" -}} -app.kubernetes.io/name: {{ include "misskey.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end }} - -{{/* -Create the name of the service account to use -*/}} -{{- define "misskey.serviceAccountName" -}} -{{- if .Values.serviceAccount.create }} -{{- default (include "misskey.fullname" .) .Values.serviceAccount.name }} -{{- else }} -{{- default "default" .Values.serviceAccount.name }} -{{- end }} -{{- end }} diff --git a/chart/values.yml b/chart/values.yml deleted file mode 100644 index a7031538a9..0000000000 --- a/chart/values.yml +++ /dev/null @@ -1,3 +0,0 @@ -url: https://example.tld/ -image: okteto.dev/misskey -environment: production diff --git a/cypress.config.ts b/cypress.config.ts deleted file mode 100644 index 361acaf6e5..0000000000 --- a/cypress.config.ts +++ /dev/null @@ -1,7 +0,0 @@ -import { defineConfig } from 'cypress' - -export default defineConfig({ - e2e: { - baseUrl: 'http://localhost:61812', - }, -}) diff --git a/cypress/e2e/basic.cy.ts b/cypress/e2e/basic.cy.ts deleted file mode 100644 index 4f2f700146..0000000000 --- a/cypress/e2e/basic.cy.ts +++ /dev/null @@ -1,265 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -describe('Before setup instance', () => { - beforeEach(() => { - cy.resetState(); - }); - - afterEach(() => { - // テスト終了直前にページ遷移するようなテストケース(例えばアカウント作成)だと、たぶんCypressのバグでブラウザの内容が次のテストケースに引き継がれてしまう(例えばアカウントが作成し終わった段階からテストが始まる)。 - // waitを入れることでそれを防止できる - cy.wait(1000); - }); - - it('successfully loads', () => { - cy.visitHome(); - }); - - it('setup instance', () => { - cy.visitHome(); - - cy.intercept('POST', '/api/admin/accounts/create').as('signup'); - - cy.get('[data-cy-admin-initial-password] input').type('example_password_please_change_this_or_you_will_get_hacked'); - cy.get('[data-cy-admin-username] input').type('admin'); - cy.get('[data-cy-admin-password] input').type('admin1234'); - cy.get('[data-cy-admin-ok]').click(); - - // なぜか動かない - //cy.wait('@signup').should('have.property', 'response.statusCode'); - cy.wait('@signup'); - - cy.intercept('POST', '/api/admin/update-meta').as('update-meta'); - - cy.get('[data-cy-next]').click(); - cy.get('[data-cy-server-name] input').type('Testskey'); - cy.get('[data-cy-server-setup-wizard-apply]').click(); - - cy.wait('@update-meta'); - }); -}); - -describe('After setup instance', () => { - beforeEach(() => { - cy.resetState(); - - // インスタンス初期セットアップ - cy.registerUser('admin', 'pass', true); - }); - - afterEach(() => { - // テスト終了直前にページ遷移するようなテストケース(例えばアカウント作成)だと、たぶんCypressのバグでブラウザの内容が次のテストケースに引き継がれてしまう(例えばアカウントが作成し終わった段階からテストが始まる)。 - // waitを入れることでそれを防止できる - cy.wait(1000); - }); - - it('successfully loads', () => { - cy.visitHome(); - }); - - it('signup', () => { - cy.visitHome(); - - cy.intercept('POST', '/api/signup').as('signup'); - - cy.get('[data-cy-signup]').click(); - cy.get('[data-cy-signup-rules-continue]').should('be.disabled'); - cy.get('[data-cy-signup-rules-notes-agree] [data-cy-switch-toggle]').click(); - cy.get('[data-cy-modal-dialog-ok]').click(); - cy.get('[data-cy-signup-rules-continue]').should('not.be.disabled'); - cy.get('[data-cy-signup-rules-continue]').click(); - - cy.get('[data-cy-signup-submit]').should('be.disabled'); - cy.get('[data-cy-signup-username] input').type('alice'); - cy.get('[data-cy-signup-submit]').should('be.disabled'); - cy.get('[data-cy-signup-password] input').type('alice1234'); - cy.get('[data-cy-signup-submit]').should('be.disabled'); - cy.get('[data-cy-signup-password-retype] input').type('alice1234'); - cy.get('[data-cy-signup-submit]').should('be.disabled'); - cy.get('[data-cy-signup-invitation-code] input').type('test-invitation-code'); - cy.get('[data-cy-signup-submit]').should('not.be.disabled'); - cy.get('[data-cy-signup-submit]').click(); - - cy.wait('@signup'); - }); - - it('signup with duplicated username', () => { - cy.registerUser('alice', 'alice1234'); - - cy.visitHome(); - - // ユーザー名が重複している場合の挙動確認 - cy.get('[data-cy-signup]').click(); - cy.get('[data-cy-signup-rules-continue]').should('be.disabled'); - cy.get('[data-cy-signup-rules-notes-agree] [data-cy-switch-toggle]').click(); - cy.get('[data-cy-modal-dialog-ok]').click(); - cy.get('[data-cy-signup-rules-continue]').should('not.be.disabled'); - cy.get('[data-cy-signup-rules-continue]').click(); - - cy.get('[data-cy-signup-username] input').type('alice'); - cy.get('[data-cy-signup-password] input').type('alice1234'); - cy.get('[data-cy-signup-password-retype] input').type('alice1234'); - cy.get('[data-cy-signup-submit]').should('be.disabled'); - }); -}); - -describe('After user signup', () => { - beforeEach(() => { - cy.resetState(); - - // インスタンス初期セットアップ - cy.registerUser('admin', 'pass', true); - - // ユーザー作成 - cy.registerUser('alice', 'alice1234'); - }); - - afterEach(() => { - // テスト終了直前にページ遷移するようなテストケース(例えばアカウント作成)だと、たぶんCypressのバグでブラウザの内容が次のテストケースに引き継がれてしまう(例えばアカウントが作成し終わった段階からテストが始まる)。 - // waitを入れることでそれを防止できる - cy.wait(1000); - }); - - it('successfully loads', () => { - cy.visitHome(); - }); - - it('signin', () => { - cy.visitHome(); - - cy.intercept('POST', '/api/signin-flow').as('signin'); - - cy.get('[data-cy-signin]').click(); - - cy.get('[data-cy-signin-page-input]').should('be.visible', { timeout: 1000 }); - // Enterキーで続行できるかの確認も兼ねる - cy.get('[data-cy-signin-username] input').type('alice{enter}'); - - cy.get('[data-cy-signin-page-password]').should('be.visible', { timeout: 10000 }); - // Enterキーで続行できるかの確認も兼ねる - cy.get('[data-cy-signin-password] input').type('alice1234{enter}'); - - cy.wait('@signin'); - }); - - it('suspend', function() { - cy.request('POST', '/api/admin/suspend-user', { - i: this.admin.token, - userId: this.alice.id, - }); - - cy.visitHome(); - - cy.get('[data-cy-signin]').click(); - - cy.get('[data-cy-signin-page-input]').should('be.visible', { timeout: 1000 }); - cy.get('[data-cy-signin-username] input').type('alice{enter}'); - - // TODO: cypressにブラウザの言語指定できる機能が実装され次第英語のみテストするようにする - cy.contains(/アカウントが凍結されています|This account has been suspended due to/gi); - }); -}); - -describe('After user signed in', () => { - beforeEach(() => { - cy.resetState(); - - // インスタンス初期セットアップ - cy.registerUser('admin', 'pass', true); - - // ユーザー作成 - cy.registerUser('alice', 'alice1234'); - - cy.login('alice', 'alice1234'); - }); - - afterEach(() => { - // テスト終了直前にページ遷移するようなテストケース(例えばアカウント作成)だと、たぶんCypressのバグでブラウザの内容が次のテストケースに引き継がれてしまう(例えばアカウントが作成し終わった段階からテストが始まる)。 - // waitを入れることでそれを防止できる - cy.wait(1000); - }); - - it('successfully loads', () => { - // 表示に時間がかかるのでデフォルト秒数だとタイムアウトする - cy.get('[data-cy-user-setup-continue]', { timeout: 30000 }).should('be.visible'); - }); - - it('account setup wizard', () => { - // 表示に時間がかかるのでデフォルト秒数だとタイムアウトする - cy.get('[data-cy-user-setup-continue]', { timeout: 30000 }).click(); - - cy.get('[data-cy-user-setup-user-name] input').type('ありす'); - cy.get('[data-cy-user-setup-user-description] textarea').type('ほげ'); - // TODO: アイコン設定テスト - - cy.get('[data-cy-user-setup-continue]').click(); - - // プライバシー設定 - - cy.get('[data-cy-user-setup-continue]').click(); - - // フォローはスキップ - - cy.get('[data-cy-user-setup-continue]').click(); - - // プッシュ通知設定はスキップ - - cy.get('[data-cy-user-setup-continue]').click(); - - cy.get('[data-cy-user-setup-continue]').click(); - }); -}); - -describe('After user setup', () => { - beforeEach(() => { - cy.resetState(); - - // インスタンス初期セットアップ - cy.registerUser('admin', 'pass', true); - - // ユーザー作成 - cy.registerUser('alice', 'alice1234'); - - cy.login('alice', 'alice1234'); - - // アカウント初期設定ウィザード - // 表示に時間がかかるのでデフォルト秒数だとタイムアウトする - cy.get('[data-cy-user-setup] [data-cy-modal-window-close]', { timeout: 30000 }).click(); - cy.get('[data-cy-modal-dialog-ok]').click(); - }); - - afterEach(() => { - // テスト終了直前にページ遷移するようなテストケース(例えばアカウント作成)だと、たぶんCypressのバグでブラウザの内容が次のテストケースに引き継がれてしまう(例えばアカウントが作成し終わった段階からテストが始まる)。 - // waitを入れることでそれを防止できる - cy.wait(1000); - }); - - it('note', () => { - cy.get('[data-cy-open-post-form]').should('be.visible'); - cy.get('[data-cy-open-post-form]').click(); - cy.get('[data-cy-post-form-text]').type('Hello, Misskey!'); - cy.get('[data-cy-open-post-form-submit]').click(); - - cy.contains('Hello, Misskey!', { timeout: 15000 }); - }); - - it('open note form with hotkey', () => { - // Wait until the page loads - cy.get('[data-cy-open-post-form]').should('be.visible'); - // Use trigger() to give different `code` to test if hotkeys also work on non-QWERTY keyboards. - cy.document().trigger("keydown", { eventConstructor: 'KeyboardEvent', key: "n", code: "KeyL" }); - // See if the form is opened - cy.get('[data-cy-post-form-text]').should('be.visible'); - // Close it - cy.focused().trigger("keydown", { eventConstructor: 'KeyboardEvent', key: "Escape", code: "Escape" }); - // See if the form is closed - cy.get('[data-cy-post-form-text]').should('not.be.visible'); - }); -}); - -// TODO: 投稿フォームの公開範囲指定のテスト -// TODO: 投稿フォームのファイル添付のテスト -// TODO: 投稿フォームのハッシュタグ保持フィールドのテスト diff --git a/cypress/e2e/router.cy.ts b/cypress/e2e/router.cy.ts deleted file mode 100644 index 8d8fb3af31..0000000000 --- a/cypress/e2e/router.cy.ts +++ /dev/null @@ -1,35 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -describe('Router transition', () => { - describe('Redirect', () => { - // サーバの初期化。ルートのテストに関しては各describeごとに1度だけ実行で十分だと思う（使いまわした方が早い） - before(() => { - cy.resetState(); - - // インスタンス初期セットアップ - cy.registerUser('admin', 'pass', true); - - // ユーザー作成 - cy.registerUser('alice', 'alice1234'); - - cy.login('alice', 'alice1234'); - - // アカウント初期設定ウィザード - // 表示に時間がかかるのでデフォルト秒数だとタイムアウトする - cy.get('[data-cy-user-setup] [data-cy-modal-window-close]', { timeout: 30000 }).click(); - cy.wait(500); - cy.get('[data-cy-modal-dialog-ok]').click(); - }); - - it('redirect to user profile', () => { - // テストのためだけに用意されたリダイレクト用ルートに飛ぶ - cy.visit('/redirect-test'); - - // プロフィールページのURLであることを確認する - cy.url().should('include', '/@alice') - }); - }); -}); diff --git a/cypress/e2e/widgets.cy.ts b/cypress/e2e/widgets.cy.ts deleted file mode 100644 index 847801a69f..0000000000 --- a/cypress/e2e/widgets.cy.ts +++ /dev/null @@ -1,76 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -/* flaky -describe('After user signed in', () => { - beforeEach(() => { - cy.resetState(); - cy.viewport('macbook-16'); - - // インスタンス初期セットアップ - cy.registerUser('admin', 'pass', true); - - // ユーザー作成 - cy.registerUser('alice', 'alice1234'); - - cy.login('alice', 'alice1234'); - - // アカウント初期設定ウィザード - cy.get('[data-cy-user-setup] [data-cy-modal-window-close]').click(); - cy.get('[data-cy-modal-dialog-ok]').click(); - }); - - afterEach(() => { - // テスト終了直前にページ遷移するようなテストケース(例えばアカウント作成)だと、たぶんCypressのバグでブラウザの内容が次のテストケースに引き継がれてしまう(例えばアカウントが作成し終わった段階からテストが始まる)。 - // waitを入れることでそれを防止できる - cy.wait(1000); - }); - - it('widget edit toggle is visible', () => { - cy.get('[data-cy-widget-edit]').should('be.visible'); - }); - - it('widget select should be visible in edit mode', () => { - cy.get('[data-cy-widget-edit]').click(); - cy.get('[data-cy-widget-select]').should('be.visible'); - }); - - it('first widget should be removed', () => { - cy.get('[data-cy-widget-edit]').click(); - cy.get('[data-cy-customize-container]:first-child [data-cy-customize-container-remove]._button').click(); - cy.get('[data-cy-customize-container]').should('have.length', 2); - }); - - function buildWidgetTest(widgetName) { - it(`${widgetName} widget should get added`, () => { - cy.get('[data-cy-widget-edit]').click(); - cy.get('[data-cy-widget-select] select').select(widgetName, { force: true }); - cy.get('[data-cy-bg]._modalBg[data-cy-transparent]').click({ multiple: true, force: true }); - cy.get('[data-cy-widget-add]').click({ force: true }); - cy.get(`[data-cy-mkw-${widgetName}]`).should('exist'); - }); - } - - buildWidgetTest('memo'); - buildWidgetTest('notifications'); - buildWidgetTest('timeline'); - buildWidgetTest('calendar'); - buildWidgetTest('rss'); - buildWidgetTest('trends'); - buildWidgetTest('clock'); - buildWidgetTest('activity'); - buildWidgetTest('photos'); - buildWidgetTest('digitalClock'); - buildWidgetTest('federation'); - buildWidgetTest('postForm'); - buildWidgetTest('slideshow'); - buildWidgetTest('serverMetric'); - buildWidgetTest('onlineUsers'); - buildWidgetTest('jobQueue'); - buildWidgetTest('button'); - buildWidgetTest('aiscript'); - buildWidgetTest('aichan'); -}); -*/ diff --git a/cypress/fixtures/example.json b/cypress/fixtures/example.json deleted file mode 100644 index 02e4254378..0000000000 --- a/cypress/fixtures/example.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "name": "Using fixtures to represent data", - "email": "hello@cypress.io", - "body": "Fixtures are a great way to mock data for responses to routes" -} diff --git a/cypress/support/commands.ts b/cypress/support/commands.ts deleted file mode 100644 index 197ff963ac..0000000000 --- a/cypress/support/commands.ts +++ /dev/null @@ -1,67 +0,0 @@ -// *********************************************** -// This example commands.js shows you how to -// create various custom commands and overwrite -// existing commands. -// -// For more comprehensive examples of custom -// commands please read more here: -// https://on.cypress.io/custom-commands -// *********************************************** -// -// -// -- This is a parent command -- -// Cypress.Commands.add('login', (email, password) => { ... }) -// -// -// -- This is a child command -- -// Cypress.Commands.add('drag', { prevSubject: 'element'}, (subject, options) => { ... }) -// -// -// -- This is a dual command -- -// Cypress.Commands.add('dismiss', { prevSubject: 'optional'}, (subject, options) => { ... }) -// -// -// -- This will overwrite an existing command -- -// Cypress.Commands.overwrite('visit', (originalFn, url, options) => { ... }) - -Cypress.Commands.add('visitHome', () => { - cy.visit('/'); - cy.get('button', { timeout: 30000 }).should('be.visible'); -}) - -Cypress.Commands.add('resetState', () => { - // iframe.contentWindow.indexedDB.deleteDatabase() がchromeのバグで使用できないため、indexedDBを無効化している。 - // see https://github.com/misskey-dev/misskey/issues/13605#issuecomment-2053652123 - /* - cy.window().then(win => { - win.indexedDB.deleteDatabase('keyval-store'); - }); - */ - cy.request('POST', '/api/reset-db', {}).as('reset'); - cy.get('@reset').its('status').should('equal', 204); - cy.reload(true); -}); - -Cypress.Commands.add('registerUser', (username, password, isAdmin = false) => { - const route = isAdmin ? '/api/admin/accounts/create' : '/api/signup'; - - cy.request('POST', route, { - username: username, - password: password, - ...(isAdmin ? { setupPassword: 'example_password_please_change_this_or_you_will_get_hacked' } : {}), - }).its('body').as(username); -}); - -Cypress.Commands.add('login', (username, password) => { - cy.visitHome(); - - cy.intercept('POST', '/api/signin-flow').as('signin'); - - cy.get('[data-cy-signin]').click(); - cy.get('[data-cy-signin-page-input]').should('be.visible', { timeout: 1000 }); - cy.get('[data-cy-signin-username] input').type(`${username}{enter}`); - cy.get('[data-cy-signin-page-password]').should('be.visible', { timeout: 10000 }); - cy.get('[data-cy-signin-password] input').type(`${password}{enter}`); - - cy.wait('@signin').as('signedIn'); -}); diff --git a/cypress/support/e2e.ts b/cypress/support/e2e.ts deleted file mode 100644 index 827a326d18..0000000000 --- a/cypress/support/e2e.ts +++ /dev/null @@ -1,34 +0,0 @@ -// *********************************************************** -// This example support/index.js is processed and -// loaded automatically before your test files. -// -// This is a great place to put global configuration and -// behavior that modifies Cypress. -// -// You can change the location of this file or turn off -// automatically serving support files with the -// 'supportFile' configuration option. -// -// You can read more here: -// https://on.cypress.io/configuration -// *********************************************************** - -// Import commands.js using ES2015 syntax: -import './commands' - -// Alternatively you can use CommonJS syntax: -// require('./commands') - -Cypress.on('uncaught:exception', (err, runnable) => { - if ([ - 'The source image cannot be decoded', - - // Chrome - 'ResizeObserver loop limit exceeded', - - // Firefox - 'ResizeObserver loop completed with undelivered notifications', - ].some(msg => err.message.includes(msg))) { - return false; - } -}); diff --git a/cypress/support/index.ts b/cypress/support/index.ts deleted file mode 100644 index c1bed21979..0000000000 --- a/cypress/support/index.ts +++ /dev/null @@ -1,19 +0,0 @@ -declare global { - namespace Cypress { - interface Chainable { - login(username: string, password: string): Chainable; - - registerUser( - username: string, - password: string, - isAdmin?: boolean - ): Chainable; - - resetState(): Chainable; - - visitHome(): Chainable; - } - } -} - -export {} diff --git a/cypress/tsconfig.json b/cypress/tsconfig.json deleted file mode 100644 index b91e34dc12..0000000000 --- a/cypress/tsconfig.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "compilerOptions": { - "lib": ["dom"], - "target": "esnext", - "types": ["cypress", "node"] - }, - "include": ["./**/*.ts"] -} diff --git a/locales/ca-ES.yml b/locales/ca-ES.yml index be6ece175e..a4e5d21cb0 100644 --- a/locales/ca-ES.yml +++ b/locales/ca-ES.yml @@ -619,6 +619,8 @@ output: "Sortida" script: "Script" disablePagesScript: "Desactivar AiScript a les pàgines " updateRemoteUser: "Actualitzar la informació de l'usuari remot" +unsetMfa: "Desactiva l'autenticació de dos factors" +unsetMfaConfirm: "Voldries desactivar l'autenticació de dos factors?" unsetUserAvatar: "Desactiva l'avatar " unsetUserAvatarConfirm: "Segur que vols desactivar l'avatar?" unsetUserBanner: "Desactiva el bàner " @@ -1361,14 +1363,11 @@ information: "Informació" chat: "Xat" directMessage: "Xateja amb aquest usuari" directMessage_short: "Missatge" -migrateOldSettings: "Migrar la configuració anterior" -migrateOldSettings_description: "Normalment això es fa automàticament, però si la transició no es fa, el procés es pot iniciar manualment. S'esborrarà la configuració actual." compress: "Comprimir " right: "Dreta" bottom: "A baix " top: "A dalt " embed: "Incrustar" -settingsMigrating: "Estem migrant la teva configuració. Si us plau espera un moment... (També pots fer la migració més tard, manualment, anant a Preferències → Altres → Migrar configuració antiga)" readonly: "Només lectura" goToDeck: "Tornar al tauler" federationJobs: "Treballs de federació" @@ -1420,6 +1419,8 @@ addToEmojiPalette: "Afegeix al calaix d'emojis" emojiPaletteAlreadyAddedConfirm: "Aquest emoji ja està inclòs en aquest calaix d'emojis. Vols afegir-lo de nou?" append: "Afegeix al final" prepend: "Afegeix al principi" +urlPreviewSensitiveList: "Llista d'URLs per restringir la visualització de miniatures" +urlPreviewSensitiveListDescription: "Si separeu els termes amb un espai, s'interpretarà com una condició 'AND'; si els separeu amb un salt de línia, s'interpretarà com una condició 'OR'. Si envolupeu els termes amb barres obliqües, s'interpretarà com una expressió regular. Si es troba una coincidència, la miniatura ja no es mostrarà." _imageEditing: _vars: caption: "Títol de l'arxiu" @@ -2152,6 +2153,15 @@ _sensitiveMediaDetection: setSensitiveFlagAutomaticallyDescription: "Els resultats de la detecció interna seran desats, inclòs si aquesta opció es troba desactivada." analyzeVideos: "Activar anàlisis de vídeos " analyzeVideosDescription: "Analitzar els vídeos a més de les imatges. Això incrementarà lleugerament la càrrega del servidor." + externalServiceInfo: "La detecció de mitjans sensibles s'ha delegat a un servei extern (sensitive-detector). Per utilitzar aquesta funció, cal configurar un servei sidecar separat i establir els detalls de connexió que es detallen a continuació. Si no es configuren els detalls de connexió, no es durà a terme cap detecció (el contingut es tractarà com a no sensible)." + apiUrl: "URL per connectar-se al servei de verificació" + apiUrlDescription: "L'URL base del servei de detector de sensibilitat (per exemple, http://localhost:3009). Si us connecteu a un servei en una xarxa privada, afegiu la xarxa de destinació a la configuració `allowedPrivateNetworks` del fitxer de configuració. Si utilitzeu un proxy, configureu també `proxyBypassHosts`. Si es deixa en blanc, no es realitzaran comprovacions de sensibilitat." + apiKey: "Clau de l'API" + apiKeyDescription: "Introduïu això si l'autenticació (token Bearer) està configurada al costat del servei d'autenticació. Si no està configurada, deixeu aquest camp en blanc." + timeout: "Temps d'espera (mil·lisegons)" + timeoutDescription: "1 Aquesta és la durada del temps mort per sol·licitud de validació." + maxImagesPerRequest: "1 Nombre màxim d'imatges per sol·licitud" + maxImagesPerRequestDescription: "1 Quan s'analitzen múltiples fotogrames, com en un vídeo, aquest és el nombre màxim d'imatges que es poden enviar en una única petició. Qualsevol imatge que superi aquest límit es dividirà en parts i s'enviarà seqüencialment. Assegureu-vos que aquest ajust no superi el valor de `maxParts` al costat del `sensitive-detector` (per defecte: 10). Si es supera aquest límit, totes les imatges d'aquest lot es consideraran no sensibles." _emailUnavailable: used: "Aquest correu electrònic ja s'està fent servir" format: "El format del correu electrònic és invàlid " @@ -2464,6 +2474,7 @@ _permissions: "read:admin:show-moderation-log": "Veure registre de moderació " "read:admin:show-user": "Veure informació privada de l'usuari " "write:admin:suspend-user": "Suspendre usuari" + "write:admin:unset-mfa": "Desactiva l'autenticació de dos factors per a un usuari" "write:admin:unset-user-avatar": "Esborrar avatar d'usuari " "write:admin:unset-user-banner": "Esborrar bàner de l'usuari " "write:admin:unsuspend-user": "Treure la suspensió d'un usuari" @@ -2979,6 +2990,7 @@ _moderationLogTypes: createAvatarDecoration: "Decoració de l'avatar creada" updateAvatarDecoration: "S'ha actualitzat la decoració de l'avatar " deleteAvatarDecoration: "S'ha esborrat la decoració de l'avatar " + unsetMfa: "Desactiva l'autenticació de dos factors per a l'usuari" unsetUserAvatar: "Esborrar l'avatar d'aquest usuari" unsetUserBanner: "Esborrar el bàner d'aquest usuari" createSystemWebhook: "Crear un SystemWebhook" diff --git a/locales/de-DE.yml b/locales/de-DE.yml index a53634aa58..17b1a8c7e7 100644 --- a/locales/de-DE.yml +++ b/locales/de-DE.yml @@ -1358,14 +1358,11 @@ information: "Über" chat: "Chat" directMessage: "Mit dem Benutzer chatten" directMessage_short: "Nachrichten" -migrateOldSettings: "Alte Client-Einstellungen migrieren" -migrateOldSettings_description: "Dies sollte normalerweise automatisch geschehen, aber wenn die Migration aus irgendeinem Grund nicht erfolgreich war, kannst du den Migrationsprozess selbst manuell auslösen. Die aktuellen Konfigurationsinformationen werden dabei überschrieben." compress: "Komprimieren" right: "Rechts" bottom: "Unten" top: "Oben" embed: "Einbetten" -settingsMigrating: "Deine Einstellungen werden gerade migriert. Bitte warte einen Moment... (Du kannst die Einstellungen später auch manuell migrieren, indem du zu Einstellungen → Anderes → Alte Einstellungen migrieren gehst)" readonly: "Nur Lesezugriff" goToDeck: "Zurück zum Deck" federationJobs: "Föderation Jobs" diff --git a/locales/en-US.yml b/locales/en-US.yml index d1627feb14..813d7ccc9a 100644 --- a/locales/en-US.yml +++ b/locales/en-US.yml @@ -619,6 +619,8 @@ output: "Output" script: "Script" disablePagesScript: "Disable AiScript on Pages" updateRemoteUser: "Update remote user information" +unsetMfa: "Reset two-factor authentication" +unsetMfaConfirm: "Are you sure you want to reset two-factor authentication?" unsetUserAvatar: "Unset avatar" unsetUserAvatarConfirm: "Are you sure you want to unset the avatar?" unsetUserBanner: "Unset banner" @@ -1361,14 +1363,11 @@ information: "About" chat: "Chat" directMessage: "Chat with user" directMessage_short: "Message" -migrateOldSettings: "Migrate old client settings" -migrateOldSettings_description: "This should be done automatically but if for some reason the migration was not successful, you can trigger the migration process yourself manually. The current configuration information will be overwritten." compress: "Compress" right: "Right" bottom: "Bottom" top: "Top" embed: "Embed" -settingsMigrating: "Settings are being migrated, please wait a moment... (You can also migrate manually later by going to Settings→Others→Migrate old settings)" readonly: "Read only" goToDeck: "Return to Deck" federationJobs: "Federation Jobs" @@ -1420,6 +1419,8 @@ addToEmojiPalette: "Add to emoji palette" emojiPaletteAlreadyAddedConfirm: "This emoji is already included in this emoji palette. Do you want to add it again?" append: "Append to end" prepend: "Append to beginning" +urlPreviewSensitiveList: "URL to restrict thumbnail display" +urlPreviewSensitiveListDescription: "Use spaces to specify AND conditions, and line breaks to specify OR conditions. Enclose text in slashes to use regular expressions. If a match is found, the thumbnail will be hidden." _imageEditing: _vars: caption: "File caption" @@ -2152,6 +2153,15 @@ _sensitiveMediaDetection: setSensitiveFlagAutomaticallyDescription: "The results of the internal detection will be retained even if this option is turned off." analyzeVideos: "Enable analysis of videos" analyzeVideosDescription: "Analyzes videos in addition to images. This will slightly increase the load on the server." + externalServiceInfo: "The detection of sensitive media has been offloaded to an external service (sensitive-detector). To use this feature, you must set up a separate service and configure the connection details provided below. If no connection details are configured, no detection will be performed (it will be treated as non-sensitive)." + apiUrl: "Detection service endpoint URL" + apiUrlDescription: "The base URL for the sensitive-detector service (e.g., http://localhost:3009). If you are connecting to a service on a private network, please allow the target network in the allowedPrivateNetworks setting in the configuration file. If you are using a proxy, please also configure proxyBypassHosts. If left blank, sensitive media detection will not be performed." + apiKey: "API key" + apiKeyDescription: "Enter this if authentication (Bearer token) is configured on the detector service. If it is not configured, please leave it blank." + timeout: "Timeout (Milliseconds)" + timeoutDescription: "Timeout duration for each judgment request." + maxImagesPerRequest: "Max images per request" + maxImagesPerRequestDescription: "Maximum number of images that can be sent in a single request when processing multi-frame at once, such as videos. Any images exceeding this limit will be split and sent sequentially. Please ensure this is set to not exceed the maxParts setting (default: 10) on the detector service. If it exceeds this limit, all items in that chunk will be treated as non-sensitive." _emailUnavailable: used: "This email address is already being used" format: "The format of this email address is invalid" @@ -2464,6 +2474,7 @@ _permissions: "read:admin:show-moderation-log": "View moderation log" "read:admin:show-user": "View private user info" "write:admin:suspend-user": "Suspend user" + "write:admin:unset-mfa": "Reset two-factor authentication for the user" "write:admin:unset-user-avatar": "Remove user avatar" "write:admin:unset-user-banner": "Remove user banner" "write:admin:unsuspend-user": "Unsuspend user" @@ -2979,6 +2990,7 @@ _moderationLogTypes: createAvatarDecoration: "Avatar decoration created" updateAvatarDecoration: "Avatar decoration updated" deleteAvatarDecoration: "Avatar decoration deleted" + unsetMfa: "Reset two-factor authentication for the user" unsetUserAvatar: "User avatar unset" unsetUserBanner: "User banner unset" createSystemWebhook: "System Webhook created" diff --git a/locales/es-ES.yml b/locales/es-ES.yml index 82879fdff7..4dee653286 100644 --- a/locales/es-ES.yml +++ b/locales/es-ES.yml @@ -584,7 +584,7 @@ showFixedPostForm: "Mostrar formulario de publicación sobre la línea de tiempo showFixedPostFormInChannel: "Mostrar el formulario de publicación por encima de la cronología (Canales)" withRepliesByDefaultForNewlyFollowed: "Incluir por defecto respuestas de usuarios recién seguidos en la línea de tiempo" newNoteRecived: "Tienes una nota nueva" -newNote: "Nueva nota" +newNote: "Nuevas notas" sounds: "Sonidos" sound: "Sonidos" notificationSoundSettings: "Configuración del sonido de las notificaciones" @@ -619,6 +619,8 @@ output: "Salida" script: "Script" disablePagesScript: "Deshabilitar AiScript en Páginas" updateRemoteUser: "Actualizar información de usuario remoto" +unsetMfa: "Desactivar la autenticación de dos factores" +unsetMfaConfirm: "¿Desea desactivar la autenticación de dos factores?" unsetUserAvatar: "Quitar avatar" unsetUserAvatarConfirm: "¿Confirmas que quieres quitar tu avatar?" unsetUserBanner: "Quitar banner" @@ -1219,7 +1221,7 @@ keepScreenOn: "Mantener pantalla encendida" verifiedLink: "Propiedad del enlace verificada" notifyNotes: "Notificar nuevas notas" unnotifyNotes: "Dejar de notificar nuevas notas" -notifyUsers: "Usuarios que han activado las notificaciones de publicaciones" +notifyUsers: "Cuentas con notificaciones activadas" authentication: "Autenticación" authenticationRequiredToContinue: "Por favor, autentifícate para continuar" dateAndTime: "Fecha y hora" @@ -1361,14 +1363,11 @@ information: "Información" chat: "Chat" directMessage: "Chatear" directMessage_short: "Mensajes" -migrateOldSettings: "Migrar la configuración anterior" -migrateOldSettings_description: "Esto debería hacerse automáticamente, pero si por alguna razón la migración no ha tenido éxito, puede activar usted mismo el proceso de migración manualmente. Se sobrescribirá la información de configuración actual." compress: "Compresión de la imagen" right: "Derecha" bottom: "Abajo" top: "Arriba" embed: "Insertar" -settingsMigrating: "La configuración está siendo migrada, por favor espera un momento... (También puedes migrar manualmente más tarde yendo a Ajustes otros migrar configuración antigua" readonly: "Solo Lectura" goToDeck: "Volver al Deck" federationJobs: "Trabajos de Federación" @@ -1420,6 +1419,8 @@ addToEmojiPalette: "Añadir a la paleta de emojis" emojiPaletteAlreadyAddedConfirm: "Este emoji ya está incluido en esta paleta de emojis. ¿Quieres volver a añadirlo?" append: "Añadir al final" prepend: "Añadir al principio" +urlPreviewSensitiveList: "URL para restringir la visualización de miniaturas" +urlPreviewSensitiveListDescription: "Si se separan con un espacio, se interpretará como una condición «AND»; si se separan con un salto de línea, se interpretará como una condición «OR». Si se escriben entre barras, se interpretarán como expresiones regulares. Si se encuentra una coincidencia, no se mostrará la miniatura." _imageEditing: _vars: caption: "Título del archivo" @@ -2152,6 +2153,15 @@ _sensitiveMediaDetection: setSensitiveFlagAutomaticallyDescription: "Los resultados de la detección interna pueden ser retenidos incluso si la opción está desactivada." analyzeVideos: "Habilitar el análisis de videos" analyzeVideosDescription: "Analizar videos en adición a las imágenes. Esto puede incrementar ligeramente la carga del servidor." + externalServiceInfo: "La detección de contenidos sensibles se ha externalizado a un servicio externo (sensitive-detector). Para utilizar esta función, es necesario configurar por separado un servicio «sidecar» y establecer los datos de conexión que se indican a continuación. Si no se configuran los datos de conexión, no se realizará la detección (se tratará como contenido no sensible)." + apiUrl: "URL de conexión al servicio de verificación" + apiUrlDescription: "URL base del servicio «sensitive-detector» (por ejemplo: http://localhost:3009). Si te conectas a un servicio situado en una red privada, debes permitir la red de destino en el parámetro «allowedPrivateNetworks» del archivo de configuración. Si utilizas un proxy, configura también el parámetro «proxyBypassHosts». Si se deja en blanco, no se realizará la evaluación de datos sensibles." + apiKey: "Clave API" + apiKeyDescription: "Introduce este dato si se ha configurado la autenticación (token Bearer) en el servicio de validación. Si no se ha configurado, déjalo en blanco." + timeout: "Tiempo de espera (milisegundos)" + timeoutDescription: "Duración del tiempo de espera para cada solicitud de resolución." + maxImagesPerRequest: "Número máximo de imágenes por solicitud" + maxImagesPerRequestDescription: "Es el número máximo de imágenes que se pueden enviar en una sola solicitud al analizar varios fotogramas, como en un vídeo. Las imágenes que superen este límite se dividirán y se enviarán de forma secuencial. Configúralo de manera que no se supere el valor de «maxParts» de sensitive-detector (por defecto: 10). Si se supera este límite, todas las imágenes de ese fragmento se considerarán no sensibles." _emailUnavailable: used: "Ya fue usado" format: "Formato no válido." @@ -2464,6 +2474,7 @@ _permissions: "read:admin:show-moderation-log": "Ver log de moderación" "read:admin:show-user": "Ver información privada de usuario" "write:admin:suspend-user": "Suspender cuentas de usuario" + "write:admin:unset-mfa": "Desactivar la autenticación de dos factores del usuario" "write:admin:unset-user-avatar": "Quitar avatares de usuario" "write:admin:unset-user-banner": "Quitar banner de usuarios" "write:admin:unsuspend-user": "Quitar suspensión de cuentas de usuario" @@ -2979,6 +2990,7 @@ _moderationLogTypes: createAvatarDecoration: "Decoración de avatar creada" updateAvatarDecoration: "Decoración de avatar actualizada" deleteAvatarDecoration: "Decoración de avatar eliminada" + unsetMfa: "Desactivar la autenticación de dos factores del usuario" unsetUserAvatar: "Quitar decoración de avatar de este usuario" unsetUserBanner: "Quitar banner de este usuario" createSystemWebhook: "Crear un SystemWebhook" diff --git a/locales/id-ID.yml b/locales/id-ID.yml index 24127b6a84..1b3e057583 100644 --- a/locales/id-ID.yml +++ b/locales/id-ID.yml @@ -1358,6 +1358,7 @@ _chat: invitations: "Undang" history: "Riwayat obrolan" noHistory: "Tidak ada riwayat" + join: "Gabung" members: "Anggota" home: "Beranda" send: "Kirim" @@ -2555,6 +2556,8 @@ _deck: useSimpleUiForNonRootPages: "Gunakan antarmuka sederhana ke halaman yang dituju" usedAsMinWidthWhenFlexible: "Lebar minimum akan digunakan untuk ini ketika opsi \"Atur-otomatis lebar\" dinyalakan" flexible: "Atur-otomatis lebar" + _howToUse: + settings_title: "Pengaturan UI" _columns: main: "Utama" widgets: "Widget" diff --git a/locales/it-IT.yml b/locales/it-IT.yml index 06880a4ea2..564dd458ce 100644 --- a/locales/it-IT.yml +++ b/locales/it-IT.yml @@ -619,6 +619,8 @@ output: "Output" script: "Script" disablePagesScript: "Disabilitare AiScript nelle pagine" updateRemoteUser: "Aggiorna dati dal profilo remoto" +unsetMfa: "Rimuovere l'autenticazione a due fattori (2FA/MFA)" +unsetMfaConfirm: "Vuoi davvero rimuovere l'autenticazione a due fattori?" unsetUserAvatar: "Rimozione foto profilo" unsetUserAvatarConfirm: "Vuoi davvero rimuovere la foto profilo?" unsetUserBanner: "Rimuovi intestazione profilo" @@ -1361,14 +1363,11 @@ information: "Informazioni" chat: "Chat" directMessage: "Chattare insieme" directMessage_short: "Messaggio" -migrateOldSettings: "Migrare le vecchie impostazioni" -migrateOldSettings_description: "Di solito, viene fatto automaticamente. Se per qualche motivo non fossero migrate con successo, è possibile avviare il processo di migrazione manualmente, sovrascrivendo le configurazioni attuali." compress: "Compressione" right: "Destra" bottom: "Sotto" top: "Sopra" embed: "Incorporare" -settingsMigrating: "Migrazione delle impostazioni. Attendere prego ... (Puoi anche migrare manualmente in un secondo momento, nel menu: Impostazioni → Altro → Migrazione delle impostazioni)" readonly: "Sola lettura" goToDeck: "Torna al Deck" federationJobs: "Coda di federazione" @@ -1420,6 +1419,8 @@ addToEmojiPalette: "Aggiungi alla tavolozza emoji" emojiPaletteAlreadyAddedConfirm: "Questa emoji è già inclusa in nella tavolozza. Vuoi davvero aggiungerla?" append: "Accodare" prepend: "Anteporre" +urlPreviewSensitiveList: "URL da impedire alla vista delle anteprime" +urlPreviewSensitiveListDescription: "Separando con uno spazio si indica E, separando con una linea si indica O. Circondando con barre / si indica una Espressione Regolare.\nLe URL che coincidono con le indicazioni non verranno visualizzate." _imageEditing: _vars: caption: "Didascalia dell'immagine" @@ -2152,6 +2153,15 @@ _sensitiveMediaDetection: setSensitiveFlagAutomaticallyDescription: "Anche se questa impostazione è disattivata, il risultato della decisione viene conservato internamente." analyzeVideos: "Abilitazione dell'analisi video." analyzeVideosDescription: "Assicuratevi che vengano analizzati anche i video oltre alle immagini fisse. Il carico del server aumenterà leggermente." + externalServiceInfo: "Abbiamo spostato esternamente il riconoscimento di media espliciti (sensitive-detector). Per usufruirne devi impostare un servizio separato e indicare di seguito la destinazione. Se non è impostata, non viene emesso alcun giudizio (contenuto NON esplicito)." + apiUrl: "URL di connessione a Sensitive-Detector" + apiUrlDescription: "L'URL di base del servizio (ad esempio http://localhost:3009). Collegandosi a una rete privata, autorizzare la rete nel parametro allowedPrivateNetworks nel file di configurazione.\nCollegandosi con un proxy, è anche necessario impostare proxyBypassHosts. Nel caso il campo sia vuoto, non vengono emessi giudizi di sensibilità." + apiKey: "Chiave API" + apiKeyDescription: "Indicare il token di autenticazione (Bearer token), soltanto se occorre, altrimenti lasciare vuoto." + timeout: "Timeout (ms)" + timeoutDescription: "Tempo limite per ogni richiesta" + maxImagesPerRequest: "Numero massimo di media per ogni richiesta" + maxImagesPerRequestDescription: "In caso ci siano più fotogrammi, come nei video, questo è il numero massimo di immagini da inviare in una richiesta. Oltre questo numero, il totale sarà diviso e inviato in sequenza.\nImpostare in modo che non superi il valore maxParts (default: 10) in Sensitive-Detector. Se viene superato, il media sarà considerato non esplicito." _emailUnavailable: used: "Email già in uso" format: "Formato email non valido" @@ -2464,6 +2474,7 @@ _permissions: "read:admin:show-moderation-log": "Vedere lo storico di moderazione" "read:admin:show-user": "Vedere le informazioni private dei profili" "write:admin:suspend-user": "Sospendere i profili" + "write:admin:unset-mfa": "Può rimuovere l'autenticazione a due fattori (2FA/MFA)" "write:admin:unset-user-avatar": "Rimuovere la foto profilo dai profili" "write:admin:unset-user-banner": "Rimuovere l'immagine testata dai profili" "write:admin:unsuspend-user": "Rimuovere la sospensione ai profili" @@ -2979,6 +2990,7 @@ _moderationLogTypes: createAvatarDecoration: "Crea una decorazione della foto profilo" updateAvatarDecoration: "Modifica una decorazione della foto profilo" deleteAvatarDecoration: "Elimina una decorazione della foto profilo" + unsetMfa: "Rimossa l'autenticazione a due fattori (2FA7MFA)" unsetUserAvatar: "Toglie una foto profilo" unsetUserBanner: "Toglie una immagine di intestazione profilo" createSystemWebhook: "Aggiunge un System Webhook" diff --git a/locales/ja-JP.yml b/locales/ja-JP.yml index 44dfea6a54..56fb6bc544 100644 --- a/locales/ja-JP.yml +++ b/locales/ja-JP.yml @@ -1421,6 +1421,8 @@ addToEmojiPalette: "絵文字パレットに追加" emojiPaletteAlreadyAddedConfirm: "この絵文字はすでにこの絵文字パレットに含まれています。追加しなおしますか？" append: "末尾に追加" prepend: "先頭に追加" +urlPreviewSensitiveList: "サムネイルの表示を制限するURL" +urlPreviewSensitiveListDescription: "スペースで区切るとAND指定になり、改行で区切るとOR指定になります。スラッシュで囲むと正規表現になります。一致した場合、サムネイルが表示されなくなります。" _imageEditing: _vars: diff --git a/locales/ja-KS.yml b/locales/ja-KS.yml index 75f9b8224d..802695ea0e 100644 --- a/locales/ja-KS.yml +++ b/locales/ja-KS.yml @@ -1332,9 +1332,6 @@ preferenceSyncConflictChoiceCancel: "同期の有効化はやめとくわ" postForm: "投稿フォーム" information: "情報" directMessage: "チャットしよか" -migrateOldSettings: "旧設定情報をお引っ越し" -migrateOldSettings_description: "通常これは自動で行われるはずなんやけど、なんかの理由で上手く移行できへんかったときは手動で移行処理をポチっとできるで。今の設定情報は上書きされるで。" -settingsMigrating: "設定を移行しとるで。ちょっと待っとってな... (後で、設定→その他→旧設定情報を移行 で手動で移行することもできるで)" driveAboutTip: "ドライブでは、今までアップロードしたファイルがずらーっと表示されるで。
\nノートにファイルをもっかいのっけたり、あとで投稿するファイルをその辺に置いとくこともできるねん。
\nファイルをほかすと、前にそのファイルをのっけた全部の場所(ノート、ページ、アバター、バナー等)からも見えんくなるから気いつけてな。
\nフォルダを作って整理することもできるで。" turnItOn: "オンにしとこ" turnItOff: "オフでええわ" diff --git a/locales/ko-KR.yml b/locales/ko-KR.yml index 5b21341ae9..6cb6d46a89 100644 --- a/locales/ko-KR.yml +++ b/locales/ko-KR.yml @@ -619,6 +619,8 @@ output: "출력" script: "스크립트" disablePagesScript: "Pages 에서 AiScript 를 사용하지 않음" updateRemoteUser: "리모트 유저 정보 갱신" +unsetMfa: "2단계 인증 해제" +unsetMfaConfirm: "2단계 인증을 해제하시겠습니까?" unsetUserAvatar: "아바타 제거" unsetUserAvatarConfirm: "아바타를 제거할까요?" unsetUserBanner: "배너 제거" @@ -1361,14 +1363,11 @@ information: "정보" chat: "채팅" directMessage: "채팅하기" directMessage_short: "메시지" -migrateOldSettings: "기존 설정 정보를 이전" -migrateOldSettings_description: "보통은 자동으로 이루어지지만, 어떤 이유로 인해 성공적으로 이전이 이루어지지 않는 경우 수동으로 이전을 실행할 수 있습니다. 현재 설정 정보는 덮어쓰게 됩니다." compress: "압축" right: "오른쪽" bottom: "아래" top: "위" embed: "임베드" -settingsMigrating: "설정을 이전하는 중입니다. 잠시 기다려주십시오... (나중에 '환경설정 → 기타 → 기존 설정 정보를 이전'에서 수동으로 이전할 수도 있습니다)" readonly: "읽기 전용" goToDeck: "덱으로 돌아가기" federationJobs: "연합 작업" @@ -1420,6 +1419,8 @@ addToEmojiPalette: "이모지 팔레트에 추가" emojiPaletteAlreadyAddedConfirm: "이 이모지는 이미 이 이모지 팔레트에 포함돼있습니다. 다시 추가하시겠습니까?" append: "맨뒤에 추가" prepend: "맨앞에 추가" +urlPreviewSensitiveList: "썸네일 표시 제한 URL" +urlPreviewSensitiveListDescription: "공백으로 구분하면 AND 지정으로 되고, 줄내림으로 구분하면 OR 지정으로 됩니다. 슬래시로 감싸면 정규 표현으로 됩니다. 일치한 경우에 썸네일이 표시되지 않게 됩니다." _imageEditing: _vars: caption: "파일 설명" @@ -2152,6 +2153,15 @@ _sensitiveMediaDetection: setSensitiveFlagAutomaticallyDescription: "이 설정을 해제해도 탐지 결과는 유지됩니다." analyzeVideos: "동영상도 같이 확인하기" analyzeVideosDescription: "사진 뿐만 아니라 동영상의 NSFW 여부도 탐지합니다. 서버의 부하를 약간 증가시킵니다." + externalServiceInfo: "민감한 미디어 판정은 외부 서비스(sensitive-detector)로 분리됐습니다. 이 기능을 이용하려면 별도 사이드카 서비스를 설정하고, 아래의 접속 위치를 설정해야 합니다. 접속 위치가 설정되지 않은 경우에는 판정이 이루어지지 않습니다. (민감하지 않음 처리)" + apiUrl: "판정 서비스의 접속 위치 URL" + apiUrlDescription: "sensitive-detector 서비스의 베이스 URL(예시: http://localhost:3009). 프라이빗 네트워크상의 서비스에 접속하는 경우에는 설정 파일의 allowedPrivateNetworks로 접속 위치 네트워크를 허가해 주십시오. 프록시를 사용하고 있는 경우에는 proxyBypassHosts도 설정해 주십시오. 비어있으면 민감함 판정은 이루어지지 않습니다." + apiKey: "API 키" + apiKeyDescription: "판정 서비스 측에서 인증(Bearer 토큰)을 설정하고 있는 경우에 입력합니다. 설정하고 있지 않은 경우에는 빈칸으로 둬주십시오." + timeout: "타임아웃 (밀리초)" + timeoutDescription: "판정 요청 1회당 타임아웃 시간입니다." + maxImagesPerRequest: "한 요청당 최대 이미지 수" + maxImagesPerRequestDescription: "동영상 등 여러 프레임을 판정할 때, 한 번의 요청에 모아서 보내는 이미지의 최대 장수입니다. 이를 넘으면 분할해 순차적으로 송신됩니다. sensitive-detector 측의 maxParts 설정(기본: 10)을 남지 않도록 설정해 주십시오. 넘은 경우에는 그 청크는 전부 민감하지 않음 처리로 됩니다." _emailUnavailable: used: "이 메일 주소는 사용중입니다" format: "형식이 올바르지 않습니다" @@ -2464,6 +2474,7 @@ _permissions: "read:admin:show-moderation-log": "조정 기록 보기" "read:admin:show-user": "유저 개인정보 보기" "write:admin:suspend-user": "유저 정지하기" + "write:admin:unset-mfa": "사용자의 2단계 인증 해제" "write:admin:unset-user-avatar": "유저 아바타 삭제하기" "write:admin:unset-user-banner": "유저 배너 삭제하기" "write:admin:unsuspend-user": "유저 정지 해제하기" @@ -2979,6 +2990,7 @@ _moderationLogTypes: createAvatarDecoration: "아바타 장식 만들기" updateAvatarDecoration: "아바타 장식 수정" deleteAvatarDecoration: "아바타 장식 삭제" + unsetMfa: "사용자의 2단계 인증 해제" unsetUserAvatar: "유저 아바타 제거" unsetUserBanner: "유저 배너 제거" createSystemWebhook: "SystemWebhook을 생성" diff --git a/locales/pt-PT.yml b/locales/pt-PT.yml index 436692db3d..cadbc1e603 100644 --- a/locales/pt-PT.yml +++ b/locales/pt-PT.yml @@ -1347,14 +1347,11 @@ information: "Sobre" chat: "Conversas" directMessage: "Conversar com usuário" directMessage_short: "Mensagem" -migrateOldSettings: "Migrar configurações antigas de cliente" -migrateOldSettings_description: "Isso deve ser feito automaticamente. Caso o processo de migração tenha falhado, você pode acioná-lo manualmente. As informações atuais de migração serão substituídas." compress: "Comprimir" right: "Direita" bottom: "Inferior" top: "Superior" embed: "Embed" -settingsMigrating: "Configurações estão sendo migradas, aguarde... (Você pode migrar manualmente em Configurações→Outros→Migrar configurações antigas de cliente)" readonly: "Ler apenas" goToDeck: "Voltar ao Deck" federationJobs: "Tarefas de Federação" diff --git a/locales/th-TH.yml b/locales/th-TH.yml index 69b8d69788..da1e8659af 100644 --- a/locales/th-TH.yml +++ b/locales/th-TH.yml @@ -1358,14 +1358,11 @@ information: "เกี่ยวกับ" chat: "แชต" directMessage: "แชตเลย" directMessage_short: "ข้อความ" -migrateOldSettings: "ย้ายข้อมูลการตั้งค่าเก่า" -migrateOldSettings_description: "โดยปกติจะทำโดยอัตโนมัติ แต่หากด้วยเหตุผลบางประการที่ไม่สามารถย้ายได้สำเร็จ สามารถสั่งย้ายด้วยตนเองได้ การตั้งค่าปัจจุบันจะถูกเขียนทับ" compress: "บีบอัด" right: "ขวา" bottom: "ล่าง" top: "บน" embed: "ฝัง" -settingsMigrating: "กำลังย้ายการตั้งค่า กรุณารอสักครู่... (สามารถย้ายด้วยตนเองภายหลังได้ที่ การตั้งค่า → อื่นๆ → ย้ายข้อมูลการตั้งค่าเก่า)" readonly: "อ่านได้อย่างเดียว" goToDeck: "กลับไปยังเด็ค" federationJobs: "งานสหพันธ์" diff --git a/locales/tl-PH.yml b/locales/tl-PH.yml new file mode 100644 index 0000000000..ed97d539c0 --- /dev/null +++ b/locales/tl-PH.yml @@ -0,0 +1 @@ +--- diff --git a/locales/tr-TR.yml b/locales/tr-TR.yml index f98d684556..83a49d3850 100644 --- a/locales/tr-TR.yml +++ b/locales/tr-TR.yml @@ -1358,14 +1358,11 @@ information: "Hakkında" chat: "Sohbet" directMessage: "Kullanıcıyla sohbet et" directMessage_short: "Mesaj" -migrateOldSettings: "Eski istemci ayarlarını taşıma" -migrateOldSettings_description: "Bu işlem otomatik olarak yapılmalıdır, ancak herhangi bir nedenle geçiş başarısız olursa, geçiş işlemini manuel olarak kendin başlatabilirsin. Mevcut yapılandırma bilgileri üzerine yazılacaktır." compress: "Sıkıştır" right: "Sağ" bottom: "Alt" top: "Üst" embed: "Göm" -settingsMigrating: "Ayarlar taşınıyor, lütfen bir dakika bekle... (Daha sonra Ayarlar→Diğerler→Eski ayarları taşı seçeneğine giderek manuel olarak da taşıyabilirsin)" readonly: "Sadece okuma" goToDeck: "Güverteye Dön" federationJobs: "Federasyon İşleri" diff --git a/locales/uk-UA.yml b/locales/uk-UA.yml index 1cf4b29cd4..ca64191ab6 100644 --- a/locales/uk-UA.yml +++ b/locales/uk-UA.yml @@ -1314,14 +1314,11 @@ information: "Інформація" chat: "Чат" directMessage: "Чат із користувачем" directMessage_short: "Повідомлення" -migrateOldSettings: "Перенести минулі налаштування клієнта" -migrateOldSettings_description: "Це має відбутися автоматично, але якщо з якоїсь причини перенесення не вдалося, ви можете запустити його вручну. Поточні дані конфігурації буде перезаписано." compress: "Стиснути" right: "Праворуч" bottom: "Зверху" top: "Знизу" embed: "Вбудувати" -settingsMigrating: "Налаштування переносяться, зачекайте трохи... (Ви також можете перенести їх вручну пізніше: Налаштування → Інше → Перенести старі налаштування)" readonly: "Лише для читання" goToDeck: "Повернутися до Деки" federationJobs: "Завдання федерації" diff --git a/locales/vi-VN.yml b/locales/vi-VN.yml index a9d07a3020..6436fd4c0a 100644 --- a/locales/vi-VN.yml +++ b/locales/vi-VN.yml @@ -1219,8 +1219,6 @@ paste: "dán" postForm: "Mẫu đăng" information: "Giới thiệu" chat: "Trò chuyện" -migrateOldSettings: "Di chuyển cài đặt cũ" -migrateOldSettings_description: "Thông thường, quá trình này diễn ra tự động, nhưng nếu vì lý do nào đó mà quá trình di chuyển không thành công, bạn có thể kích hoạt thủ công quy trình di chuyển, quá trình này sẽ ghi đè lên thông tin cấu hình hiện tại của bạn." driveAboutTip: "Trong Drive, danh sách các tệp bạn đã tải lên trước đây sẽ được hiển thị.
\nBạn có thể sử dụng lại chúng khi đính kèm vào ghi chú, hoặc tải lên trước các tệp để đăng sau.
\nLưu ý rằng nếu bạn xóa một tệp, tệp đó cũng sẽ biến mất khỏi tất cả những nơi đã sử dụng tệp đó (ghi chú, trang, ảnh đại diện, biểu ngữ, v.v.).
\nBạn cũng có thể tạo các thư mục để sắp xếp chúng." inMinutes: "phút" inDays: "ngày" diff --git a/locales/zh-CN.yml b/locales/zh-CN.yml index acae660cfb..53f7e865ad 100644 --- a/locales/zh-CN.yml +++ b/locales/zh-CN.yml @@ -57,7 +57,7 @@ deleteAndEditConfirm: "要删除该帖并重新编辑吗？该帖下的所有回 addToList: "添加至列表" addToAntenna: "添加到天线" sendMessage: "发送消息" -copyRSS: "复制RSS" +copyRSS: "复制 RSS" copyUsername: "复制用户名" copyUserId: "复制用户 ID" copyNoteId: "复制帖子 ID" @@ -456,8 +456,8 @@ about: "关于" aboutMisskey: "关于 Misskey" administrator: "管理员" token: "Token (令牌)" -2fa: "双因素认证" -setupOf2fa: "设置双因素认证" +2fa: "双重验证" +setupOf2fa: "设置双重验证" totp: "验证器" totpDescription: "使用验证器输入一次性密码" moderator: "监察员" @@ -485,7 +485,7 @@ markAsReadAllNotifications: "将所有通知标为已读" markAsReadAllUnreadNotes: "将所有帖子标记为已读" markAsReadAllTalkMessages: "将所有私信标记为已读" help: "帮助" -inputMessageHere: "在此键入信息" +inputMessageHere: "在此输入信息" close: "关闭" invites: "邀请" members: "成员" @@ -619,6 +619,8 @@ output: "输出" script: "脚本" disablePagesScript: "禁用页面脚本" updateRemoteUser: "更新远程用户信息" +unsetMfa: "解除双重验证" +unsetMfaConfirm: "确认解除双重验证吗？" unsetUserAvatar: "清除头像" unsetUserAvatarConfirm: "要清除头像吗？" unsetUserBanner: "清除横幅" @@ -1361,14 +1363,11 @@ information: "关于" chat: "聊天" directMessage: "私信" directMessage_short: "消息" -migrateOldSettings: "迁移旧设置信息" -migrateOldSettings_description: "通常设置信息将自动迁移。但如果由于某种原因迁移不成功，则可以手动触发迁移过程。当前的配置信息将被覆盖。" compress: "压缩" right: "右" bottom: "下" top: "上" embed: "嵌入" -settingsMigrating: "正在迁移设置，请稍候。（之后也可以在设置 → 其它 → 迁移旧设置来手动迁移）" readonly: "只读" goToDeck: "返回至 Deck" federationJobs: "联邦作业" @@ -1420,6 +1419,8 @@ addToEmojiPalette: "添加至表情符号选择器" emojiPaletteAlreadyAddedConfirm: "此表情符号已存在于此表情符号选择器中。要再次添加吗？" append: "加到最后" prepend: "加到最前" +urlPreviewSensitiveList: "限制显示缩略图的 URL" +urlPreviewSensitiveListDescription: "AND 条件用空格分隔，OR 条件用换行符分隔，正则表达式用斜线包裹。成功匹配则不再显示缩略图。" _imageEditing: _vars: caption: "文件标题" @@ -2152,6 +2153,15 @@ _sensitiveMediaDetection: setSensitiveFlagAutomaticallyDescription: "即使关闭此配置，识别结果也会在内部保存。" analyzeVideos: "启用对视频的检测" analyzeVideosDescription: "除了静止图像之外，还对视频进行分析。服务器负载会略微增加。" + externalServiceInfo: "检测敏感媒体已分离至外部服务 (sensitive-detector)。若要使用，需额外部署 Sidecar 服务，并设置下方的连接 URL。未设定时将不会进行检测（视为非敏感媒体）。" + apiUrl: "检测服务的连接 URL" + apiUrlDescription: "sensitive-detector 服务的 base URL（如：http://localhost:3009）。若是连接至部署在专用网络上的服务，请在配置文件中的 allowedPrivateNetworks 里允许目标网络。若是使用了代理，请一并设置 proxyBypassHosts。留空则不进行敏感媒体检测。" + apiKey: "API 密钥" + apiKeyDescription: "若服务端有设置验证（Bearer token）则填写，未设置则留空。" + timeout: "超时（毫秒）" + timeoutDescription: "此为单次检测请求的超时时长。" + maxImagesPerRequest: "单次检测请求最大图像数量" + maxImagesPerRequestDescription: "此为在检测动画等多帧图像时，单次请求中可发送的图像数量上限。超出此值时动画将被拆分并按序发送。请勿将此值设为超出 sensitive-detector 侧的 maxParts 的值（默认：10），否则对应的分块将全被视为非敏感媒体。" _emailUnavailable: used: "已经被使用过" format: "无效的格式" @@ -2464,6 +2474,7 @@ _permissions: "read:admin:show-moderation-log": "查看管理日志" "read:admin:show-user": "查看用户的非公开信息" "write:admin:suspend-user": "冻结用户" + "write:admin:unset-mfa": "解除用户的双重验证" "write:admin:unset-user-avatar": "删除用户头像" "write:admin:unset-user-banner": "删除用户横幅" "write:admin:unsuspend-user": "解除用户冻结" @@ -2589,7 +2600,7 @@ _widgetOptions: _jobQueue: sound: "播放音效" _rss: - url: "RSS feed 的 URL" + url: "RSS 订阅源网址" refreshIntervalSec: "更新间隔（秒）" maxEntries: "最大显示个数" _rssTicker: @@ -2979,6 +2990,7 @@ _moderationLogTypes: createAvatarDecoration: "新建头像挂件" updateAvatarDecoration: "更新头像挂件" deleteAvatarDecoration: "删除头像挂件" + unsetMfa: "解除用户的双重验证" unsetUserAvatar: "清除用户头像" unsetUserBanner: "清除用户横幅" createSystemWebhook: "新建了 SystemWebhook" diff --git a/locales/zh-TW.yml b/locales/zh-TW.yml index 4b99e32f38..59f04cd83c 100644 --- a/locales/zh-TW.yml +++ b/locales/zh-TW.yml @@ -1361,14 +1361,11 @@ information: "關於" chat: "聊天" directMessage: "直接訊息" directMessage_short: "訊息" -migrateOldSettings: "遷移舊設定資訊" -migrateOldSettings_description: "通常情況下，這會自動進行，但若因某些原因未能順利遷移，您可以手動觸發遷移處理。請注意，當前的設定資訊將會被覆寫。" compress: "壓縮" right: "右" bottom: "下" top: "上" embed: "嵌入" -settingsMigrating: "正在移轉設定。請稍候……（之後也可以到「設定 → 其他 → 舊設定資訊移轉」中手動進行移轉）" readonly: "唯讀" goToDeck: "回到多欄模式" federationJobs: "聯邦通訊作業" diff --git a/package.json b/package.json index e7af2057a0..733db4f22a 100644 --- a/package.json +++ b/package.json @@ -1,26 +1,28 @@ { "name": "misskey", - "version": "2026.6.1-alpha.1", + "version": "2026.7.0-beta.6", "codename": "nasubi", "repository": { "type": "git", "url": "https://github.com/misskey-dev/misskey.git" }, - "packageManager": "pnpm@11.8.0", + "packageManager": "pnpm@11.11.0", "workspaces": [ - "packages/misskey-js", + "packages/backend", + "packages/frontend-shared", + "packages/frontend", + "packages/frontend-builder", + "packages/frontend-embed", "packages/i18n", + "packages/icons-subsetter", + "packages/sw", + "packages/misskey-js", + "packages/misskey-js/generator", "packages/misskey-reversi", "packages/misskey-bubble-game", "packages/misskey-world", - "packages/icons-subsetter", "packages/frontend-misskey-world-engine", - "packages/frontend-shared", - "packages/frontend-builder", - "packages/sw", - "packages/backend", - "packages/frontend", - "packages/frontend-embed" + "packages-private/*" ], "private": true, "scripts": { @@ -41,11 +43,10 @@ "migrateandstart": "pnpm migrate && pnpm start", "watch": "pnpm dev", "dev": "node scripts/dev.mjs", - "lint": "pnpm --no-bail -r lint", - "cy:open": "pnpm cypress open --config-file=cypress.config.ts", - "cy:run": "pnpm cypress run", - "e2e": "pnpm start-server-and-test start:test http://localhost:61812 cy:run", - "e2e-dev-container": "ncp ./.config/cypress-devcontainer.yml ./.config/test.yml && pnpm start-server-and-test start:test http://localhost:61812 cy:run", + "check-dts": "node scripts/check-dts.mjs", + "lint": "pnpm --no-bail -r lint && pnpm check-dts", + "e2e": "pnpm start-server-and-test start:test http://localhost:61812 \"pnpm --filter frontend test:e2e\"", + "e2e-dev-container": "ncp ./.config/playwright-devcontainer.yml ./.config/test.yml && pnpm start-server-and-test start:test http://localhost:61812 \"pnpm --filter frontend test:e2e\"", "backend-unit-test": "cd packages/backend && pnpm test", "backend-unit-test-and-coverage": "cd packages/backend && pnpm test-and-coverage", "test": "pnpm -r test", @@ -58,24 +59,22 @@ "esbuild": "0.28.1", "execa": "9.6.1", "ignore-walk": "9.0.0", - "js-yaml": "4.2.0", - "tar": "7.5.16" + "js-yaml": "5.2.2", + "tar": "7.5.21" }, "devDependencies": { - "@eslint/js": "9.39.4", + "@eslint/js": "9.39.5", "@misskey-dev/eslint-plugin": "2.1.0", - "@types/js-yaml": "4.0.9", - "@types/node": "26.0.0", - "@typescript-eslint/eslint-plugin": "8.61.1", - "@typescript-eslint/parser": "8.61.1", - "@typescript/native-preview": "7.0.0-dev.20260426.1", + "@types/node": "26.1.1", + "@typescript-eslint/eslint-plugin": "8.63.0", + "@typescript-eslint/parser": "8.63.0", + "@typescript/native": "npm:typescript@7.0.2", "cross-env": "10.1.0", - "cypress": "15.17.0", - "eslint": "9.39.4", - "globals": "17.6.0", + "eslint": "9.39.5", + "globals": "17.7.0", "ncp": "2.0.0", - "pnpm": "11.8.0", + "pnpm": "11.11.0", "start-server-and-test": "3.0.11", - "typescript": "5.9.3" + "typescript": "npm:@typescript/typescript6@6.0.2" } } diff --git a/packages-private/README.md b/packages-private/README.md new file mode 100644 index 0000000000..0c14dffdf5 --- /dev/null +++ b/packages-private/README.md @@ -0,0 +1,32 @@ +# packages-private + +`packages-private` は、CIなどで使用するためのユーティリティなどを格納する場所です。この配下にあるものは**プロダクションの `/packages`**の依存となるものではありません。 + +## パッケージ一覧 + +| パッケージ | 用途 | +| --- | --- | +| [`diagnostics-shared`](./diagnostics-shared) | 計測系パッケージが共有する統計・書式整形・V8 heap snapshot解析のユーティリティ | +| [`diagnostics-backend`](./diagnostics-backend) | バックエンドのメモリ使用量をbase/headで比較し、PRコメント用のMarkdownを生成する | +| [`diagnostics-frontend`](./diagnostics-frontend) | フロントエンドをブラウザで起動した際のメトリクスと生成されたchunk情報等をbase/headで比較する | +| [`changelog-checker`](./changelog-checker) | `CHANGELOG.md` の追記内容を検証する | + +## GitHub Actionsからの呼び出し方 + +ワークフロー側にロジックを持たせず、各パッケージのnpm scriptを呼ぶだけにしている。 +CLIに渡すパスはすべて呼び出し側のカレントディレクトリ基準で解決されるため、 +ワークフローからは `$GITHUB_WORKSPACE` / `$RUNNER_TEMP` を使った絶対パスを渡すこと。 + +```yaml +- name: Generate report + working-directory: head + run: >- + pnpm --filter diagnostics-frontend run render-md + "$GITHUB_WORKSPACE/base" "$GITHUB_WORKSPACE/head" + "$REPORT_DIR/base-bundle-stats.json" "$REPORT_DIR/head-bundle-stats.json" + "$REPORT_DIR/base-browser.json" "$REPORT_DIR/head-browser.json" + "$REPORT_DIR/report.md" +``` + +base/head を比較する計測では、**head側のチェックアウトにあるハーネスで両方を計測する** +(計測コードの差分ではなくビルド成果物の差分だけが結果に出るようにするため)。 diff --git a/scripts/changelog-checker/eslint.config.js b/packages-private/changelog-checker/eslint.config.js similarity index 71% rename from scripts/changelog-checker/eslint.config.js rename to packages-private/changelog-checker/eslint.config.js index 813e96981a..a3b10c3e77 100644 --- a/scripts/changelog-checker/eslint.config.js +++ b/packages-private/changelog-checker/eslint.config.js @@ -1,10 +1,16 @@ import tsParser from '@typescript-eslint/parser'; import sharedConfig from '../../packages/shared/eslint.config.js'; +// eslint-disable-next-line import/no-default-export export default [ ...sharedConfig, { - files: ['src/**/*.ts', 'src/**/*.tsx'], + ignores: [ + '**/node_modules', + ], + }, + { + files: ['**/*.ts', '**/*.tsx'], languageOptions: { parserOptions: { parser: tsParser, diff --git a/packages-private/changelog-checker/package.json b/packages-private/changelog-checker/package.json new file mode 100644 index 0000000000..30a3af2dd9 --- /dev/null +++ b/packages-private/changelog-checker/package.json @@ -0,0 +1,24 @@ +{ + "name": "changelog-checker", + "private": true, + "type": "module", + "scripts": { + "check": "tsx src/index.ts", + "eslint": "eslint './**/*.{js,jsx,ts,tsx}'", + "lint": "pnpm typecheck && pnpm eslint", + "test": "vitest run", + "test:coverage": "vitest run --coverage", + "typecheck": "tsc --noEmit" + }, + "devDependencies": { + "@types/mdast": "4.0.4", + "@types/node": "26.1.1", + "@vitest/coverage-v8": "4.1.10", + "mdast-util-to-string": "4.0.0", + "remark": "15.0.1", + "remark-parse": "11.0.0", + "tsx": "4.23.1", + "unified": "11.0.5", + "vitest": "4.1.10" + } +} diff --git a/scripts/changelog-checker/src/checker.ts b/packages-private/changelog-checker/src/checker.ts similarity index 77% rename from scripts/changelog-checker/src/checker.ts rename to packages-private/changelog-checker/src/checker.ts index 8fd6ff5629..01c2598a74 100644 --- a/scripts/changelog-checker/src/checker.ts +++ b/packages-private/changelog-checker/src/checker.ts @@ -33,16 +33,22 @@ export function checkNewRelease(base: Release[], head: Release[]): Result { return Result.ofFailed('Invalid release count.'); } - const baseLatest = base[0]; - const headPrevious = head[releaseCountDiff]; - - if (baseLatest.releaseName !== headPrevious.releaseName) { - return Result.ofFailed('Contains unexpected releases.'); + // 追加分を除いた残り (= base に既にあったリリース) が順序ごと一致することを確認する。 + // 先頭だけ比較していると、より古いリリースが書き換えられていても素通りしてしまう + const existingReleases = head.slice(releaseCountDiff); + for (let relIdx = 0; relIdx < base.length; relIdx++) { + if (base[relIdx].releaseName !== existingReleases[relIdx].releaseName) { + return Result.ofFailed(`Contains unexpected releases. base:${base[relIdx].releaseName}, head:${existingReleases[relIdx].releaseName}`); + } } return Result.ofSuccess(); } +function isSameItems(base: string[], head: string[]) { + return base.length === head.length && base.every((item, idx) => item === head[idx]); +} + /** * topic -> developまたはtopic -> masterを想定したパターン。 * head側の最新リリース配下に書き加えられているかをチェックする。 @@ -78,9 +84,9 @@ export function checkNewTopic(base: Release[], head: Release[]): Result { return Result.ofFailed(`Category is different. base:${baseCategory.categoryName}, head:${headCategory.categoryName}`); } - if (baseCategory.items.length !== headCategory.items.length) { + if (!isSameItems(baseCategory.items, headCategory.items)) { if (headLatest.releaseName !== headItem.releaseName) { - // 最新リリース以外に追記されていた場合 + // 最新リリース以外が変更されていた場合 return Result.ofFailed(`There is an error in the update history. expected additions:${headLatest.releaseName}, actual additions:${headItem.releaseName}`); } } diff --git a/scripts/changelog-checker/src/index.ts b/packages-private/changelog-checker/src/index.ts similarity index 91% rename from scripts/changelog-checker/src/index.ts rename to packages-private/changelog-checker/src/index.ts index 0e2c5ce057..868ca3f3b6 100644 --- a/scripts/changelog-checker/src/index.ts +++ b/packages-private/changelog-checker/src/index.ts @@ -18,7 +18,7 @@ function abort(message?: string) { function main() { if (!fs.existsSync('./CHANGELOG-base.md') || !fs.existsSync('./CHANGELOG-head.md')) { - console.error('CHANGELOG-base.md or CHANGELOG-head.md is missing.'); + abort('CHANGELOG-base.md or CHANGELOG-head.md is missing.'); return; } diff --git a/scripts/changelog-checker/src/parser.ts b/packages-private/changelog-checker/src/parser.ts similarity index 90% rename from scripts/changelog-checker/src/parser.ts rename to packages-private/changelog-checker/src/parser.ts index 894d60d6af..46b74a4ed2 100644 --- a/scripts/changelog-checker/src/parser.ts +++ b/packages-private/changelog-checker/src/parser.ts @@ -51,6 +51,9 @@ export function parseChangeLog(path: string): Release[] { // リリース release = new Release(toString(it)); releases.push(release); + // 直前のリリースのカテゴリを引き継ぐと、カテゴリ見出しの無いリスト項目が + // 前のリリースに混入するのでリセットする + category = null; } else if (isHeading(it) && it.depth === 3 && release) { // リリース配下のカテゴリ category = new ReleaseCategory(toString(it)); diff --git a/scripts/changelog-checker/test/checker.test.ts b/packages-private/changelog-checker/test/checker.test.ts similarity index 56% rename from scripts/changelog-checker/test/checker.test.ts rename to packages-private/changelog-checker/test/checker.test.ts index 9ca8fa85f2..459379f8db 100644 --- a/scripts/changelog-checker/test/checker.test.ts +++ b/packages-private/changelog-checker/test/checker.test.ts @@ -3,52 +3,62 @@ * SPDX-License-Identifier: AGPL-3.0-only */ -import {expect, suite, test} from "vitest"; -import {Release, ReleaseCategory} from "../src/parser"; -import {checkNewRelease, checkNewTopic} from "../src/checker"; +import { expect, describe, test } from 'vitest'; +import { Release, ReleaseCategory } from '../src/parser.js'; +import { checkNewRelease, checkNewTopic } from '../src/checker.js'; -suite('checkNewRelease', () => { +describe('checkNewRelease', () => { test('headに新しいリリースがある1', () => { - const base = [new Release('2024.12.0')] - const head = [new Release('2024.12.1'), new Release('2024.12.0')] + const base = [new Release('2024.12.0')]; + const head = [new Release('2024.12.1'), new Release('2024.12.0')]; - const result = checkNewRelease(base, head) + const result = checkNewRelease(base, head); - expect(result.success).toBe(true) - }) + expect(result.success).toBe(true); + }); test('headに新しいリリースがある2', () => { - const base = [new Release('2024.12.0')] - const head = [new Release('2024.12.2'), new Release('2024.12.1'), new Release('2024.12.0')] + const base = [new Release('2024.12.0')]; + const head = [new Release('2024.12.2'), new Release('2024.12.1'), new Release('2024.12.0')]; - const result = checkNewRelease(base, head) - - expect(result.success).toBe(true) - }) + const result = checkNewRelease(base, head); + expect(result.success).toBe(true); + }); test('リリースの数が同じ', () => { - const base = [new Release('2024.12.0')] - const head = [new Release('2024.12.0')] + const base = [new Release('2024.12.0')]; + const head = [new Release('2024.12.0')]; - const result = checkNewRelease(base, head) + const result = checkNewRelease(base, head); - console.log(result.message) - expect(result.success).toBe(false) - }) + console.log(result.message); + expect(result.success).toBe(false); + }); test('baseにあるリリースがheadにない', () => { - const base = [new Release('2024.12.0')] - const head = [new Release('2024.12.2'), new Release('2024.12.1')] + const base = [new Release('2024.12.0')]; + const head = [new Release('2024.12.2'), new Release('2024.12.1')]; - const result = checkNewRelease(base, head) + const result = checkNewRelease(base, head); - console.log(result.message) - expect(result.success).toBe(false) - }) -}) + console.log(result.message); + expect(result.success).toBe(false); + }); -suite('checkNewTopic', () => { + // 先頭だけ比較していると、より古いリリースの書き換えを見逃す + test('追加分の直後は一致しているが、より古いリリースが書き換えられている', () => { + const base = [new Release('2024.12.1'), new Release('2024.12.0')]; + const head = [new Release('2024.12.2'), new Release('2024.12.1'), new Release('2024.11.0')]; + + const result = checkNewRelease(base, head); + + console.log(result.message); + expect(result.success).toBe(false); + }); +}); + +describe('checkNewTopic', () => { test('追記なし', () => { const base = [ new Release('2024.12.1', [ @@ -59,7 +69,7 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) + ]), ]), new Release('2024.12.0', [ new ReleaseCategory('Server', [ @@ -69,9 +79,9 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) - ]) - ] + ]), + ]), + ]; const head = [ new Release('2024.12.1', [ @@ -82,7 +92,7 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) + ]), ]), new Release('2024.12.0', [ new ReleaseCategory('Server', [ @@ -92,14 +102,14 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) - ]) - ] + ]), + ]), + ]; - const result = checkNewTopic(base, head) + const result = checkNewTopic(base, head); - expect(result.success).toBe(true) - }) + expect(result.success).toBe(true); + }); test('最新バージョンにカテゴリを追加したときはエラーにならない', () => { const base = [ @@ -117,9 +127,9 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) - ]) - ] + ]), + ]), + ]; const head = [ new Release('2024.12.1', [ @@ -130,7 +140,7 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) + ]), ]), new Release('2024.12.0', [ new ReleaseCategory('Server', [ @@ -140,14 +150,14 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) - ]) - ] + ]), + ]), + ]; - const result = checkNewTopic(base, head) + const result = checkNewTopic(base, head); - expect(result.success).toBe(true) - }) + expect(result.success).toBe(true); + }); test('最新バージョンからカテゴリを削除したときはエラーにならない', () => { const base = [ @@ -159,7 +169,7 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) + ]), ]), new Release('2024.12.0', [ new ReleaseCategory('Server', [ @@ -169,9 +179,9 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) - ]) - ] + ]), + ]), + ]; const head = [ new Release('2024.12.1', [ @@ -188,14 +198,14 @@ suite('checkNewTopic', () => { new ReleaseCategory('Client', [ 'feat3', 'feat4', - ]) - ]) - ] + ]), + ]), + ]; - const result = checkNewTopic(base, head) + const result = checkNewTopic(base, head); - expect(result.success).toBe(true) - }) + expect(result.success).toBe(true); + }); test('最新バージョンに追記したときはエラーにならない', () => { const base = [ @@ -210,8 +220,8 @@ suite('checkNewTopic', () => { 'feat1', 'feat2', ]), - ]) - ] + ]), + ]; const head = [ new Release('2024.12.1', [ @@ -226,13 +236,13 @@ suite('checkNewTopic', () => { 'feat1', 'feat2', ]), - ]) - ] + ]), + ]; - const result = checkNewTopic(base, head) + const result = checkNewTopic(base, head); - expect(result.success).toBe(true) - }) + expect(result.success).toBe(true); + }); test('最新バージョンから削除したときはエラーにならない', () => { const base = [ @@ -247,8 +257,8 @@ suite('checkNewTopic', () => { 'feat1', 'feat2', ]), - ]) - ] + ]), + ]; const head = [ new Release('2024.12.1', [ @@ -261,13 +271,13 @@ suite('checkNewTopic', () => { 'feat1', 'feat2', ]), - ]) - ] + ]), + ]; - const result = checkNewTopic(base, head) + const result = checkNewTopic(base, head); - expect(result.success).toBe(true) - }) + expect(result.success).toBe(true); + }); test('古いバージョンにカテゴリを追加したときはエラーになる', () => { const base = [ @@ -282,8 +292,8 @@ suite('checkNewTopic', () => { 'feat1', 'feat2', ]), - ]) - ] + ]), + ]; const head = [ new Release('2024.12.1', [ @@ -301,14 +311,14 @@ suite('checkNewTopic', () => { 'feat1', 'feat2', ]), - ]) - ] + ]), + ]; - const result = checkNewTopic(base, head) + const result = checkNewTopic(base, head); - console.log(result.message) - expect(result.success).toBe(false) - }) + console.log(result.message); + expect(result.success).toBe(false); + }); test('古いバージョンからカテゴリを削除したときはエラーになる', () => { const base = [ @@ -323,8 +333,8 @@ suite('checkNewTopic', () => { 'feat1', 'feat2', ]), - ]) - ] + ]), + ]; const head = [ new Release('2024.12.1', [ @@ -334,14 +344,14 @@ suite('checkNewTopic', () => { ]), ]), new Release('2024.12.0', [ - ]) - ] + ]), + ]; - const result = checkNewTopic(base, head) + const result = checkNewTopic(base, head); - console.log(result.message) - expect(result.success).toBe(false) - }) + console.log(result.message); + expect(result.success).toBe(false); + }); test('古いバージョンに追記したときはエラーになる', () => { const base = [ @@ -356,8 +366,8 @@ suite('checkNewTopic', () => { 'feat1', 'feat2', ]), - ]) - ] + ]), + ]; const head = [ new Release('2024.12.1', [ @@ -372,14 +382,14 @@ suite('checkNewTopic', () => { 'feat2', 'feat3', ]), - ]) - ] + ]), + ]; - const result = checkNewTopic(base, head) + const result = checkNewTopic(base, head); - console.log(result.message) - expect(result.success).toBe(false) - }) + console.log(result.message); + expect(result.success).toBe(false); + }); test('古いバージョンから削除したときはエラーになる', () => { const base = [ @@ -394,8 +404,8 @@ suite('checkNewTopic', () => { 'feat1', 'feat2', ]), - ]) - ] + ]), + ]; const head = [ new Release('2024.12.1', [ @@ -408,12 +418,88 @@ suite('checkNewTopic', () => { new ReleaseCategory('Server', [ 'feat1', ]), - ]) - ] + ]), + ]; - const result = checkNewTopic(base, head) + const result = checkNewTopic(base, head); - console.log(result.message) - expect(result.success).toBe(false) - }) -}) + console.log(result.message); + expect(result.success).toBe(false); + }); + + // 件数が同じでも内容が変わっていれば履歴の書き換えなのでエラーにする + test('古いバージョンの項目が書き換えられたときはエラーになる', () => { + const base = [ + new Release('2024.12.1', [ + new ReleaseCategory('Server', ['feat1']), + ]), + new Release('2024.12.0', [ + new ReleaseCategory('Server', ['feat1', 'feat2']), + ]), + ]; + + const head = [ + new Release('2024.12.1', [ + new ReleaseCategory('Server', ['feat1']), + ]), + new Release('2024.12.0', [ + new ReleaseCategory('Server', ['feat1', 'feat2-rewritten']), + ]), + ]; + + const result = checkNewTopic(base, head); + + console.log(result.message); + expect(result.success).toBe(false); + }); + + test('古いバージョンの項目が並べ替えられたときはエラーになる', () => { + const base = [ + new Release('2024.12.1', [ + new ReleaseCategory('Server', ['feat1']), + ]), + new Release('2024.12.0', [ + new ReleaseCategory('Server', ['feat1', 'feat2']), + ]), + ]; + + const head = [ + new Release('2024.12.1', [ + new ReleaseCategory('Server', ['feat1']), + ]), + new Release('2024.12.0', [ + new ReleaseCategory('Server', ['feat2', 'feat1']), + ]), + ]; + + const result = checkNewTopic(base, head); + + console.log(result.message); + expect(result.success).toBe(false); + }); + + // 最新リリースの書き換えは通常の編集なので許容する + test('最新バージョンの項目を書き換えたときはエラーにならない', () => { + const base = [ + new Release('2024.12.1', [ + new ReleaseCategory('Server', ['feat1']), + ]), + new Release('2024.12.0', [ + new ReleaseCategory('Server', ['feat1']), + ]), + ]; + + const head = [ + new Release('2024.12.1', [ + new ReleaseCategory('Server', ['feat1-rewritten']), + ]), + new Release('2024.12.0', [ + new ReleaseCategory('Server', ['feat1']), + ]), + ]; + + const result = checkNewTopic(base, head); + + expect(result.success).toBe(true); + }); +}); diff --git a/scripts/changelog-checker/tsconfig.json b/packages-private/changelog-checker/tsconfig.json similarity index 89% rename from scripts/changelog-checker/tsconfig.json rename to packages-private/changelog-checker/tsconfig.json index 32f1547eb8..a728e49d4d 100644 --- a/scripts/changelog-checker/tsconfig.json +++ b/packages-private/changelog-checker/tsconfig.json @@ -14,18 +14,17 @@ "experimentalDecorators": true, "noImplicitReturns": true, "esModuleInterop": true, - "typeRoots": [ - "./node_modules/@types" - ], + "skipLibCheck": true, + "types": ["node"], "lib": [ "esnext" ] }, "include": [ - "src/**/*" + "src/**/*", + "test/**/*" ], "exclude": [ "node_modules", - "test/**/*" ] } diff --git a/packages-private/diagnostics-backend/eslint.config.js b/packages-private/diagnostics-backend/eslint.config.js new file mode 100644 index 0000000000..11bc648243 --- /dev/null +++ b/packages-private/diagnostics-backend/eslint.config.js @@ -0,0 +1,25 @@ +import tsParser from '@typescript-eslint/parser'; +import sharedConfig from '../../packages/shared/eslint.config.js'; + +// eslint-disable-next-line import/no-default-export +export default [ + ...sharedConfig, + { + ignores: [ + '**/node_modules', + '**/__snapshots__', + 'test/fixtures', + ], + }, + { + files: ['**/*.ts', '**/*.tsx'], + languageOptions: { + parserOptions: { + parser: tsParser, + project: ['./tsconfig.json'], + sourceType: 'module', + tsconfigRootDir: import.meta.dirname, + }, + }, + }, +]; diff --git a/packages-private/diagnostics-backend/package.json b/packages-private/diagnostics-backend/package.json new file mode 100644 index 0000000000..20fd36e30a --- /dev/null +++ b/packages-private/diagnostics-backend/package.json @@ -0,0 +1,27 @@ +{ + "name": "diagnostics-backend", + "private": true, + "type": "module", + "scripts": { + "eslint": "eslint './**/*.{js,jsx,ts,tsx}'", + "inspect": "tsx src/inspect.ts", + "lint": "pnpm typecheck && pnpm eslint", + "measure": "tsx src/measure-once.ts", + "render-md": "tsx src/render-md.ts", + "test": "vitest run", + "typecheck": "tsc --noEmit" + }, + "dependencies": { + "diagnostics-shared": "workspace:*", + "execa": "9.6.1" + }, + "devDependencies": { + "@types/node": "26.1.1", + "@types/pg": "8.20.0", + "ioredis": "5.11.1", + "pg": "8.22.0", + "tsx": "4.23.1", + "typescript": "5.9.3", + "vitest": "4.1.10" + } +} diff --git a/packages-private/diagnostics-backend/src/compare.ts b/packages-private/diagnostics-backend/src/compare.ts new file mode 100644 index 0000000000..2ae1694f82 --- /dev/null +++ b/packages-private/diagnostics-backend/src/compare.ts @@ -0,0 +1,197 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { copyFile, rm, writeFile } from 'node:fs/promises'; +import { join, resolve } from 'node:path'; +import { execa } from 'execa'; +import { readBooleanEnv, readIntegerEnv, readOptionalEnv } from 'diagnostics-shared/env'; +import { median } from 'diagnostics-shared/stats'; +import { summarizeHeapSnapshotDataSamples, defaultHeapSnapshotBreakdownTopN } from 'diagnostics-shared/heap-snapshot'; +import { resetState } from './db'; +import { + clampHeapSnapshotRounds, + selectRepresentativeHeapSnapshotRound, + shouldCollectHeapSnapshot, +} from './heap-snapshot-sampling'; +import { measureBackendMemory } from './measure'; +import { memoryPhases, type MemoryReport } from './types'; + +const heapSnapshotLabels = ['base', 'head'] as const; + +type HeapSnapshotLabel = typeof heapSnapshotLabels[number]; + +export type CompareOptions = { + baseDir: string; + headDir: string; + baseOutput: string; + headOutput: string; +}; + +const HEAP_SNAPSHOT_BREAKDOWN_TOP_N = readIntegerEnv('MK_MEMORY_HEAP_SNAPSHOT_BREAKDOWN_TOP_N', defaultHeapSnapshotBreakdownTopN, 1); +// 成果物 (artifact) としてアップロードされるファイルの出力先。CIではworkspace直下を指す +const HEAP_SNAPSHOT_OUTPUT_DIR = resolve(readOptionalEnv('MK_MEMORY_HEAP_SNAPSHOT_OUTPUT_DIR') ?? process.cwd()); +const HEAP_SNAPSHOT_WORK_DIRS = { + base: join(HEAP_SNAPSHOT_OUTPUT_DIR, 'base-heap-snapshots'), + head: join(HEAP_SNAPSHOT_OUTPUT_DIR, 'head-heap-snapshots'), +}; +const HEAP_SNAPSHOT_OUTPUT_PATHS = { + base: join(HEAP_SNAPSHOT_OUTPUT_DIR, 'base-heap-snapshot.heapsnapshot'), + head: join(HEAP_SNAPSHOT_OUTPUT_DIR, 'head-heap-snapshot.heapsnapshot'), +}; + +export function summarizeSamples(samples: MemoryReport['samples']) { + const summary = {} as MemoryReport['summary']; + + for (const phase of memoryPhases) { + summary[phase] = { + memoryUsage: {}, + }; + + const metricKeys = new Set(); + for (const sample of samples) { + for (const key of Object.keys(sample.phases[phase].memoryUsage)) { + metricKeys.add(key); + } + } + + for (const key of metricKeys) { + const values = samples.map(sample => sample.phases[phase].memoryUsage[key]); + summary[phase].memoryUsage[key] = median(values); + } + + const heapSnapshot = summarizeHeapSnapshotDataSamples( + samples, + sample => sample.phases[phase].heapSnapshot, + { breakdownTopN: HEAP_SNAPSHOT_BREAKDOWN_TOP_N }, + ); + if (heapSnapshot != null) summary[phase].heapSnapshot = heapSnapshot; + } + + return summary; +} + +type GenSampleOptions = { + collectHeapSnapshot?: boolean; + heapSnapshotSavePath?: string; +}; + +async function genSample(label: string, repoDir: string, round: number, options: GenSampleOptions = {}) { + process.stderr.write(`[${label}] Resetting database and Redis\n`); + await resetState(); + + process.stderr.write(`[${label}] Running migrations\n`); + // 出力はログとして流しつつ手元にも残す (失敗時にexecaが例外メッセージへ含めてくれる) + await execa('pnpm', ['--filter', 'backend', 'migrate'], { + cwd: repoDir, + stdout: ['pipe', process.stderr], + stderr: ['pipe', process.stderr], + }); + + process.stderr.write(`[${label}] Measuring memory\n`); + return await measureBackendMemory(resolve(repoDir, 'packages/backend'), { + // warmupラウンド (round <= 0) は捨てるので、重いheap snapshotは取らない + ...(round <= 0 || options.collectHeapSnapshot === false ? { heapSnapshot: false } : {}), + heapSnapshotSavePath: options.heapSnapshotSavePath ?? null, + }); +} + +function heapSnapshotPath(label: HeapSnapshotLabel, round: number) { + return join(HEAP_SNAPSHOT_WORK_DIRS[label], `round-${round}.heapsnapshot`); +} + +async function saveRepresentativeHeapSnapshot(label: HeapSnapshotLabel, samples: MemoryReport['samples'], summary: MemoryReport['summary']) { + const round = selectRepresentativeHeapSnapshotRound( + samples, + summary.afterGc.heapSnapshot?.categories.total, + sample => sample.phases.afterGc.heapSnapshot?.categories.total, + ); + if (round == null) return; + + await copyFile(heapSnapshotPath(label, round), HEAP_SNAPSHOT_OUTPUT_PATHS[label]); + process.stderr.write(`Selected ${label} heap snapshot round ${round} for artifact\n`); + await rm(HEAP_SNAPSHOT_WORK_DIRS[label], { recursive: true, force: true }); +} + +/** + * base / head を交互に計測してJSONレポートを書き出す。 + * 交互にするのは、実行順やマシンの状態による偏りを両者に均等に載せるため。 + */ +export async function compareBackendMemory(options: CompareOptions) { + const rounds = readIntegerEnv('MK_MEMORY_COMPARE_ROUNDS', 5, 1); + const warmupRounds = readIntegerEnv('MK_MEMORY_COMPARE_WARMUP_ROUNDS', 1, 0); + const heapSnapshotsEnabled = readBooleanEnv('MK_MEMORY_HEAP_SNAPSHOT', false); + const requestedHeapSnapshotRounds = heapSnapshotsEnabled + ? readIntegerEnv('MK_MEMORY_HEAP_SNAPSHOT_ROUNDS', rounds, 1) + : 0; + const heapSnapshotRounds = clampHeapSnapshotRounds(rounds, requestedHeapSnapshotRounds); + const startedAt = new Date().toISOString(); + + for (const label of heapSnapshotLabels) { + await rm(HEAP_SNAPSHOT_WORK_DIRS[label], { recursive: true, force: true }); + await rm(HEAP_SNAPSHOT_OUTPUT_PATHS[label], { force: true }); + } + + const reports = { + base: { + dir: options.baseDir, + samples: [] as MemoryReport['samples'], + }, + head: { + dir: options.headDir, + samples: [] as MemoryReport['samples'], + }, + }; + + for (let round = 1; round <= warmupRounds; round++) { + process.stderr.write(`Starting warmup round ${round}/${warmupRounds}\n`); + for (const label of heapSnapshotLabels) { + await genSample(label, reports[label].dir, -round); + } + } + + for (let round = 1; round <= rounds; round++) { + const order = round % 2 === 1 ? ['base', 'head'] as const : ['head', 'base'] as const; + process.stderr.write(`Starting measurement round ${round}/${rounds}: ${order.join(' -> ')}\n`); + + for (const label of order) { + const collectHeapSnapshot = shouldCollectHeapSnapshot(round, rounds, heapSnapshotRounds); + const sample = await genSample(label, reports[label].dir, round, { + collectHeapSnapshot, + ...(collectHeapSnapshot ? { heapSnapshotSavePath: heapSnapshotPath(label, round) } : {}), + }); + reports[label].samples.push({ + ...sample, + round, + }); + } + } + + const summaries = { + base: summarizeSamples(reports.base.samples), + head: summarizeSamples(reports.head.samples), + }; + for (const label of heapSnapshotLabels) { + await saveRepresentativeHeapSnapshot(label, reports[label].samples, summaries[label]); + } + + for (const label of heapSnapshotLabels) { + const report: MemoryReport = { + timestamp: new Date().toISOString(), + sampleCount: reports[label].samples.length, + aggregation: 'median', + comparison: { + strategy: 'interleaved-pairs', + rounds, + warmupRounds, + heapSnapshotRounds, + startedAt, + }, + summary: summaries[label], + samples: reports[label].samples, + }; + + await writeFile(label === 'base' ? options.baseOutput : options.headOutput, `${JSON.stringify(report, null, 2)}\n`); + } +} diff --git a/packages-private/diagnostics-backend/src/db.ts b/packages-private/diagnostics-backend/src/db.ts new file mode 100644 index 0000000000..13045b6b90 --- /dev/null +++ b/packages-private/diagnostics-backend/src/db.ts @@ -0,0 +1,35 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import Redis from 'ioredis'; +import pg from 'pg'; + +/** + * 計測ラウンド間で状態を持ち越さないよう、テスト用DBを作り直しRedisを空にする。 + * 接続先はCIのテスト用サービス (.github/misskey/test.yml) と揃えてある。 + */ +export async function resetState() { + const postgres = new pg.Client({ + host: '127.0.0.1', + port: 54312, + database: 'postgres', + user: 'postgres', + }); + + await postgres.connect(); + try { + await postgres.query('DROP DATABASE IF EXISTS "test-misskey" WITH (FORCE)'); + await postgres.query('CREATE DATABASE "test-misskey"'); + } finally { + await postgres.end(); + } + + const redis = new Redis({ host: '127.0.0.1', port: 56312 }); + try { + await redis.flushall(); + } finally { + redis.disconnect(); + } +} diff --git a/packages-private/diagnostics-backend/src/heap-snapshot-sampling.ts b/packages-private/diagnostics-backend/src/heap-snapshot-sampling.ts new file mode 100644 index 0000000000..791840f9bd --- /dev/null +++ b/packages-private/diagnostics-backend/src/heap-snapshot-sampling.ts @@ -0,0 +1,37 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +export function clampHeapSnapshotRounds(totalRounds: number, requestedRounds: number) { + return Math.min(totalRounds, requestedRounds); +} + +export function shouldCollectHeapSnapshot(round: number, totalRounds: number, requestedRounds: number) { + const snapshotRounds = clampHeapSnapshotRounds(totalRounds, requestedRounds); + return round > totalRounds - snapshotRounds; +} + +/** + * 中央値に最も近いラウンドを代表として選ぶ。外れ値のスナップショットを成果物にしないため。 + */ +export function selectRepresentativeHeapSnapshotRound( + samples: T[], + medianTotal: number | null | undefined, + getTotal: (sample: T) => number | null | undefined, +) { + if (medianTotal == null || !Number.isFinite(medianTotal)) return null; + + let selected: { round: number; distance: number } | null = null; + for (const sample of samples) { + const total = getTotal(sample); + if (total == null || !Number.isFinite(total)) continue; + + const distance = Math.abs(total - medianTotal); + if (selected == null || distance < selected.distance || (distance === selected.distance && sample.round < selected.round)) { + selected = { round: sample.round, distance }; + } + } + + return selected?.round ?? null; +} diff --git a/packages-private/diagnostics-backend/src/inspect.ts b/packages-private/diagnostics-backend/src/inspect.ts new file mode 100644 index 0000000000..c98ea6b644 --- /dev/null +++ b/packages-private/diagnostics-backend/src/inspect.ts @@ -0,0 +1,27 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { resolve } from 'node:path'; +import { compareBackendMemory } from './compare'; + +const [baseDirArg, headDirArg, baseOutputArg, headOutputArg] = process.argv.slice(2); + +// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition +if (baseDirArg == null || headDirArg == null || baseOutputArg == null || headOutputArg == null) { + console.error('Usage: inspect '); + process.exit(1); +} + +try { + await compareBackendMemory({ + baseDir: resolve(baseDirArg), + headDir: resolve(headDirArg), + baseOutput: resolve(baseOutputArg), + headOutput: resolve(headOutputArg), + }); +} catch (err) { + console.error(err); + process.exit(1); +} diff --git a/packages-private/diagnostics-backend/src/measure-once.ts b/packages-private/diagnostics-backend/src/measure-once.ts new file mode 100644 index 0000000000..6efdbf09d0 --- /dev/null +++ b/packages-private/diagnostics-backend/src/measure-once.ts @@ -0,0 +1,29 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { resolve } from 'node:path'; +import { readOptionalEnv } from 'diagnostics-shared/env'; +import { measureBackendMemory } from './measure'; + +// ローカルデバッグ用: バックエンド1回分の計測結果をJSONで出力する +const [backendDirArg] = process.argv.slice(2); + +if (backendDirArg == null) { + console.error('Usage: measure '); + process.exit(1); +} + +try { + const sample = await measureBackendMemory(resolve(backendDirArg), { + heapSnapshotSavePath: readOptionalEnv('MK_MEMORY_HEAP_SNAPSHOT_SAVE_PATH'), + }); + console.log(JSON.stringify(sample, null, 2)); +} catch (err) { + console.error(JSON.stringify({ + error: err instanceof Error ? err.message : String(err), + timestamp: new Date().toISOString(), + })); + process.exit(1); +} diff --git a/packages-private/diagnostics-backend/src/measure/index.ts b/packages-private/diagnostics-backend/src/measure/index.ts new file mode 100644 index 0000000000..3acf1b0e66 --- /dev/null +++ b/packages-private/diagnostics-backend/src/measure/index.ts @@ -0,0 +1,129 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import * as fs from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { readBooleanEnv, readIntegerEnv } from 'diagnostics-shared/env'; +import { analyzeHeapSnapshot, defaultHeapSnapshotBreakdownTopN, type HeapSnapshotData } from 'diagnostics-shared/heap-snapshot'; +import { getMemoryUsage, getSmapsRollupMemoryUsage } from './proc'; +import { + forkBackendServer, + getRuntimeMemoryUsage, + requestHeapSnapshot, + shutdownBackendServer, + triggerGc, + waitForServerReady, +} from './server'; +import { measureMemoryUntilStable } from './stability'; +import type { MemorySample } from '../types'; + +export type MeasureBackendMemoryOptions = { + /** heap snapshotを取得するか (既定: MK_MEMORY_HEAP_SNAPSHOT) */ + heapSnapshot?: boolean; + /** 取得したheap snapshotの保存先。未指定なら解析後に破棄する */ + heapSnapshotSavePath?: string | null; + heapSnapshotBreakdownTopN?: number; + heapSnapshotTimeoutMs?: number; + startupTimeoutMs?: number; + ipcTimeoutMs?: number; +}; + +function resolveOptions(options: MeasureBackendMemoryOptions) { + return { + heapSnapshot: options.heapSnapshot ?? readBooleanEnv('MK_MEMORY_HEAP_SNAPSHOT', false), + heapSnapshotSavePath: options.heapSnapshotSavePath ?? null, + heapSnapshotBreakdownTopN: options.heapSnapshotBreakdownTopN ?? readIntegerEnv('MK_MEMORY_HEAP_SNAPSHOT_BREAKDOWN_TOP_N', defaultHeapSnapshotBreakdownTopN, 1), + heapSnapshotTimeoutMs: options.heapSnapshotTimeoutMs ?? readIntegerEnv('MK_MEMORY_HEAP_SNAPSHOT_TIMEOUT_MS', 120000, 1), + startupTimeoutMs: options.startupTimeoutMs ?? readIntegerEnv('MK_MEMORY_STARTUP_TIMEOUT_MS', 120000, 1), + ipcTimeoutMs: options.ipcTimeoutMs ?? readIntegerEnv('MK_MEMORY_IPC_TIMEOUT_MS', 30000, 1), + }; +} + +/** + * バックエンドを1回起動し、GC後のメモリ使用量を計測して1サンプル分の結果を返す。 + */ +export async function measureBackendMemory(backendDir: string, options: MeasureBackendMemoryOptions = {}): Promise { + const settings = resolveOptions(options); + const serverProcess = forkBackendServer(backendDir); + + // 起動完了メッセージを取りこぼさないよう、他のハンドラより先に待ち受ける + const serverReady = waitForServerReady(serverProcess, settings.startupTimeoutMs); + + serverProcess.stdout?.on('data', (data) => { + process.stderr.write(`[server stdout] ${data}`); + }); + + serverProcess.stderr?.on('data', (data) => { + process.stderr.write(`[server stderr] ${data}`); + }); + + serverProcess.on('error', (err) => { + process.stderr.write(`[server error] ${err}\n`); + }); + + // 途中で失敗しても子プロセスを残さない。残すと次のラウンドがポート衝突で落ちる + try { + const startupStartTime = Date.now(); + await serverReady; + + const startupTime = Date.now() - startupStartTime; + process.stderr.write(`Server started in ${startupTime}ms\n`); + + await triggerGc(serverProcess, settings.ipcTimeoutMs); + + const pid = serverProcess.pid!; + const stableSmapsRollup = await measureMemoryUntilStable(() => getSmapsRollupMemoryUsage(pid)); + const afterGc = { + memoryUsage: { + ...await getMemoryUsage(pid), + ...stableSmapsRollup.memoryUsage, + ...await getRuntimeMemoryUsage(serverProcess, settings.ipcTimeoutMs), + }, + stability: stableSmapsRollup.stability, + }; + process.stderr.write(`Memory ${afterGc.stability.converged ? 'stabilized' : 'did not stabilize'} after ${afterGc.stability.readingCount} readings over ${Math.round(afterGc.stability.elapsedMs)}ms\n`); + + const heapSnapshotAfterGc = await getHeapSnapshotStatistics(serverProcess, settings); + + return { + timestamp: new Date().toISOString(), + phases: { + afterGc: { + memoryUsage: afterGc.memoryUsage, + memoryStability: afterGc.stability, + heapSnapshot: heapSnapshotAfterGc, + }, + }, + }; + } finally { + await shutdownBackendServer(serverProcess); + } +} + +async function getHeapSnapshotStatistics( + serverProcess: ReturnType, + settings: ReturnType, +): Promise { + if (!settings.heapSnapshot) return null; + + const snapshotPath = join(tmpdir(), `misskey-backend-heap-${process.pid}-${serverProcess.pid}-${Date.now()}.heapsnapshot`); + const writtenPath = await requestHeapSnapshot(serverProcess, snapshotPath, settings.heapSnapshotTimeoutMs); + + try { + if (settings.heapSnapshotSavePath != null && settings.heapSnapshotSavePath !== '') { + await fs.mkdir(dirname(settings.heapSnapshotSavePath), { recursive: true }); + await fs.copyFile(writtenPath, settings.heapSnapshotSavePath); + } + + const snapshot = JSON.parse(await fs.readFile(writtenPath, 'utf-8')); + return analyzeHeapSnapshot(snapshot, { breakdownTopN: settings.heapSnapshotBreakdownTopN }); + } finally { + // 数百MBになることがあるため、解析後は必ず消す + await fs.unlink(writtenPath).catch(err => { + process.stderr.write(`Failed to delete heap snapshot ${writtenPath}: ${err.message}\n`); + }); + } +} diff --git a/packages-private/diagnostics-backend/src/measure/proc.ts b/packages-private/diagnostics-backend/src/measure/proc.ts new file mode 100644 index 0000000000..435c3ac54c --- /dev/null +++ b/packages-private/diagnostics-backend/src/measure/proc.ts @@ -0,0 +1,43 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import * as fs from 'node:fs/promises'; + +export const procStatusKeys = ['VmPeak', 'VmSize', 'VmHWM', 'VmRSS', 'VmData', 'VmStk', 'VmExe', 'VmLib', 'VmPTE', 'VmSwap'] as const; +export const smapsRollupKeys = ['Pss', 'Shared_Clean', 'Shared_Dirty', 'Private_Clean', 'Private_Dirty', 'Swap', 'SwapPss'] as const; + +/** + * `/proc` 配下の `Key: 1234 kB` 形式のファイルから指定キーを取り出す。 + * 1つでも欠けていると以降の集計が静かに壊れるため、見つからなければ例外にする。 + */ +export function parseMemoryFile(content: string, keys: KS, path: string): Record { + const result = {} as Record; + for (const _key of keys) { + const key = _key as KS[number]; + const match = content.match(new RegExp(`${key}:\\s+(\\d+)\\s+kB`)); + if (match) { + result[key] = parseInt(match[1], 10); + } else { + throw new Error(`Failed to parse ${key} from ${path}`); + } + } + return result; +} + +export function bytesToKiB(value: number) { + return Math.round(value / 1024); +} + +export async function getMemoryUsage(pid: number) { + const path = `/proc/${pid}/status`; + const status = await fs.readFile(path, 'utf-8'); + return parseMemoryFile(status, procStatusKeys, path); +} + +export async function getSmapsRollupMemoryUsage(pid: number) { + const path = `/proc/${pid}/smaps_rollup`; + const smapsRollup = await fs.readFile(path, 'utf-8'); + return parseMemoryFile(smapsRollup, smapsRollupKeys, path); +} diff --git a/packages-private/diagnostics-backend/src/measure/server.ts b/packages-private/diagnostics-backend/src/measure/server.ts new file mode 100644 index 0000000000..ee3d74ba9f --- /dev/null +++ b/packages-private/diagnostics-backend/src/measure/server.ts @@ -0,0 +1,199 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { fork, type ChildProcess } from 'node:child_process'; +import { join } from 'node:path'; +import { bytesToKiB } from './proc'; + +type GcMessage = 'gc ok' | 'gc unavailable'; +type RuntimeMemoryUsageMessage = { + type: 'memory usage'; + value: NodeJS.MemoryUsage; +}; +type HeapSnapshotMessage = { + type: 'heap snapshot'; + path?: string; +}; +type HeapSnapshotErrorMessage = { + type: 'heap snapshot error'; + message: string; +}; +type HeapSnapshotResponseMessage = HeapSnapshotMessage | HeapSnapshotErrorMessage; + +function isRecord(value: unknown): value is Record { + return value != null && typeof value === 'object'; +} + +function isGcMessage(message: unknown): message is GcMessage { + return message === 'gc ok' || message === 'gc unavailable'; +} + +function isRuntimeMemoryUsageMessage(message: unknown): message is RuntimeMemoryUsageMessage { + return isRecord(message) && message.type === 'memory usage' && isRecord(message.value); +} + +function isHeapSnapshotResponseMessage(message: unknown): message is HeapSnapshotResponseMessage { + if (!isRecord(message)) return false; + if (message.type === 'heap snapshot') return true; + return message.type === 'heap snapshot error' && typeof message.message === 'string'; +} + +export function waitForMessage(serverProcess: ChildProcess, predicate: (message: unknown) => message is T, description: string, timeout: number) { + return new Promise((resolve, reject) => { + const cleanup = () => { + globalThis.clearTimeout(timer); + serverProcess.off('message', onMessage); + serverProcess.off('exit', onExit); + serverProcess.off('error', onError); + serverProcess.off('disconnect', onDisconnect); + }; + + const timer = globalThis.setTimeout(() => { + cleanup(); + reject(new Error(`Timed out waiting for ${description}`)); + }, timeout); + + const onMessage = (message: unknown) => { + if (!predicate(message)) return; + cleanup(); + resolve(message); + }; + + // 子が死んだ場合、待ち続けてもメッセージは来ない。 + // タイムアウトまで待って誤解を招くエラーを出すより、理由を添えて即座に失敗させる + const onExit = (code: number | null, signal: string | null) => { + cleanup(); + reject(new Error(`Server exited (code=${code}, signal=${signal}) while waiting for ${description}`)); + }; + + const onError = (err: Error) => { + cleanup(); + reject(new Error(`Server errored while waiting for ${description}: ${err.message}`)); + }; + + const onDisconnect = () => { + cleanup(); + reject(new Error(`Server IPC channel closed while waiting for ${description}`)); + }; + + serverProcess.on('message', onMessage); + serverProcess.once('exit', onExit); + serverProcess.once('error', onError); + serverProcess.once('disconnect', onDisconnect); + }); +} + +/** + * ビルド済みバックエンドを子プロセスとして起動する。 + * execArgv は親から引き継がず `--expose-gc` のみを渡す: 親は tsx 経由で動くため、 + * 引き継ぐと計測対象プロセスにTSローダーが載ってしまいメモリ量が歪む。 + */ +export function forkBackendServer(backendDir: string) { + return fork(join(backendDir, 'built/entry.js'), [], { + cwd: backendDir, + env: { + ...process.env, + NODE_ENV: 'production', + MK_DISABLE_CLUSTERING: '1', + MK_ONLY_SERVER: '1', + MK_NO_DAEMONS: '1', + }, + stdio: ['pipe', 'pipe', 'pipe', 'ipc'], + execArgv: ['--expose-gc'], + }); +} + +export function waitForServerReady(serverProcess: ChildProcess, timeout: number) { + return waitForMessage( + serverProcess, + (message): message is 'ok' => message === 'ok', + 'server startup', + timeout, + ); +} + +export async function triggerGc(serverProcess: ChildProcess, timeout: number) { + // 送信前にlistenerを張らないと、応答を取りこぼす可能性がある + const ok = waitForMessage(serverProcess, isGcMessage, 'GC completion', timeout); + + serverProcess.send('gc'); + + const message = await ok; + if (message === 'gc unavailable') { + throw new Error('GC is unavailable. Start the process with --expose-gc to enable this feature.'); + } +} + +export async function getRuntimeMemoryUsage(serverProcess: ChildProcess, timeout: number) { + const response = waitForMessage( + serverProcess, + isRuntimeMemoryUsageMessage, + 'memory usage', + timeout, + ); + + serverProcess.send('memory usage'); + + const message = await response; + const memoryUsage = message.value; + + // /proc 由来の値と単位を揃える + return { + HeapTotal: bytesToKiB(memoryUsage.heapTotal), + HeapUsed: bytesToKiB(memoryUsage.heapUsed), + External: bytesToKiB(memoryUsage.external), + ArrayBuffers: bytesToKiB(memoryUsage.arrayBuffers), + }; +} + +/** + * heap snapshotの書き出しを依頼し、実際に書かれたパスを返す。 + */ +export async function requestHeapSnapshot(serverProcess: ChildProcess, snapshotPath: string, timeout: number) { + const response = waitForMessage( + serverProcess, + isHeapSnapshotResponseMessage, + 'heap snapshot', + timeout, + ); + + serverProcess.send({ + type: 'heap snapshot', + path: snapshotPath, + }); + + const message = await response; + if (message.type === 'heap snapshot error') { + throw new Error(`Failed to write heap snapshot: ${message.message}`); + } + + return typeof message.path === 'string' ? message.path : snapshotPath; +} + +/** + * SIGTERMで終了を促し、一定時間で落ちなければSIGKILLする。 + */ +export async function shutdownBackendServer(serverProcess: ChildProcess) { + // 既に終了しているなら 'exit' はもう発火しないので、待つと無駄に10秒止まる + if (serverProcess.exitCode != null || serverProcess.signalCode != null) return; + + await new Promise(resolve => { + let forceTimer: NodeJS.Timeout | undefined; + const termTimer = globalThis.setTimeout(() => { + serverProcess.kill('SIGKILL'); + // SIGKILLは無視できないので通常はここで 'exit' が来る。 + // D状態などで落ちない場合に計測全体を止めないよう、待ち時間には上限を設ける + forceTimer = globalThis.setTimeout(resolve, 5000); + }, 10000); + + serverProcess.once('exit', () => { + globalThis.clearTimeout(termTimer); + if (forceTimer != null) globalThis.clearTimeout(forceTimer); + resolve(); + }); + + serverProcess.kill('SIGTERM'); + }); +} diff --git a/packages-private/diagnostics-backend/src/measure/stability.ts b/packages-private/diagnostics-backend/src/measure/stability.ts new file mode 100644 index 0000000000..0c390088c2 --- /dev/null +++ b/packages-private/diagnostics-backend/src/measure/stability.ts @@ -0,0 +1,79 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { setTimeout } from 'node:timers/promises'; + +type MemoryStabilityTimer = { + now: () => number; + wait: (durationMs: number) => Promise; +}; + +const intervalMs = 2000; +const maxWaitMs = 10000; +const windowSize = 3; +const slopeThresholdKiBPerSecond = 256; +const stabilityMetrics = ['Pss', 'Private_Dirty'] as const; + +const defaultTimer: MemoryStabilityTimer = { + now: () => performance.now(), + wait: durationMs => setTimeout(durationMs), +}; + +function getMaxAbsoluteSlopes>(readings: { elapsedMs: number; memoryUsage: T }[]) { + const result = {} as Record; + + for (const metric of stabilityMetrics) { + let maxAbsoluteSlope = 0; + for (let i = 1; i < readings.length; i++) { + const previous = readings[i - 1]; + const current = readings[i]; + const durationSeconds = (current.elapsedMs - previous.elapsedMs) / 1000; + maxAbsoluteSlope = Math.max(maxAbsoluteSlope, Math.abs(current.memoryUsage[metric] - previous.memoryUsage[metric]) / durationSeconds); + } + result[metric] = maxAbsoluteSlope; + } + + return result; +} + +/** + * メモリ使用量が落ち着くまで繰り返し読み取る。 + * 起動直後は遅延初期化でじわじわ増え続けるため、直近 `windowSize` 件の傾きが十分小さくなるまで待つ。 + */ +export async function measureMemoryUntilStable>( + readMemoryUsage: () => Promise, + timer: MemoryStabilityTimer = defaultTimer, +) { + const startedAt = timer.now(); + const readings: { elapsedMs: number; memoryUsage: T }[] = []; + let maxAbsoluteSlopesKiBPerSecond: Record | null = null; + + while (true) { + const memoryUsage = await readMemoryUsage(); + const elapsedMs = timer.now() - startedAt; + readings.push({ elapsedMs, memoryUsage }); + + let converged = false; + if (readings.length >= windowSize) { + const latestSlopes = getMaxAbsoluteSlopes(readings.slice(-windowSize)); + maxAbsoluteSlopesKiBPerSecond = latestSlopes; + converged = stabilityMetrics.every(metric => latestSlopes[metric] <= slopeThresholdKiBPerSecond); + } + + if (converged || elapsedMs >= maxWaitMs) { + return { + memoryUsage, + stability: { + converged, + readingCount: readings.length, + elapsedMs, + maxAbsoluteSlopesKiBPerSecond, + }, + }; + } + + await timer.wait(Math.min(intervalMs, maxWaitMs - elapsedMs)); + } +} diff --git a/packages-private/diagnostics-backend/src/render-md.ts b/packages-private/diagnostics-backend/src/render-md.ts new file mode 100644 index 0000000000..991ac0fe53 --- /dev/null +++ b/packages-private/diagnostics-backend/src/render-md.ts @@ -0,0 +1,30 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { readFile, writeFile } from 'node:fs/promises'; +import { resolve } from 'node:path'; +import { readRequiredEnv } from 'diagnostics-shared/env'; +import { renderMemoryReportMarkdown } from './report/markdown'; +import type { MemoryReport } from './types'; + +async function main() { + const [baseFileArg, headFileArg, outputFileArg] = process.argv.slice(2); + if (baseFileArg == null || headFileArg == null || outputFileArg == null) { + throw new Error('Usage: render-md '); + } + + const base = JSON.parse(await readFile(resolve(baseFileArg), 'utf8')) as MemoryReport; + const head = JSON.parse(await readFile(resolve(headFileArg), 'utf8')) as MemoryReport; + + await writeFile(resolve(outputFileArg), renderMemoryReportMarkdown(base, head, { + baseHeapSnapshotUrl: readRequiredEnv('MK_MEMORY_HEAP_SNAPSHOT_ARTIFACT_URL_BASE'), + headHeapSnapshotUrl: readRequiredEnv('MK_MEMORY_HEAP_SNAPSHOT_ARTIFACT_URL_HEAD'), + })); +} + +await main().catch(err => { + console.error(err); + process.exit(1); +}); diff --git a/packages-private/diagnostics-backend/src/report/markdown.ts b/packages-private/diagnostics-backend/src/report/markdown.ts new file mode 100644 index 0000000000..cf4acefae5 --- /dev/null +++ b/packages-private/diagnostics-backend/src/report/markdown.ts @@ -0,0 +1,174 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { formatKiBAsMb } from 'diagnostics-shared/format'; +import { renderHeapSnapshotTable, type HeapSnapshotReport } from 'diagnostics-shared/heap-snapshot'; +import { renderMetricComparisonTable } from 'diagnostics-shared/metric-table'; +import { + independentDeltaSummary, + isOutsideObservedNoise, + type IndependentDeltaSummary, +} from 'diagnostics-shared/stats'; +import type { MemoryPhase, MemoryReport } from '../types'; + +export type RenderMemoryReportOptions = { + baseHeapSnapshotUrl: string; + headHeapSnapshotUrl: string; +}; + +const memoryReportPhases = [ + { + key: 'afterGc', + title: 'After GC', + }, +] as const satisfies readonly { key: MemoryPhase; title: string }[]; + +const memoryMetrics = [ + 'HeapUsed', + 'Pss', + 'USS', + 'External', +] as const; + +type MemoryMetric = typeof memoryMetrics[number]; + +const memoryColorThresholdKiB = 100; + +function formatMemoryMetricName(metric: MemoryMetric) { + return metric === 'Pss' ? 'PSS' : metric; +} + +function getMemoryValueFromSample(sample: MemoryReport['samples'][number], phase: MemoryPhase, metric: MemoryMetric) { + const memoryUsage = sample.phases[phase].memoryUsage; + // USSは直接取れないのでPrivateの合算で近似する + if (metric !== 'USS') return memoryUsage[metric]; + return memoryUsage.Private_Clean + memoryUsage.Private_Dirty; +} + +function summarizeMemoryMetric(base: MemoryReport, head: MemoryReport, phase: MemoryPhase, metric: MemoryMetric) { + return independentDeltaSummary( + base.samples, + head.samples, + sample => getMemoryValueFromSample(sample, phase, metric), + ); +} + +function getDeltaPercent(summary: IndependentDeltaSummary) { + if (summary.baseMedian === 0) return null; + return summary.delta * 100 / summary.baseMedian; +} + +function renderMainTableForPhase(base: MemoryReport, head: MemoryReport, phase: MemoryPhase) { + return renderMetricComparisonTable( + base.samples, + head.samples, + memoryMetrics.map(metric => ({ + label: `**${formatMemoryMetricName(metric)}**`, + getValue: sample => getMemoryValueFromSample(sample, phase, metric), + formatValue: formatKiBAsMb, + absoluteThreshold: memoryColorThresholdKiB, + })), + { onlySignificantChanges: true }, + ); +} + +function toHeapSnapshotReport(report: MemoryReport): HeapSnapshotReport | null { + const summary = report.summary.afterGc.heapSnapshot; + if (summary == null) return null; + + return { + summary, + samples: report.samples.flatMap(sample => { + const data = sample.phases.afterGc.heapSnapshot; + return data == null ? [] : [{ round: sample.round, data }]; + }), + }; +} + +function renderHeapSnapshotSection(base: MemoryReport, head: MemoryReport) { + const baseHeapSnapshotReport = toHeapSnapshotReport(base); + const headHeapSnapshotReport = toHeapSnapshotReport(head); + if (baseHeapSnapshotReport == null || headHeapSnapshotReport == null) return null; + + const table = renderHeapSnapshotTable(baseHeapSnapshotReport, headHeapSnapshotReport); + + const lines = [ + '### V8 Heap Snapshot Statistics', + '', + table, + '', + ]; + + // Sankeyはノイズが多く読み取りづらかったため現在は無効。復活させる余地を残して残置する + for (const graph of [ + //renderHeapSnapshotSankey(baseHeapSnapshotReport, 'Base'), + //renderHeapSnapshotSankey(headHeapSnapshotReport, 'Head'), + ] as (string | null)[]) { + if (graph == null) continue; + lines.push(graph); + lines.push(''); + } + + return lines.join('\n'); +} + +function countNonConvergedMemorySamples(base: MemoryReport, head: MemoryReport) { + return [base, head] + .flatMap(report => report.samples) + .filter(sample => memoryReportPhases.some(phase => !sample.phases[phase.key].memoryStability.converged)) + .length; +} + +export function renderMemoryReportMarkdown(base: MemoryReport, head: MemoryReport, options: RenderMemoryReportOptions) { + const lines = [ + '## ⚙️ Backend Diagnostics Report', + '', + ]; + + //const summary = measurementSummary(base, head); + //if (summary != null) { + // lines.push(summary); + // lines.push(''); + //} + + for (const phase of memoryReportPhases) { + lines.push(`### Memory: ${phase.title}`); + lines.push(renderMainTableForPhase(base, head, phase.key)); + lines.push(''); + } + + lines.push(''); + + const nonConvergedSamples = countNonConvergedMemorySamples(base, head); + if (nonConvergedSamples > 0) { + const noun = nonConvergedSamples === 1 ? 'sample' : 'samples'; + lines.push(`⚠️ **Measurement warning**: ${nonConvergedSamples} memory ${noun} did not converge.`); + lines.push(''); + } + + const heapSnapshotSection = renderHeapSnapshotSection(base, head); + if (heapSnapshotSection != null) { + lines.push(heapSnapshotSection); + lines.push(''); + } + + lines.push(`Download representative heap snapshot: [base](${options.baseHeapSnapshotUrl}) / [head](${options.headHeapSnapshotUrl})`); + lines.push(''); + + const warningMetric = 'Pss'; + const warningSummary = summarizeMemoryMetric(base, head, 'afterGc', warningMetric); + const warningDiffPercent = getDeltaPercent(warningSummary); + if ( + warningSummary.delta > 0 && + warningDiffPercent != null && + warningDiffPercent > 5 && + isOutsideObservedNoise(warningSummary) + ) { + lines.push(`⚠️ **Warning**: Memory usage (${formatMemoryMetricName(warningMetric)}) has increased by more than 5% and exceeds the observed sample noise. Please verify this is not an unintended change.`); + lines.push(''); + } + + return `${lines.join('\n')}\n`; +} diff --git a/packages-private/diagnostics-backend/src/types.ts b/packages-private/diagnostics-backend/src/types.ts new file mode 100644 index 0000000000..e095572f29 --- /dev/null +++ b/packages-private/diagnostics-backend/src/types.ts @@ -0,0 +1,50 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { HeapSnapshotData } from 'diagnostics-shared/heap-snapshot'; + +/** 計測フェーズ。将来的に増やせるようリスト化してある */ +export const memoryPhases = ['afterGc'] as const; + +export type MemoryPhase = typeof memoryPhases[number]; + +export type MemoryStability = { + converged: boolean; + readingCount: number; + elapsedMs: number; + maxAbsoluteSlopesKiBPerSecond: Record | null; +}; + +/** バックエンドを1回起動して得られる計測結果 */ +export type MemorySample = { + timestamp: string; + phases: Record; + memoryStability: MemoryStability; + heapSnapshot: HeapSnapshotData | null; + }>; +}; + +/** base / head それぞれについて出力されるJSONレポート */ +export type MemoryReport = { + timestamp: string; + sampleCount: number; + aggregation: string; + comparison?: { + strategy: string; + rounds: number; + warmupRounds: number; + heapSnapshotRounds?: number; + startedAt: string; + }; + summary: Record; + heapSnapshot?: HeapSnapshotData; + }>; + samples: (MemorySample & { + round: number; + })[]; +}; diff --git a/packages-private/diagnostics-backend/test/__snapshots__/render-md.md b/packages-private/diagnostics-backend/test/__snapshots__/render-md.md new file mode 100644 index 0000000000..c7fdf641a7 --- /dev/null +++ b/packages-private/diagnostics-backend/test/__snapshots__/render-md.md @@ -0,0 +1,31 @@ +## ⚙️ Backend Diagnostics Report + +### Memory: After GC +| Metric | @ Base | @ Head | Δ | MAD | +| --- | ---: | ---: | ---: | ---: | +| **HeapUsed** | 152 MB
± 1 MB | 168 MB
± 1 MB | $\color{orange}{\text{+16~MB}}$
$\color{orange}{\text{+10.5\\%}}$ | 1.4 MB | +| **PSS** | 202 MB
± 1 MB | 218 MB
± 1 MB | $\color{orange}{\text{+16~MB}}$
$\color{orange}{\text{+7.9\\%}}$ | 1.4 MB | +| **USS** | 184 MB
± 1 MB | 200 MB
± 1 MB | $\color{orange}{\text{+16~MB}}$
$\color{orange}{\text{+8.7\\%}}$ | 1.4 MB | + +Only metrics showing significant changes are displayed. + + +### V8 Heap Snapshot Statistics + +| Metric | @ Base | @ Head | Δ | MAD | +| --- | ---: | ---: | ---: | ---: | +| $\color{gray}{\rule{8pt}{8pt}}$ **Total** | 40 MB
± 200 KB | 44 MB
± 200 KB | $\color{orange}{\text{+3.2 MB}}$
$\color{orange}{\text{+7.9\\%}}$ | 283 KB | +| | | | | | +| $\color{orange}{\rule{8pt}{8pt}}$ **Code** | 4.7 MB | 5.1 MB | $\color{orange}{\text{+376 KB}}$ | 33 KB | +| $\color{red}{\rule{8pt}{8pt}}$ **Strings** | 4.4 MB | 4.8 MB | $\color{orange}{\text{+352 KB}}$ | 31 KB | +| $\color{cyan}{\rule{8pt}{8pt}}$ **JS arrays** | 4.1 MB | 4.5 MB | $\color{orange}{\text{+328 KB}}$ | 29 KB | +| $\color{green}{\rule{8pt}{8pt}}$ **Typed arrays** | 3.8 MB | 4.1 MB | $\color{orange}{\text{+304 KB}}$ | 27 KB | +| $\color{yellow}{\rule{8pt}{8pt}}$ **System objects** | 3.5 MB | 3.8 MB | $\color{orange}{\text{+280 KB}}$ | 25 KB | +| $\color{violet}{\rule{8pt}{8pt}}$ **Other JS objs** | 3.2 MB | 3.5 MB | $\color{orange}{\text{+256 KB}}$ | 23 KB | +| $\color{pink}{\rule{8pt}{8pt}}$ **Other non-JS objs** | 2.9 MB | 3.2 MB | $\color{orange}{\text{+232 KB}}$ | 21 KB | + + +Download representative heap snapshot: [base](https://example.invalid/base) / [head](https://example.invalid/head) + +⚠️ **Warning**: Memory usage (PSS) has increased by more than 5% and exceeds the observed sample noise. Please verify this is not an unintended change. + diff --git a/packages-private/diagnostics-backend/test/fixtures/base.json b/packages-private/diagnostics-backend/test/fixtures/base.json new file mode 100644 index 0000000000..d9022b9b3c --- /dev/null +++ b/packages-private/diagnostics-backend/test/fixtures/base.json @@ -0,0 +1,200 @@ +{ + "timestamp": "2026-07-18T07:57:17.653Z", + "sampleCount": 3, + "aggregation": "median", + "comparison": { + "strategy": "interleaved-pairs", + "rounds": 3, + "warmupRounds": 1, + "startedAt": "2026-07-18T07:57:17.653Z" + }, + "summary": { + "afterGc": { + "memoryUsage": { + "VmRSS": 202000, + "Pss": 202000, + "Private_Clean": 2020, + "Private_Dirty": 182000, + "HeapUsed": 152000, + "HeapTotal": 170000, + "External": 8200, + "ArrayBuffers": 2000 + }, + "heapSnapshot": { + "categories": { + "total": 40400000, + "code": 4747000, + "strings": 4444000, + "jsArrays": 4141000, + "typedArrays": 3838000, + "systemObjects": 3535000, + "otherJsObjects": 3232000, + "otherNonJsObjects": 2929000 + }, + "nodeCounts": { + "total": 404000, + "code": 47470, + "strings": 44440, + "jsArrays": 41410, + "typedArrays": 38380, + "systemObjects": 35350, + "otherJsObjects": 32320, + "otherNonJsObjects": 29290 + }, + "breakdowns": {} + } + } + }, + "samples": [ + { + "round": 1, + "timestamp": "2026-07-18T07:57:17.652Z", + "phases": { + "afterGc": { + "memoryUsage": { + "VmRSS": 201000, + "Pss": 201000, + "Private_Clean": 2010, + "Private_Dirty": 181000, + "HeapUsed": 151000, + "HeapTotal": 170000, + "External": 8100, + "ArrayBuffers": 2000 + }, + "memoryStability": { + "converged": true, + "readingCount": 3, + "elapsedMs": 4000, + "maxAbsoluteSlopesKiBPerSecond": { + "Pss": 10, + "Private_Dirty": 5 + } + }, + "heapSnapshot": { + "categories": { + "total": 40200000, + "code": 4723500, + "strings": 4422000, + "jsArrays": 4120500, + "typedArrays": 3819000, + "systemObjects": 3517500, + "otherJsObjects": 3216000, + "otherNonJsObjects": 2914500 + }, + "nodeCounts": { + "total": 402000, + "code": 47235, + "strings": 44220, + "jsArrays": 41205, + "typedArrays": 38190, + "systemObjects": 35175, + "otherJsObjects": 32160, + "otherNonJsObjects": 29145 + }, + "breakdowns": {} + } + } + } + }, + { + "round": 2, + "timestamp": "2026-07-18T07:57:17.653Z", + "phases": { + "afterGc": { + "memoryUsage": { + "VmRSS": 202000, + "Pss": 202000, + "Private_Clean": 2020, + "Private_Dirty": 182000, + "HeapUsed": 152000, + "HeapTotal": 170000, + "External": 8200, + "ArrayBuffers": 2000 + }, + "memoryStability": { + "converged": true, + "readingCount": 3, + "elapsedMs": 4000, + "maxAbsoluteSlopesKiBPerSecond": { + "Pss": 10, + "Private_Dirty": 5 + } + }, + "heapSnapshot": { + "categories": { + "total": 40400000, + "code": 4747000, + "strings": 4444000, + "jsArrays": 4141000, + "typedArrays": 3838000, + "systemObjects": 3535000, + "otherJsObjects": 3232000, + "otherNonJsObjects": 2929000 + }, + "nodeCounts": { + "total": 404000, + "code": 47470, + "strings": 44440, + "jsArrays": 41410, + "typedArrays": 38380, + "systemObjects": 35350, + "otherJsObjects": 32320, + "otherNonJsObjects": 29290 + }, + "breakdowns": {} + } + } + } + }, + { + "round": 3, + "timestamp": "2026-07-18T07:57:17.653Z", + "phases": { + "afterGc": { + "memoryUsage": { + "VmRSS": 203000, + "Pss": 203000, + "Private_Clean": 2030, + "Private_Dirty": 183000, + "HeapUsed": 153000, + "HeapTotal": 170000, + "External": 8300, + "ArrayBuffers": 2000 + }, + "memoryStability": { + "converged": true, + "readingCount": 3, + "elapsedMs": 4000, + "maxAbsoluteSlopesKiBPerSecond": { + "Pss": 10, + "Private_Dirty": 5 + } + }, + "heapSnapshot": { + "categories": { + "total": 40600000, + "code": 4770500, + "strings": 4466000, + "jsArrays": 4161500, + "typedArrays": 3857000, + "systemObjects": 3552500, + "otherJsObjects": 3248000, + "otherNonJsObjects": 2943500 + }, + "nodeCounts": { + "total": 406000, + "code": 47705, + "strings": 44660, + "jsArrays": 41615, + "typedArrays": 38570, + "systemObjects": 35525, + "otherJsObjects": 32480, + "otherNonJsObjects": 29435 + }, + "breakdowns": {} + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages-private/diagnostics-backend/test/fixtures/head.json b/packages-private/diagnostics-backend/test/fixtures/head.json new file mode 100644 index 0000000000..3f9040ec93 --- /dev/null +++ b/packages-private/diagnostics-backend/test/fixtures/head.json @@ -0,0 +1,200 @@ +{ + "timestamp": "2026-07-18T07:57:17.654Z", + "sampleCount": 3, + "aggregation": "median", + "comparison": { + "strategy": "interleaved-pairs", + "rounds": 3, + "warmupRounds": 1, + "startedAt": "2026-07-18T07:57:17.654Z" + }, + "summary": { + "afterGc": { + "memoryUsage": { + "VmRSS": 218000, + "Pss": 218000, + "Private_Clean": 2020, + "Private_Dirty": 198000, + "HeapUsed": 168000, + "HeapTotal": 186000, + "External": 8200, + "ArrayBuffers": 2000 + }, + "heapSnapshot": { + "categories": { + "total": 43600000, + "code": 5123000, + "strings": 4796000, + "jsArrays": 4469000, + "typedArrays": 4142000, + "systemObjects": 3815000, + "otherJsObjects": 3488000, + "otherNonJsObjects": 3161000 + }, + "nodeCounts": { + "total": 436000, + "code": 51230, + "strings": 47960, + "jsArrays": 44690, + "typedArrays": 41420, + "systemObjects": 38150, + "otherJsObjects": 34880, + "otherNonJsObjects": 31610 + }, + "breakdowns": {} + } + } + }, + "samples": [ + { + "round": 1, + "timestamp": "2026-07-18T07:57:17.654Z", + "phases": { + "afterGc": { + "memoryUsage": { + "VmRSS": 217000, + "Pss": 217000, + "Private_Clean": 2010, + "Private_Dirty": 197000, + "HeapUsed": 167000, + "HeapTotal": 186000, + "External": 8100, + "ArrayBuffers": 2000 + }, + "memoryStability": { + "converged": true, + "readingCount": 3, + "elapsedMs": 4000, + "maxAbsoluteSlopesKiBPerSecond": { + "Pss": 10, + "Private_Dirty": 5 + } + }, + "heapSnapshot": { + "categories": { + "total": 43400000, + "code": 5099500, + "strings": 4774000, + "jsArrays": 4448500, + "typedArrays": 4123000, + "systemObjects": 3797500, + "otherJsObjects": 3472000, + "otherNonJsObjects": 3146500 + }, + "nodeCounts": { + "total": 434000, + "code": 50995, + "strings": 47740, + "jsArrays": 44485, + "typedArrays": 41230, + "systemObjects": 37975, + "otherJsObjects": 34720, + "otherNonJsObjects": 31465 + }, + "breakdowns": {} + } + } + } + }, + { + "round": 2, + "timestamp": "2026-07-18T07:57:17.654Z", + "phases": { + "afterGc": { + "memoryUsage": { + "VmRSS": 218000, + "Pss": 218000, + "Private_Clean": 2020, + "Private_Dirty": 198000, + "HeapUsed": 168000, + "HeapTotal": 186000, + "External": 8200, + "ArrayBuffers": 2000 + }, + "memoryStability": { + "converged": true, + "readingCount": 3, + "elapsedMs": 4000, + "maxAbsoluteSlopesKiBPerSecond": { + "Pss": 10, + "Private_Dirty": 5 + } + }, + "heapSnapshot": { + "categories": { + "total": 43600000, + "code": 5123000, + "strings": 4796000, + "jsArrays": 4469000, + "typedArrays": 4142000, + "systemObjects": 3815000, + "otherJsObjects": 3488000, + "otherNonJsObjects": 3161000 + }, + "nodeCounts": { + "total": 436000, + "code": 51230, + "strings": 47960, + "jsArrays": 44690, + "typedArrays": 41420, + "systemObjects": 38150, + "otherJsObjects": 34880, + "otherNonJsObjects": 31610 + }, + "breakdowns": {} + } + } + } + }, + { + "round": 3, + "timestamp": "2026-07-18T07:57:17.654Z", + "phases": { + "afterGc": { + "memoryUsage": { + "VmRSS": 219000, + "Pss": 219000, + "Private_Clean": 2030, + "Private_Dirty": 199000, + "HeapUsed": 169000, + "HeapTotal": 186000, + "External": 8300, + "ArrayBuffers": 2000 + }, + "memoryStability": { + "converged": true, + "readingCount": 3, + "elapsedMs": 4000, + "maxAbsoluteSlopesKiBPerSecond": { + "Pss": 10, + "Private_Dirty": 5 + } + }, + "heapSnapshot": { + "categories": { + "total": 43800000, + "code": 5146500, + "strings": 4818000, + "jsArrays": 4489500, + "typedArrays": 4161000, + "systemObjects": 3832500, + "otherJsObjects": 3504000, + "otherNonJsObjects": 3175500 + }, + "nodeCounts": { + "total": 438000, + "code": 51465, + "strings": 48180, + "jsArrays": 44895, + "typedArrays": 41610, + "systemObjects": 38325, + "otherJsObjects": 35040, + "otherNonJsObjects": 31755 + }, + "breakdowns": {} + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages-private/diagnostics-backend/test/heap-snapshot-sampling.test.ts b/packages-private/diagnostics-backend/test/heap-snapshot-sampling.test.ts new file mode 100644 index 0000000000..8409fe166b --- /dev/null +++ b/packages-private/diagnostics-backend/test/heap-snapshot-sampling.test.ts @@ -0,0 +1,42 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test } from 'vitest'; +import { + clampHeapSnapshotRounds, + selectRepresentativeHeapSnapshotRound, + shouldCollectHeapSnapshot, +} from '../src/heap-snapshot-sampling'; + +describe('heap snapshot sampling', () => { + test('collects only the final requested rounds', () => { + const selected = Array.from({ length: 15 }, (_, index) => index + 1) + .filter(round => shouldCollectHeapSnapshot(round, 15, 3)); + expect(selected).toStrictEqual([13, 14, 15]); + }); + + test('clamps the requested count to the measured round count', () => { + expect(clampHeapSnapshotRounds(5, 20)).toBe(5); + expect(Array.from({ length: 5 }, (_, index) => index + 1) + .filter(round => shouldCollectHeapSnapshot(round, 5, 20))) + .toStrictEqual([1, 2, 3, 4, 5]); + }); + + test('collects no snapshots when the effective count is zero', () => { + expect(Array.from({ length: 5 }, (_, index) => index + 1) + .filter(round => shouldCollectHeapSnapshot(round, 5, 0))) + .toStrictEqual([]); + }); + + test('selects the snapshot nearest the median and ignores missing rounds', () => { + const samples = [ + { round: 12, total: null }, + { round: 13, total: 100 }, + { round: 14, total: 110 }, + { round: 15, total: 130 }, + ]; + expect(selectRepresentativeHeapSnapshotRound(samples, 110, sample => sample.total)).toBe(14); + }); +}); diff --git a/packages-private/diagnostics-backend/test/render-md.test.ts b/packages-private/diagnostics-backend/test/render-md.test.ts new file mode 100644 index 0000000000..8b8305be60 --- /dev/null +++ b/packages-private/diagnostics-backend/test/render-md.test.ts @@ -0,0 +1,130 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { readFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { expect, test } from 'vitest'; +import { pairedDeltaSummary } from 'diagnostics-shared/stats'; +import { renderMemoryReportMarkdown } from '../src/report/markdown'; +import type { MemoryReport } from '../src/types'; + +const fixturesDir = join(import.meta.dirname, 'fixtures'); + +async function loadFixture(name: string) { + return JSON.parse(await readFile(join(fixturesDir, `${name}.json`), 'utf8')) as MemoryReport; +} + +function replacePssSamples(report: MemoryReport, values: number[]) { + const templates = structuredClone(report.samples); + report.samples = values.map((Pss, index) => { + const sample = structuredClone(templates[index % templates.length]); + sample.round = index + 1; + sample.phases.afterGc.memoryUsage.Pss = Pss; + const privateClean = sample.phases.afterGc.memoryUsage.Private_Clean; + sample.phases.afterGc.memoryUsage.Private_Dirty = Pss - privateClean; + return sample; + }); + report.sampleCount = report.samples.length; + return report; +} + +function findMetricRow(markdown: string, metric: string) { + const row = markdown.split('\n').find(line => line.startsWith(`| **${metric}**`)); + if (row === undefined) throw new Error(`expected memory report to contain a ${metric} row`); + return row; +} + +/** + * 出力をゴールデンファイルで固定する。 + * 意図的に変更したときは `vitest -u` で更新し、__snapshots__ の差分もレビューすること。 + */ +test('renders the backend memory report', async () => { + const markdown = renderMemoryReportMarkdown(await loadFixture('base'), await loadFixture('head'), { + baseHeapSnapshotUrl: 'https://example.invalid/base', + headHeapSnapshotUrl: 'https://example.invalid/head', + }); + + await expect(markdown).toMatchFileSnapshot('./__snapshots__/render-md.md'); +}); + +test('throws when filtering leaves fewer than two heap snapshots per side', async () => { + const base = await loadFixture('base'); + const head = await loadFixture('head'); + for (const report of [base, head]) { + for (const sample of report.samples.slice(0, -1)) { + sample.phases.afterGc.heapSnapshot = null; + } + } + + expect(() => renderMemoryReportMarkdown(base, head, { + baseHeapSnapshotUrl: 'https://example.invalid/base', + headHeapSnapshotUrl: 'https://example.invalid/head', + })).toThrow('At least two samples per side are required'); +}); + +test('omits a paired-looking PSS delta that stays within observed noise', async () => { + const base = replacePssSamples(await loadFixture('base'), [290_000, 292_900, 295_800, 298_700, 301_600]); + const head = replacePssSamples(await loadFixture('head'), [292_900, 296_300, 298_700, 301_600, 290_000]); + + expect(pairedDeltaSummary( + base.samples, + head.samples, + sample => sample.phases.afterGc.memoryUsage.Pss, + ).median).toBe(2_900); + + const markdown = renderMemoryReportMarkdown(base, head, { + baseHeapSnapshotUrl: 'https://example.invalid/base', + headHeapSnapshotUrl: 'https://example.invalid/head', + }); + + expect(markdown).not.toContain('| **PSS** |'); + expect(markdown).toContain('Only metrics showing significant changes are displayed.'); + expect(markdown).not.toContain('⚠️ **Warning**: Memory usage (PSS)'); +}); + +test('keeps the convergence warning without suppressing table colour or the PSS warning', async () => { + const base = await loadFixture('base'); + const head = await loadFixture('head'); + base.samples[0].phases.afterGc.memoryStability.converged = false; + + const markdown = renderMemoryReportMarkdown(base, head, { + baseHeapSnapshotUrl: 'https://example.invalid/base', + headHeapSnapshotUrl: 'https://example.invalid/head', + }); + const memorySection = markdown.slice(0, markdown.indexOf('### V8 Heap Snapshot Statistics')); + + expect(memorySection).not.toContain('inconclusive'); + expect(findMetricRow(markdown, 'PSS')).toContain('\\color{orange}'); + expect(markdown).toContain('⚠️ **Measurement warning**: 1 memory sample did not converge.'); + expect(markdown).not.toContain('results are marked inconclusive'); + expect(markdown).toContain('⚠️ **Warning**: Memory usage (PSS)'); +}); + +test('throws for an undersampled memory comparison', async () => { + const base = await loadFixture('base'); + const head = await loadFixture('head'); + base.samples = base.samples.slice(0, 1); + head.samples = head.samples.slice(0, 1); + + expect(() => renderMemoryReportMarkdown(base, head, { + baseHeapSnapshotUrl: 'https://example.invalid/base', + headHeapSnapshotUrl: 'https://example.invalid/head', + })).toThrow('At least two samples per side are required'); +}); + +test('renders an unavailable percentage when the base median is zero', async () => { + const base = await loadFixture('base'); + const head = await loadFixture('head'); + for (const sample of base.samples) sample.phases.afterGc.memoryUsage.External = 0; + + const markdown = renderMemoryReportMarkdown(base, head, { + baseHeapSnapshotUrl: 'https://example.invalid/base', + headHeapSnapshotUrl: 'https://example.invalid/head', + }); + + expect(findMetricRow(markdown, 'External')).toContain('
-'); + expect(markdown).toContain('| Metric | @ Base | @ Head | Δ | MAD |'); + expect(markdown).not.toContain('| Metric | @ Base | @ Head | Δ | MAD | Result |'); +}); diff --git a/packages-private/diagnostics-backend/test/server.test.ts b/packages-private/diagnostics-backend/test/server.test.ts new file mode 100644 index 0000000000..b99c6f9151 --- /dev/null +++ b/packages-private/diagnostics-backend/test/server.test.ts @@ -0,0 +1,77 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { EventEmitter } from 'node:events'; +import type { ChildProcess } from 'node:child_process'; +import { describe, expect, test } from 'vitest'; +import { waitForMessage } from '../src/measure/server'; + +/** waitForMessage が使うのは message/exit/error/disconnect の購読だけなので EventEmitter で足りる */ +function createFakeServer() { + return new EventEmitter() as unknown as ChildProcess; +} + +function isPing(message: unknown): message is 'ping' { + return message === 'ping'; +} + +describe('waitForMessage', () => { + test('resolves with the first matching message', async () => { + const server = createFakeServer(); + const received = waitForMessage(server, isPing, 'ping', 1_000); + + server.emit('message', 'noise'); + server.emit('message', 'ping'); + + await expect(received).resolves.toBe('ping'); + }); + + // 子が死んだあとメッセージは来ないので、タイムアウトまで待たずに理由を添えて失敗させる + test('rejects immediately when the server exits', async () => { + const server = createFakeServer(); + const received = waitForMessage(server, isPing, 'ping', 60_000); + + server.emit('exit', 1, null); + + await expect(received).rejects.toThrow(/Server exited \(code=1, signal=null\) while waiting for ping/); + }); + + test('rejects immediately when the server errors', async () => { + const server = createFakeServer(); + const received = waitForMessage(server, isPing, 'ping', 60_000); + + server.emit('error', new Error('spawn failed')); + + await expect(received).rejects.toThrow(/spawn failed/); + }); + + test('rejects immediately when the IPC channel closes', async () => { + const server = createFakeServer(); + const received = waitForMessage(server, isPing, 'ping', 60_000); + + server.emit('disconnect'); + + await expect(received).rejects.toThrow(/IPC channel closed/); + }); + + test('rejects on timeout', async () => { + const server = createFakeServer(); + await expect(waitForMessage(server, isPing, 'ping', 1)).rejects.toThrow(/Timed out waiting for ping/); + }); + + // 待機が終わった後もリスナーが残っていると、ラウンドを重ねるごとに積み上がる + test('removes every listener once settled', async () => { + const server = createFakeServer(); + const emitter = server as unknown as EventEmitter; + const received = waitForMessage(server, isPing, 'ping', 1_000); + + server.emit('message', 'ping'); + await received; + + for (const event of ['message', 'exit', 'error', 'disconnect']) { + expect(emitter.listenerCount(event)).toBe(0); + } + }); +}); diff --git a/packages-private/diagnostics-backend/test/stability.test.ts b/packages-private/diagnostics-backend/test/stability.test.ts new file mode 100644 index 0000000000..e27332b279 --- /dev/null +++ b/packages-private/diagnostics-backend/test/stability.test.ts @@ -0,0 +1,109 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { expect, test } from 'vitest'; +import { measureMemoryUntilStable } from '../src/measure/stability'; + +function createTimer() { + let elapsedMs = 0; + + return { + now: () => elapsedMs, + wait: async (durationMs: number) => { + elapsedMs += durationMs; + }, + }; +} + +async function measure(readings: Record[]) { + let readCount = 0; + const result = await measureMemoryUntilStable( + async () => readings[readCount++], + createTimer(), + ); + return { result, readCount }; +} + +test('adopts the latest reading once Pss and Private_Dirty slopes converge', async () => { + const readings = [ + { Pss: 1000, Private_Dirty: 500, HeapUsed: 300 }, + { Pss: 1100, Private_Dirty: 520, HeapUsed: 310 }, + { Pss: 1200, Private_Dirty: 540, HeapUsed: 320 }, + ]; + const { result, readCount } = await measure(readings); + + expect(readCount).toBe(3); + expect(result.memoryUsage).toStrictEqual(readings[2]); + expect(result.stability).toStrictEqual({ + converged: true, + readingCount: 3, + elapsedMs: 4000, + maxAbsoluteSlopesKiBPerSecond: { + Pss: 50, + Private_Dirty: 10, + }, + }); +}); + +test('uses only the latest readings when determining convergence', async () => { + const readings = [ + { Pss: 1000, Private_Dirty: 500 }, + { Pss: 2000, Private_Dirty: 1000 }, + { Pss: 3000, Private_Dirty: 1500 }, + { Pss: 3040, Private_Dirty: 1520 }, + { Pss: 3080, Private_Dirty: 1540 }, + ]; + const { result, readCount } = await measure(readings); + + expect(readCount).toBe(5); + expect(result.stability.converged).toBe(true); + expect(result.stability.maxAbsoluteSlopesKiBPerSecond).toStrictEqual({ + Pss: 20, + Private_Dirty: 10, + }); +}); + +test('bounds the wait and reports the latest slopes when memory does not converge', async () => { + const readings = [ + { Pss: 1000, Private_Dirty: 500 }, + { Pss: 1600, Private_Dirty: 520 }, + { Pss: 2200, Private_Dirty: 540 }, + { Pss: 2800, Private_Dirty: 560 }, + { Pss: 3400, Private_Dirty: 580 }, + { Pss: 4000, Private_Dirty: 600 }, + ]; + const { result, readCount } = await measure(readings); + + expect(readCount).toBe(6); + expect(result.memoryUsage).toStrictEqual(readings[5]); + expect(result.stability).toStrictEqual({ + converged: false, + readingCount: 6, + elapsedMs: 10000, + maxAbsoluteSlopesKiBPerSecond: { + Pss: 300, + Private_Dirty: 10, + }, + }); +}); + +test('does not treat opposing adjacent slopes as convergence', async () => { + const readings = [ + { Pss: 1000, Private_Dirty: 500 }, + { Pss: 1600, Private_Dirty: 500 }, + { Pss: 1000, Private_Dirty: 500 }, + { Pss: 1600, Private_Dirty: 500 }, + { Pss: 1000, Private_Dirty: 500 }, + { Pss: 1600, Private_Dirty: 500 }, + ]; + const { result, readCount } = await measure(readings); + + expect(readCount).toBe(6); + expect(result.stability.converged).toBe(false); + expect(result.stability.maxAbsoluteSlopesKiBPerSecond).toStrictEqual({ + Pss: 300, + Private_Dirty: 0, + }); +}); diff --git a/packages-private/diagnostics-backend/tsconfig.json b/packages-private/diagnostics-backend/tsconfig.json new file mode 100644 index 0000000000..6672c40acb --- /dev/null +++ b/packages-private/diagnostics-backend/tsconfig.json @@ -0,0 +1,20 @@ +{ + "$schema": "https://json.schemastore.org/tsconfig", + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "bundler", + "strict": true, + "strictFunctionTypes": true, + "strictNullChecks": true, + "esModuleInterop": true, + "skipLibCheck": true, + "noEmit": true, + "types": ["node"] + }, + "include": [ + "src/**/*.ts", + "test/**/*.ts" + ], + "exclude": [] +} diff --git a/packages-private/diagnostics-frontend/eslint.config.js b/packages-private/diagnostics-frontend/eslint.config.js new file mode 100644 index 0000000000..11bc648243 --- /dev/null +++ b/packages-private/diagnostics-frontend/eslint.config.js @@ -0,0 +1,25 @@ +import tsParser from '@typescript-eslint/parser'; +import sharedConfig from '../../packages/shared/eslint.config.js'; + +// eslint-disable-next-line import/no-default-export +export default [ + ...sharedConfig, + { + ignores: [ + '**/node_modules', + '**/__snapshots__', + 'test/fixtures', + ], + }, + { + files: ['**/*.ts', '**/*.tsx'], + languageOptions: { + parserOptions: { + parser: tsParser, + project: ['./tsconfig.json'], + sourceType: 'module', + tsconfigRootDir: import.meta.dirname, + }, + }, + }, +]; diff --git a/packages-private/diagnostics-frontend/package.json b/packages-private/diagnostics-frontend/package.json new file mode 100644 index 0000000000..8d3d2d248c --- /dev/null +++ b/packages-private/diagnostics-frontend/package.json @@ -0,0 +1,25 @@ +{ + "name": "diagnostics-frontend", + "private": true, + "type": "module", + "scripts": { + "eslint": "eslint './**/*.{js,jsx,ts,tsx}'", + "inspect-browser": "tsx src/inspect-browser.ts", + "lint": "pnpm typecheck && pnpm eslint", + "render-browser-html": "tsx src/render-browser-html.ts", + "render-md": "tsx src/render-md.ts", + "test": "vitest run", + "typecheck": "tsc --noEmit" + }, + "dependencies": { + "diagnostics-shared": "workspace:*" + }, + "devDependencies": { + "@types/node": "26.1.1", + "frontend": "workspace:*", + "playwright": "1.61.1", + "tsx": "4.23.1", + "typescript": "5.9.3", + "vitest": "4.1.10" + } +} diff --git a/packages-private/diagnostics-frontend/src/browser/controller.ts b/packages-private/diagnostics-frontend/src/browser/controller.ts new file mode 100644 index 0000000000..dd17be2a62 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/browser/controller.ts @@ -0,0 +1,211 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { writeFile } from 'node:fs/promises'; +import { chromium } from 'playwright'; +import { enableNetworkTracking, type NetworkTracker } from './network'; +import type { Browser, BrowserContext, CDPSession, Page } from 'playwright'; +import type { BrowserDiagnostics, BrowserMeasurement, NetworkRequest, TabMemory, WebSocketConnection } from './types'; + +export type HeadlessChromeOptions = { + scenarioTimeoutMs: number; + baseUrl: string; +}; + +export class HeadlessChromeController { + private readonly diagnostics = { + pageErrorCount: 0, + console: {} as Record, + }; + private readonly browser: Browser; + private readonly context: BrowserContext; + public readonly page: Page; + private readonly cdp: CDPSession; + private networkTracker: NetworkTracker | null = null; + + private constructor( + browser: Browser, + context: BrowserContext, + page: Page, + cdp: CDPSession, + options: HeadlessChromeOptions, + ) { + this.browser = browser; + this.context = context; + this.page = page; + this.cdp = cdp; + this.page.setDefaultTimeout(options.scenarioTimeoutMs); + this.page.setDefaultNavigationTimeout(options.scenarioTimeoutMs); + this.page.on('pageerror', () => { + this.diagnostics.pageErrorCount++; + }); + this.page.on('console', message => { + const type = message.type(); + this.diagnostics.console[type] = (this.diagnostics.console[type] ?? 0) + 1; + }); + } + + public get networkRequests(): NetworkRequest[] { + return this.networkTracker?.networkRequests ?? []; + } + + public get webSocketConnections(): WebSocketConnection[] { + return this.networkTracker?.webSocketConnections ?? []; + } + + static async create(label: string, options: HeadlessChromeOptions): Promise { + process.stderr.write(`[${label}] Launching Playwright Chromium\n`); + const browser = await chromium.launch({ + channel: 'chromium', + headless: true, + args: [ + '--disable-gpu', + '--disable-dev-shm-usage', + '--disable-background-networking', + '--disable-default-apps', + '--disable-extensions', + '--disable-sync', + '--metrics-recording-only', + '--no-first-run', + '--no-default-browser-check', + '--no-sandbox', + ], + }); + + try { + const context = await browser.newContext({ + baseURL: options.baseUrl, + locale: 'en-US', + }); + await context.addInitScript(() => { + // @ts-expect-error Test-only runtime hint consumed by Misskey frontend code. + window.isPlaywright = true; + }); + + const page = await context.newPage(); + const cdp = await context.newCDPSession(page); + return new HeadlessChromeController(browser, context, page, cdp, options); + } catch (error) { + await browser.close().catch(() => undefined); + throw error; + } + } + + static async with(label: string, options: HeadlessChromeOptions, callback: (browser: HeadlessChromeController) => T | Promise): Promise { + const browser = await HeadlessChromeController.create(label, options); + try { + return await callback(browser); + } finally { + await browser.close(); + } + } + + public async enableNetworkTracking() { + this.networkTracker = await enableNetworkTracking(this.cdp); + } + + public async waitForNetworkDetails() { + await this.networkTracker?.waitForDetails(); + } + + public async evaluate(expression: string, timeoutMs = 30_000): Promise { + return await Promise.race([ + this.page.evaluate(expression), + new Promise((_, reject) => setTimeout(() => reject(new Error(`Playwright evaluate timed out after ${timeoutMs}ms`)), timeoutMs).unref()), + ]) as T; + } + + public collectDiagnostics(): BrowserDiagnostics { + return { + pageErrorCount: this.diagnostics.pageErrorCount, + console: { + log: this.diagnostics.console.log ?? 0, + warning: this.diagnostics.console.warning ?? 0, + error: this.diagnostics.console.error ?? 0, + info: this.diagnostics.console.info ?? 0, + }, + }; + } + + public async collectPerformance(): Promise { + const cdpMetricsResult = await this.cdp.send('Performance.getMetrics'); + const cdpMetrics = Object.fromEntries(cdpMetricsResult.metrics.map(metric => [metric.name, metric.value])); + const runtimeHeap = await this.cdp.send('Runtime.getHeapUsage').catch(() => undefined); + const tabMemory = await this.collectTabMemory(); + const webVitals = await this.evaluate(`(() => { + const navigation = performance.getEntriesByType('navigation')[0]; + const paintEntries = Object.fromEntries(performance.getEntriesByType('paint').map(entry => [entry.name, entry.startTime])); + const longTasks = performance.getEntriesByType('longtask'); + const resourceEntries = performance.getEntriesByType('resource'); + return { + firstPaintMs: paintEntries['first-paint'], + firstContentfulPaintMs: paintEntries['first-contentful-paint'], + domContentLoadedEventEndMs: navigation?.domContentLoadedEventEnd, + loadEventEndMs: navigation?.loadEventEnd, + longTaskCount: longTasks.length, + longTaskDurationMs: longTasks.reduce((sum, entry) => sum + entry.duration, 0), + maxLongTaskDurationMs: longTasks.reduce((max, entry) => Math.max(max, entry.duration), 0), + resourceEntryCount: resourceEntries.length, + domElements: document.getElementsByTagName('*').length, + }; + })()`); + + return { + cdpMetrics, + runtimeHeap, + tabMemory, + webVitals, + }; + } + + public async collectTabMemory(): Promise { + const userAgentSpecificMemory = await this.evaluate<{ bytes?: number }>(`(async () => { + const measureMemory = performance.measureUserAgentSpecificMemory; + if (typeof measureMemory !== 'function') return {}; + const result = await measureMemory.call(performance); + return { bytes: result.bytes }; + })()`, 60_000); + + const userAgentSpecificBytes = userAgentSpecificMemory?.bytes; + if (!Number.isFinite(userAgentSpecificBytes)) { + throw new Error('performance.measureUserAgentSpecificMemory() did not return finite bytes'); + } + + return { + totalBytes: userAgentSpecificBytes as number, + }; + } + + public async takeHeapSnapshot(savePath?: string) { + const chunks: string[] = []; + const onChunk = (params: { chunk: string }) => { + chunks.push(params.chunk); + }; + this.cdp.on('HeapProfiler.addHeapSnapshotChunk', onChunk); + + let content: string; + try { + await this.cdp.send('HeapProfiler.enable'); + await this.cdp.send('HeapProfiler.collectGarbage'); + await this.cdp.send('HeapProfiler.takeHeapSnapshot', { reportProgress: false }); + content = chunks.join(''); + } finally { + // 外さないとラウンドごとに積み上がり、古い配列にチャンクを流し続ける + this.cdp.off('HeapProfiler.addHeapSnapshotChunk', onChunk); + } + + if (savePath != null) { + await writeFile(savePath, content); + } + + return JSON.parse(content); + } + + public async close() { + await this.cdp.detach().catch(() => undefined); + await this.context.close().catch(() => undefined); + await this.browser.close().catch(() => undefined); + } +} diff --git a/packages-private/diagnostics-frontend/src/browser/diagnostics.ts b/packages-private/diagnostics-frontend/src/browser/diagnostics.ts new file mode 100644 index 0000000000..a4c95bf842 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/browser/diagnostics.ts @@ -0,0 +1,21 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { median } from 'diagnostics-shared/stats'; +import type { BrowserDiagnostics } from './types'; + +export function summarizeBrowserDiagnostics(samples: BrowserDiagnostics[]): BrowserDiagnostics { + const medianOf = (select: (sample: BrowserDiagnostics) => number) => median(samples.map(select)); + + return { + pageErrorCount: medianOf(sample => sample.pageErrorCount), + console: { + log: medianOf(sample => sample.console.log), + warning: medianOf(sample => sample.console.warning), + error: medianOf(sample => sample.console.error), + info: medianOf(sample => sample.console.info), + }, + }; +} diff --git a/packages-private/diagnostics-frontend/src/browser/network.ts b/packages-private/diagnostics-frontend/src/browser/network.ts new file mode 100644 index 0000000000..77095ca689 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/browser/network.ts @@ -0,0 +1,265 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { CDPSession } from 'playwright'; +import type { NetworkRequest, NetworkSummary, WebSocketConnection } from './types'; + +function normalizeHeaders(headers: Record | undefined) { + if (headers == null) return undefined; + const normalized = {} as Record; + for (const [key, value] of Object.entries(headers)) { + normalized[key] = String(value); + } + return normalized; +} + +function webSocketFramePayloadBytes(frame: { opcode?: number; payloadData?: string } | undefined) { + if (frame?.payloadData == null) return 0; + // opcode 1 はテキストフレームで、それ以外はbase64エンコードされたバイナリとして届く + if (frame.opcode === 1) return Buffer.byteLength(frame.payloadData, 'utf8'); + return Buffer.byteLength(frame.payloadData, 'base64'); +} + +export type NetworkTracker = { + networkRequests: NetworkRequest[]; + webSocketConnections: WebSocketConnection[]; + /** CDPへの追加問い合わせ (postDataの取得) が全て決着するまで待つ */ + waitForDetails: () => Promise; +}; + +/** + * CDPのNetworkドメインを有効化し、リクエスト/WebSocketの生ログを収集し続けるトラッカーを返す。 + */ +export async function enableNetworkTracking(cdp: CDPSession): Promise { + const networkRequests: NetworkRequest[] = []; + const webSocketConnections: WebSocketConnection[] = []; + const requests = new Map(); + const webSockets = new Map(); + const pendingDetailReads: Promise[] = []; + + const readRequestBody = (row: NetworkRequest) => { + if (!row.hasRequestBody || row.requestBody != null) return; + const pending = cdp.send('Network.getRequestPostData', { + requestId: row.requestId, + }).then(result => { + row.requestBody = result.postData; + }).catch(() => { + // Some requests expose hasPostData but no longer have retrievable body data. + }); + pendingDetailReads.push(pending); + }; + + cdp.on('Network.requestWillBeSent', params => { + if (params.request?.url == null) return; + const row: NetworkRequest = { + requestId: params.requestId, + url: params.request.url, + method: params.request.method ?? 'GET', + resourceType: params.type ?? 'Other', + startedAt: params.timestamp ?? 0, + documentUrl: params.documentURL, + requestHeaders: normalizeHeaders(params.request.headers), + requestBody: typeof params.request.postData === 'string' ? params.request.postData : undefined, + hasRequestBody: params.request.hasPostData === true || typeof params.request.postData === 'string', + encodedDataLength: 0, + decodedBodyLength: 0, + fromDiskCache: false, + fromServiceWorker: false, + finished: false, + failed: false, + }; + requests.set(params.requestId, row); + networkRequests.push(row); + }); + + cdp.on('Network.webSocketCreated', params => { + if (params.requestId == null || params.url == null) return; + const row: WebSocketConnection = { + requestId: params.requestId, + url: params.url, + // Network.webSocketCreated はtimestampを持たないので、closedAt との差分は取れない + createdAt: 0, + sentFrameCount: 0, + receivedFrameCount: 0, + sentBytes: 0, + receivedBytes: 0, + errorCount: 0, + }; + webSockets.set(params.requestId, row); + webSocketConnections.push(row); + }); + + cdp.on('Network.webSocketWillSendHandshakeRequest', params => { + const row = webSockets.get(params.requestId); + if (row == null) return; + row.handshakeRequestHeaders = normalizeHeaders(params.request?.headers); + }); + + cdp.on('Network.webSocketHandshakeResponseReceived', params => { + const row = webSockets.get(params.requestId); + if (row == null) return; + row.handshakeResponseStatus = params.response?.status; + row.handshakeResponseStatusText = params.response?.statusText; + row.handshakeResponseHeaders = normalizeHeaders(params.response?.headers); + }); + + cdp.on('Network.webSocketFrameSent', params => { + const row = webSockets.get(params.requestId); + if (row == null) return; + row.sentFrameCount += 1; + row.sentBytes += webSocketFramePayloadBytes(params.response); + }); + + cdp.on('Network.webSocketFrameReceived', params => { + const row = webSockets.get(params.requestId); + if (row == null) return; + row.receivedFrameCount += 1; + row.receivedBytes += webSocketFramePayloadBytes(params.response); + }); + + cdp.on('Network.webSocketFrameError', params => { + const row = webSockets.get(params.requestId); + if (row == null) return; + row.errorCount += 1; + }); + + cdp.on('Network.webSocketClosed', params => { + const row = webSockets.get(params.requestId); + if (row == null) return; + row.closedAt = params.timestamp ?? 0; + }); + + cdp.on('Network.responseReceived', params => { + const row = requests.get(params.requestId); + if (row == null) return; + row.status = params.response?.status; + row.statusText = params.response?.statusText; + row.mimeType = params.response?.mimeType; + row.responseHeaders = normalizeHeaders(params.response?.headers); + row.protocol = params.response?.protocol; + row.remoteIPAddress = params.response?.remoteIPAddress; + row.remotePort = params.response?.remotePort; + row.requestHeaders ??= normalizeHeaders(params.response?.requestHeaders); + row.fromDiskCache = params.response?.fromDiskCache === true; + row.fromServiceWorker = params.response?.fromServiceWorker === true; + }); + + cdp.on('Network.dataReceived', params => { + const row = requests.get(params.requestId); + if (row == null) return; + row.decodedBodyLength += params.dataLength ?? 0; + row.encodedDataLength += params.encodedDataLength ?? 0; + }); + + cdp.on('Network.loadingFinished', params => { + const row = requests.get(params.requestId); + if (row == null) return; + row.finished = true; + row.encodedDataLength = Math.max(row.encodedDataLength, params.encodedDataLength ?? 0); + readRequestBody(row); + }); + + cdp.on('Network.loadingFailed', params => { + const row = requests.get(params.requestId); + if (row == null) return; + row.failed = true; + row.finished = true; + row.errorText = params.errorText; + readRequestBody(row); + }); + + await cdp.send('Network.enable'); + await cdp.send('Network.setCacheDisabled', { cacheDisabled: true }); + await cdp.send('Network.setBypassServiceWorker', { bypass: true }); + await cdp.send('Page.enable'); + await cdp.send('Runtime.enable'); + await cdp.send('Performance.enable'); + + return { + networkRequests, + webSocketConnections, + waitForDetails: async () => { + // 待っている最中にさらにpendingが増えることがあるので、増えなくなるまで繰り返す + let settledCount = 0; + while (settledCount < pendingDetailReads.length) { + const pending = pendingDetailReads.slice(settledCount); + settledCount = pendingDetailReads.length; + await Promise.allSettled(pending); + } + }, + }; +} + +function isMeasurableRequest(row: NetworkRequest) { + return !row.url.startsWith('data:') && !row.url.startsWith('blob:') && !row.url.startsWith('devtools:'); +} + +export function summarizeNetwork(requestRows: NetworkRequest[], baseUrl: string, webSocketRows?: WebSocketConnection[]): NetworkSummary { + const origin = new URL(baseUrl).origin; + const rows = requestRows.filter(isMeasurableRequest); + const byResourceType = {} as NetworkSummary['byResourceType']; + + for (const row of rows) { + const summary = byResourceType[row.resourceType] ?? { + requests: 0, + encodedBytes: 0, + decodedBodyBytes: 0, + }; + summary.requests += 1; + summary.encodedBytes += row.encodedDataLength; + summary.decodedBodyBytes += row.decodedBodyLength; + byResourceType[row.resourceType] = summary; + } + + function isSameOrigin(url: string) { + try { + return new URL(url).origin === origin; + } catch { + return false; + } + } + + return { + requestCount: rows.length, + webSocketConnectionCount: webSocketRows == null + ? rows.filter(row => row.resourceType === 'WebSocket').length + : webSocketRows.length, + webSocketSentBytes: webSocketRows?.reduce((sum, row) => sum + row.sentBytes, 0) ?? 0, + webSocketReceivedBytes: webSocketRows?.reduce((sum, row) => sum + row.receivedBytes, 0) ?? 0, + finishedRequestCount: rows.filter(row => row.finished).length, + failedRequestCount: rows.filter(row => row.failed).length, + cachedRequestCount: rows.filter(row => row.fromDiskCache).length, + serviceWorkerRequestCount: rows.filter(row => row.fromServiceWorker).length, + totalEncodedBytes: rows.reduce((sum, row) => sum + row.encodedDataLength, 0), + totalDecodedBodyBytes: rows.reduce((sum, row) => sum + row.decodedBodyLength, 0), + sameOriginEncodedBytes: rows + .filter(row => isSameOrigin(row.url)) + .reduce((sum, row) => sum + row.encodedDataLength, 0), + thirdPartyEncodedBytes: rows + .filter(row => !isSameOrigin(row.url)) + .reduce((sum, row) => sum + row.encodedDataLength, 0), + byResourceType, + largestRequests: rows + .toSorted((a, b) => b.encodedDataLength - a.encodedDataLength) + .slice(0, 15) + .map(row => ({ + url: row.url, + method: row.method, + resourceType: row.resourceType, + status: row.status, + encodedBytes: row.encodedDataLength, + decodedBodyBytes: row.decodedBodyLength, + })), + failedRequests: rows + .filter(row => row.failed) + .map(row => ({ + url: row.url, + method: row.method, + resourceType: row.resourceType, + errorText: row.errorText, + status: row.status, + })), + }; +} diff --git a/packages-private/diagnostics-frontend/src/browser/report/html-styles.ts b/packages-private/diagnostics-frontend/src/browser/report/html-styles.ts new file mode 100644 index 0000000000..8e8603ba51 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/browser/report/html-styles.ts @@ -0,0 +1,177 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +/** 差分HTMLは単体のファイルとして配布されるので、CSSも埋め込みで持つ */ +export const networkDiffHtmlStyles = ` :root { + color-scheme: light dark; + --bg: #f7f7f8; + --fg: #202124; + --muted: #5f6368; + --card: #ffffff; + --border: #dfe1e5; + --added: #137333; + --added-bg: #e6f4ea; + --removed: #a50e0e; + --removed-bg: #fce8e6; + } + @media (prefers-color-scheme: dark) { + :root { + --bg: #111315; + --fg: #e8eaed; + --muted: #bdc1c6; + --card: #1b1d20; + --border: #3c4043; + --added-bg: #17351f; + --removed-bg: #3c1f1d; + } + } + body { + margin: 0; + font: 14px/1.5 system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; + background: var(--bg); + color: var(--fg); + } + main { + max-width: 1200px; + margin: 0 auto; + padding: 24px; + } + h1 { + font-size: 24px; + margin: 0 0 8px; + } + h2 { + font-size: 18px; + margin: 32px 0 8px; + } + .meta { + color: var(--muted); + margin: 0 0 24px; + } + .summary { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(160px, 1fr)); + gap: 12px; + margin: 24px 0; + } + .summary > div, .request, table { + background: var(--card); + border: 1px solid var(--border); + border-radius: 8px; + } + .summary > div { + padding: 14px; + } + .label { + display: block; + color: var(--muted); + font-size: 12px; + } + .summary strong { + display: block; + font-size: 24px; + margin-top: 4px; + } + .added-text { + color: var(--added); + } + .removed-text { + color: var(--removed); + } + table { + border-collapse: collapse; + width: 100%; + overflow: hidden; + } + th, td { + border-bottom: 1px solid var(--border); + padding: 8px 10px; + text-align: left; + } + th { + color: var(--muted); + font-weight: 600; + } + .num { + text-align: right; + } + .requests { + display: grid; + gap: 12px; + } + .request { + padding: 14px; + overflow-wrap: anywhere; + } + .request.added { + border-left: 4px solid var(--added); + } + .request.removed { + border-left: 4px solid var(--removed); + } + .request header { + display: flex; + flex-wrap: wrap; + gap: 8px; + align-items: center; + margin-bottom: 8px; + } + .badge, .method, .type, .status { + border-radius: 999px; + padding: 2px 8px; + font-size: 12px; + font-weight: 600; + } + .added .badge { + background: var(--added-bg); + color: var(--added); + } + .removed .badge { + background: var(--removed-bg); + color: var(--removed); + } + .method, .type, .status { + background: color-mix(in srgb, var(--muted) 14%, transparent); + color: var(--fg); + } + .url { + display: block; + margin: 8px 0 12px; + color: inherit; + font-family: ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", monospace; + } + dl { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); + gap: 8px 16px; + margin: 0 0 12px; + } + dl div { + min-width: 0; + } + dt { + color: var(--muted); + font-size: 12px; + } + dd { + margin: 0; + } + details { + margin-top: 8px; + } + summary { + cursor: pointer; + color: var(--muted); + } + pre { + white-space: pre-wrap; + overflow-x: auto; + background: color-mix(in srgb, var(--muted) 10%, transparent); + border-radius: 6px; + padding: 10px; + } + .empty { + color: var(--muted); + }`; diff --git a/packages-private/diagnostics-frontend/src/browser/report/html.ts b/packages-private/diagnostics-frontend/src/browser/report/html.ts new file mode 100644 index 0000000000..ff1ca47839 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/browser/report/html.ts @@ -0,0 +1,255 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { formatBytes, formatNumber } from 'diagnostics-shared/format'; +import { type Raw, html, joinHtml, raw } from 'diagnostics-shared/html'; +import { networkDiffHtmlStyles } from './html-styles'; +import type { BrowserMeasurementSample, BrowserMetricsReport, NetworkRequest } from '../types'; + +type DiffDirection = 'added' | 'removed'; + +type RequestDiff = { + direction: DiffDirection; + round: number; + baseCount: number; + headCount: number; + request: NetworkRequest; +}; + +function isHttpRequest(request: NetworkRequest) { + try { + const { protocol } = new URL(request.url); + return protocol === 'http:' || protocol === 'https:'; + } catch { + return false; + } +} + +function requestKey(request: NetworkRequest) { + // URLに現れない文字で区切らないと、区切り文字を含むURLが別のキーと衝突しうる + return [ + request.method, + request.resourceType, + request.url, + ].join('\u0000'); +} + +function groupRequests(requests: NetworkRequest[] | undefined) { + const grouped = new Map(); + for (const request of requests ?? []) { + if (!isHttpRequest(request)) continue; + const key = requestKey(request); + const rows = grouped.get(key) ?? []; + rows.push(request); + grouped.set(key, rows); + } + return grouped; +} + +function byRound(samples: BrowserMeasurementSample[]) { + return new Map(samples.map(sample => [sample.round, sample])); +} + +/** + * 同じラウンドどうしで、同一 (method, resourceType, URL) のリクエスト本数を突き合わせる。 + * 本数が増えていればhead側の増分を added、減っていればbase側の余りを removed として扱う。 + */ +function diffRound(round: number, baseSample: BrowserMeasurementSample | undefined, headSample: BrowserMeasurementSample | undefined) { + const baseRequests = groupRequests(baseSample?.networkRequests); + const headRequests = groupRequests(headSample?.networkRequests); + const keys = [...new Set([ + ...baseRequests.keys(), + ...headRequests.keys(), + ])].toSorted(); + const diffs: RequestDiff[] = []; + + for (const key of keys) { + const baseRows = baseRequests.get(key) ?? []; + const headRows = headRequests.get(key) ?? []; + if (headRows.length > baseRows.length) { + for (const request of headRows.slice(baseRows.length)) { + diffs.push({ + direction: 'added', + round, + baseCount: baseRows.length, + headCount: headRows.length, + request, + }); + } + } else if (baseRows.length > headRows.length) { + for (const request of baseRows.slice(headRows.length)) { + diffs.push({ + direction: 'removed', + round, + baseCount: baseRows.length, + headCount: headRows.length, + request, + }); + } + } + } + + return diffs; +} + +function diffReports(base: BrowserMetricsReport, head: BrowserMetricsReport) { + const baseSamples = byRound(base.samples); + const headSamples = byRound(head.samples); + const rounds = [...new Set([ + ...baseSamples.keys(), + ...headSamples.keys(), + ])].toSorted((a, b) => a - b); + return rounds.flatMap(round => diffRound(round, baseSamples.get(round), headSamples.get(round))); +} + +function formatMaybeJson(value: string | undefined) { + if (value == null || value === '') return null; + try { + return JSON.stringify(JSON.parse(value), null, '\t'); + } catch { + return value; + } +} + +function formatHeaders(headers: Record | undefined) { + if (headers == null || Object.keys(headers).length === 0) return null; + return JSON.stringify(headers, null, '\t'); +} + +function countBy(diffs: RequestDiff[], getKey: (diff: RequestDiff) => T) { + const counts = new Map(); + for (const diff of diffs) { + counts.set(getKey(diff), (counts.get(getKey(diff)) ?? 0) + 1); + } + return [...counts].toSorted((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])); +} + +function renderSummary(base: BrowserMetricsReport, head: BrowserMetricsReport, diffs: RequestDiff[]): Raw { + const added = diffs.filter(diff => diff.direction === 'added').length; + const removed = diffs.filter(diff => diff.direction === 'removed').length; + const typeCounts = countBy(diffs, diff => diff.request.resourceType); + const typeRows = joinHtml(typeCounts.map(([type, count]) => html` + + ${type} + ${formatNumber(count)} + `), ''); + + return html` +
+
+ Base samples + ${formatNumber(base.sampleCount)} +
+
+ Head samples + ${formatNumber(head.sampleCount)} +
+
+ Added in Head + ${formatNumber(added)} +
+
+ Removed in Head + ${formatNumber(removed)} +
+
+ ${typeCounts.length === 0 ? raw('') : html` +
+

Diffs by Resource Type

+ + + ${typeRows} + +
TypeDiff requests
+
`}`; +} + +function renderDetails(title: string, content: string | null, open = false): Raw { + if (content == null || content === '') return raw(''); + return html` + + ${title} + 
${content}
+ `; +} + +function renderRequest(diff: RequestDiff): Raw { + const { request } = diff; + const requestBody = formatMaybeJson(request.requestBody); + const requestHeaders = formatHeaders(request.requestHeaders); + const responseHeaders = formatHeaders(request.responseHeaders); + const bodyNote = requestBody == null && request.hasRequestBody === true + ? html`

Request body was present but could not be retrieved from CDP.

` + : raw(''); + + return html` +
+
+ ${diff.direction === 'added' ? 'Added in Head' : 'Removed in Head'} + ${request.method} + ${request.resourceType} + ${request.status ?? '-'} +
+ ${request.url} +
+
Round
${formatNumber(diff.round)}
+
Base count
${formatNumber(diff.baseCount)}
+
Head count
${formatNumber(diff.headCount)}
+
Encoded
${formatBytes(request.encodedDataLength ?? 0)}
+
Decoded body
${formatBytes(request.decodedBodyLength ?? 0)}
+
MIME
${request.mimeType ?? '-'}
+
Protocol
${request.protocol ?? '-'}
+
Remote
${request.remoteIPAddress == null ? '-' : `${request.remoteIPAddress}:${request.remotePort ?? ''}`}
+
Failed
${request.failed ? (request.errorText ?? 'yes') : 'no'}
+
+ ${bodyNote} + ${renderDetails('Request body', requestBody, requestBody != null)} + ${renderDetails('Request headers', requestHeaders)} + ${renderDetails('Response headers', responseHeaders)} +
`; +} + +function renderRound(round: number, diffs: RequestDiff[]): Raw { + const added = diffs.filter(diff => diff.direction === 'added').length; + const removed = diffs.filter(diff => diff.direction === 'removed').length; + return html` +
+

Round ${formatNumber(round)}

+

${formatNumber(added)} added, ${formatNumber(removed)} removed

+
+ ${joinHtml(diffs.map(renderRequest), '\n')} +
+
`; +} + +export function renderBrowserDiagnosticsHtml(base: BrowserMetricsReport, head: BrowserMetricsReport) { + const diffs = diffReports(base, head); + const rounds = [...new Set(diffs.map(diff => diff.round))].toSorted((a, b) => a - b); + const generatedAt = new Date().toISOString(); + const content = diffs.length === 0 + ? html`

No added or removed HTTP(S) requests were found in paired samples.

` + : joinHtml(rounds.map(round => renderRound(round, diffs.filter(diff => diff.round === round))), '\n'); + + return String(html` + + + + + Frontend Browser Network Request Diff + + + +
+

Frontend Browser Network Request Diff

+

Generated at ${generatedAt}. Requests are compared per paired round by method, resource type, and exact URL. Bodies are shown for added/removed request instances when CDP exposes them.

+ ${renderSummary(base, head, diffs)} + ${content} +
+ + +`); +} diff --git a/packages-private/diagnostics-frontend/src/browser/scenario.ts b/packages-private/diagnostics-frontend/src/browser/scenario.ts new file mode 100644 index 0000000000..80df8532ec --- /dev/null +++ b/packages-private/diagnostics-frontend/src/browser/scenario.ts @@ -0,0 +1,31 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { closeUserSetupDialog, postNote, registerUser, resetState, signupThroughUi, visitHome } from '../../../../packages/frontend/test/e2e/shared'; +import { sleep } from './server'; +import type { HeadlessChromeController } from './controller'; + +export const scenarioDescription = 'fresh browser signup, first timeline note, after the note becomes visible'; + +/** + * 各ラウンドを同じ初期状態から始めるため、DBを消して管理者だけ作り直す。 + */ +export async function prepareInstance(baseUrl: string) { + await resetState(baseUrl); + await registerUser(baseUrl, 'admin', 'admin1234', true); +} + +export async function runSignupAndPostScenario(chrome: HeadlessChromeController, baseUrl: string) { + const page = chrome.page; + const noteText = `Frontend browser metrics ${Date.now()}`; + + await visitHome(page, baseUrl); + await signupThroughUi(page, { username: 'alice', password: 'password' }); + await closeUserSetupDialog(page); + await postNote(page, noteText, 10_000); + + // 投稿直後の非同期処理が落ち着いてから計測したいので少し待つ + await sleep(1000); +} diff --git a/packages-private/diagnostics-frontend/src/browser/server.ts b/packages-private/diagnostics-frontend/src/browser/server.ts new file mode 100644 index 0000000000..72f7a6f79e --- /dev/null +++ b/packages-private/diagnostics-frontend/src/browser/server.ts @@ -0,0 +1,104 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { spawn, spawnSync, type ChildProcess } from 'node:child_process'; + +export function sleep(ms: number) { + return new Promise(resolvePromise => setTimeout(resolvePromise, ms)); +} + +function commandName(command: string) { + if (process.platform !== 'win32') return command; + if (command === 'pnpm') return 'pnpm.cmd'; + return command; +} + +/** + * 計測対象のリポジトリで Misskey テストサーバーを起動する。 + * POSIXでは detached にして、後で子孫プロセスごとまとめて落とせるようにする。 + */ +export function startServer(label: string, repoDir: string) { + process.stderr.write(`[${label}] Starting Misskey test server\n`); + const child = spawn(commandName('pnpm'), ['start:test'], { + cwd: repoDir, + env: process.env, + stdio: ['ignore', 'pipe', 'pipe'], + detached: process.platform !== 'win32', + }); + child.stdout.on('data', data => process.stderr.write(`[server:${label}] ${data}`)); + child.stderr.on('data', data => process.stderr.write(`[server:${label}] ${data}`)); + return child; +} + +const serverStartupTimeoutMs = 120_000; + +function hasExited(child: ChildProcess) { + return child.exitCode != null || child.signalCode != null; +} + +export async function waitForServer(baseUrl: string, child: ChildProcess) { + const startedAt = Date.now(); + while (Date.now() - startedAt < serverStartupTimeoutMs) { + if (child.exitCode != null) throw new Error(`Misskey server exited early with code ${child.exitCode}`); + if (child.signalCode != null) throw new Error(`Misskey server exited early with signal ${child.signalCode}`); + try { + // 応答が返らないままだとfetchが待ち続け、外側の120秒の上限を超えてしまう + const remainingMs = serverStartupTimeoutMs - (Date.now() - startedAt); + const response = await fetch(`${baseUrl}/`, { + redirect: 'manual', + signal: AbortSignal.timeout(remainingMs), + }); + if (response.status < 500) return; + } catch { + // 中断・接続拒否いずれもまだ起動中とみなしてリトライする + } + await sleep(1_000); + } + throw new Error(`Timed out waiting for ${baseUrl}`); +} + +export async function stopServer(child: ChildProcess) { + if (hasExited(child)) return; + + if (process.platform === 'win32') { + spawnSync('taskkill', ['/pid', String(child.pid), '/t', '/f'], { stdio: 'ignore' }); + } else if (child.pid != null) { + try { + // プロセスグループごと落とさないと pnpm 配下の node が残る + process.kill(-child.pid, 'SIGTERM'); + } catch { + child.kill('SIGTERM'); + } + } + + await new Promise(resolvePromise => { + if (hasExited(child)) { + resolvePromise(); + return; + } + + const forceKillTimer = setTimeout(() => { + // 猶予の間に終了していれば、PIDが再利用されて無関係のプロセスを撃つ恐れがある + if (!hasExited(child) && child.pid != null) { + try { + if (process.platform === 'win32') { + spawnSync('taskkill', ['/pid', String(child.pid), '/t', '/f'], { stdio: 'ignore' }); + } else { + process.kill(-child.pid, 'SIGKILL'); + } + } catch { + child.kill('SIGKILL'); + } + } + resolvePromise(); + }, 10_000); + forceKillTimer.unref(); + + child.once('exit', () => { + clearTimeout(forceKillTimer); + resolvePromise(); + }); + }); +} diff --git a/packages-private/diagnostics-frontend/src/browser/summarize.ts b/packages-private/diagnostics-frontend/src/browser/summarize.ts new file mode 100644 index 0000000000..1b88002208 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/browser/summarize.ts @@ -0,0 +1,158 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { finiteMedian } from 'diagnostics-shared/stats'; +import { summarizeHeapSnapshotDataSamples } from 'diagnostics-shared/heap-snapshot'; +import { summarizeBrowserDiagnostics } from './diagnostics'; +import type { BrowserMeasurement, BrowserMeasurementSample, BrowserMetricsReport, NetworkSummary } from './types'; + +export type SummarizeOptions = { + url: string; + heapSnapshotBreakdownTopN: number; +}; + +/** + * 中央値に最も近いサンプルを1つ選ぶ。 + * largestRequests のように中央値を取れない項目は、代表サンプルの値をそのまま採用する。 + */ +export function selectRepresentativeSample(samples: BrowserMeasurementSample[], getValue: (sample: BrowserMeasurementSample) => number) { + const medianValue = finiteMedian(samples.map(getValue), 0); + let selected: { sample: BrowserMeasurementSample; distance: number } | null = null; + + for (const sample of samples) { + const value = getValue(sample); + if (!Number.isFinite(value)) continue; + const distance = Math.abs(value - medianValue); + if (selected == null || distance < selected.distance || (distance === selected.distance && sample.round < selected.sample.round)) { + selected = { + sample, + distance, + }; + } + } + + return selected?.sample ?? samples[0]; +} + +function summarizeResourceType(samples: BrowserMeasurementSample[], resourceType: string) { + return { + requests: finiteMedian(samples.map(sample => sample.network.byResourceType[resourceType]?.requests), 0), + encodedBytes: finiteMedian(samples.map(sample => sample.network.byResourceType[resourceType]?.encodedBytes), 0), + decodedBodyBytes: finiteMedian(samples.map(sample => sample.network.byResourceType[resourceType]?.decodedBodyBytes), 0), + }; +} + +export function summarizeNetworkSamples(samples: BrowserMeasurementSample[]): NetworkSummary { + const resourceTypes = new Set(); + for (const sample of samples) { + for (const resourceType of Object.keys(sample.network.byResourceType)) { + resourceTypes.add(resourceType); + } + } + + const representative = selectRepresentativeSample(samples, sample => sample.network.totalEncodedBytes); + const byResourceType = {} as NetworkSummary['byResourceType']; + for (const resourceType of resourceTypes) { + byResourceType[resourceType] = summarizeResourceType(samples, resourceType); + } + + return { + requestCount: finiteMedian(samples.map(sample => sample.network.requestCount), 0), + webSocketConnectionCount: finiteMedian(samples.map(sample => sample.network.webSocketConnectionCount), 0), + webSocketSentBytes: finiteMedian(samples.map(sample => sample.network.webSocketSentBytes), 0), + webSocketReceivedBytes: finiteMedian(samples.map(sample => sample.network.webSocketReceivedBytes), 0), + finishedRequestCount: finiteMedian(samples.map(sample => sample.network.finishedRequestCount), 0), + failedRequestCount: finiteMedian(samples.map(sample => sample.network.failedRequestCount), 0), + cachedRequestCount: finiteMedian(samples.map(sample => sample.network.cachedRequestCount), 0), + serviceWorkerRequestCount: finiteMedian(samples.map(sample => sample.network.serviceWorkerRequestCount), 0), + totalEncodedBytes: finiteMedian(samples.map(sample => sample.network.totalEncodedBytes), 0), + totalDecodedBodyBytes: finiteMedian(samples.map(sample => sample.network.totalDecodedBodyBytes), 0), + sameOriginEncodedBytes: finiteMedian(samples.map(sample => sample.network.sameOriginEncodedBytes), 0), + thirdPartyEncodedBytes: finiteMedian(samples.map(sample => sample.network.thirdPartyEncodedBytes), 0), + byResourceType, + largestRequests: representative.network.largestRequests, + failedRequests: representative.network.failedRequests, + }; +} + +export function summarizePerformanceSamples(samples: BrowserMeasurementSample[]): BrowserMeasurement['performance'] { + const cdpMetricKeys = new Set(); + for (const sample of samples) { + for (const key of Object.keys(sample.performance.cdpMetrics)) { + cdpMetricKeys.add(key); + } + } + + const cdpMetrics = {} as Record; + for (const key of cdpMetricKeys) { + cdpMetrics[key] = finiteMedian(samples.map(sample => sample.performance.cdpMetrics[key]), 0); + } + + const webVitalKeys = [ + 'firstPaintMs', + 'firstContentfulPaintMs', + 'domContentLoadedEventEndMs', + 'loadEventEndMs', + 'longTaskCount', + 'longTaskDurationMs', + 'maxLongTaskDurationMs', + 'resourceEntryCount', + 'domElements', + ] as const satisfies (keyof BrowserMeasurement['performance']['webVitals'])[]; + + const webVitals = {} as BrowserMeasurement['performance']['webVitals']; + for (const key of webVitalKeys) { + webVitals[key] = finiteMedian(samples.map(sample => sample.performance.webVitals[key]), 0); + } + + return { + cdpMetrics, + runtimeHeap: { + usedSize: finiteMedian(samples.map(sample => sample.performance.runtimeHeap?.usedSize), 0), + totalSize: finiteMedian(samples.map(sample => sample.performance.runtimeHeap?.totalSize), 0), + }, + tabMemory: { + totalBytes: finiteMedian(samples.map(sample => sample.performance.tabMemory.totalBytes), 0), + }, + webVitals, + }; +} + +export function summarizeHeapSnapshotSamples(samples: BrowserMeasurementSample[], breakdownTopN: number) { + const summary = summarizeHeapSnapshotDataSamples( + samples, + sample => sample.heapSnapshot, + { breakdownTopN }, + ); + if (summary == null) throw new Error('No heap snapshot samples'); + return summary; +} + +export function summarizeSamples(label: 'base' | 'head', samples: BrowserMeasurementSample[], options: SummarizeOptions): BrowserMetricsReport { + if (samples.length === 0) throw new Error(`No browser metric samples for ${label}`); + const representative = selectRepresentativeSample(samples, sample => sample.network.totalEncodedBytes); + const summary: BrowserMeasurement = { + label, + timestamp: new Date().toISOString(), + url: options.url, + scenario: representative.scenario, + diagnostics: summarizeBrowserDiagnostics(samples.map(sample => sample.diagnostics)), + durationMs: finiteMedian(samples.map(sample => sample.durationMs), 0), + network: summarizeNetworkSamples(samples), + performance: summarizePerformanceSamples(samples), + heapSnapshot: summarizeHeapSnapshotSamples(samples, options.heapSnapshotBreakdownTopN), + }; + + return { + label, + timestamp: new Date().toISOString(), + url: options.url, + scenario: representative.scenario, + sampleCount: samples.length, + aggregation: 'median', + summary, + samples, + }; +} diff --git a/packages-private/diagnostics-frontend/src/browser/types.ts b/packages-private/diagnostics-frontend/src/browser/types.ts new file mode 100644 index 0000000000..78e31b76f4 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/browser/types.ts @@ -0,0 +1,144 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { HeapSnapshotData } from 'diagnostics-shared/heap-snapshot'; + +// NOTE: (AI向け) +// NetworkRequest がリクエスト本文と要求・応答ヘッダを保持し、APIキーやパスワード、Cookieなどの認証情報も含まれ得るが、 +// workflow上で作られるMisskeyアカウントはテスト用アカウントであり、誰かが実際に使用するプライベートなアカウントではないため、 +// それらの情報が第三者から閲覧できても何の問題もなく、脆弱性ではない。 + +export type NetworkRequest = { + requestId: string; + url: string; + method: string; + resourceType: string; + startedAt: number; + documentUrl?: string; + requestHeaders?: Record; + requestBody?: string; + hasRequestBody: boolean; + status?: number; + statusText?: string; + mimeType?: string; + responseHeaders?: Record; + protocol?: string; + remoteIPAddress?: string; + remotePort?: number; + encodedDataLength: number; + decodedBodyLength: number; + fromDiskCache: boolean; + fromServiceWorker: boolean; + finished: boolean; + failed: boolean; + errorText?: string; +}; + +export type WebSocketConnection = { + requestId: string; + url: string; + createdAt: number; + handshakeRequestHeaders?: Record; + handshakeResponseStatus?: number; + handshakeResponseStatusText?: string; + handshakeResponseHeaders?: Record; + closedAt?: number; + sentFrameCount: number; + receivedFrameCount: number; + sentBytes: number; + receivedBytes: number; + errorCount: number; +}; + +export type NetworkSummary = { + requestCount: number; + webSocketConnectionCount: number; + webSocketSentBytes: number; + webSocketReceivedBytes: number; + finishedRequestCount: number; + failedRequestCount: number; + cachedRequestCount: number; + serviceWorkerRequestCount: number; + totalEncodedBytes: number; + totalDecodedBodyBytes: number; + sameOriginEncodedBytes: number; + thirdPartyEncodedBytes: number; + byResourceType: Record; + largestRequests: { + url: string; + method: string; + resourceType: string; + status?: number; + encodedBytes: number; + decodedBodyBytes: number; + }[]; + failedRequests: { + url: string; + method: string; + resourceType: string; + errorText?: string; + status?: number; + }[]; +}; + +export type TabMemory = { + totalBytes: number; +}; + +export type BrowserDiagnostics = { + pageErrorCount: number; + console: Record<'log' | 'warning' | 'error' | 'info', number>; +}; + +export type BrowserMeasurement = { + label: string; + timestamp: string; + url: string; + scenario: string; + diagnostics: BrowserDiagnostics; + durationMs: number; + network: NetworkSummary; + performance: { + cdpMetrics: Record; + runtimeHeap?: { + usedSize: number; + totalSize: number; + }; + tabMemory: TabMemory; + webVitals: { + firstPaintMs?: number; + firstContentfulPaintMs?: number; + domContentLoadedEventEndMs?: number; + loadEventEndMs?: number; + longTaskCount: number; + longTaskDurationMs: number; + maxLongTaskDurationMs: number; + resourceEntryCount: number; + domElements: number; + }; + }; + heapSnapshot: HeapSnapshotData; +}; + +export type BrowserMeasurementSample = BrowserMeasurement & { + round: number; + /** ラウンド単位のリクエスト差分HTMLを描くために生ログを丸ごと保持する */ + networkRequests: NetworkRequest[]; +}; + +export type BrowserMetricsReport = { + label: string; + timestamp: string; + url: string; + scenario: string; + sampleCount: number; + aggregation: 'median'; + summary: BrowserMeasurement; + samples: BrowserMeasurementSample[]; +}; diff --git a/packages-private/diagnostics-frontend/src/bundle/chunk-report.ts b/packages-private/diagnostics-frontend/src/bundle/chunk-report.ts new file mode 100644 index 0000000000..50e2758328 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/bundle/chunk-report.ts @@ -0,0 +1,218 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { + calcAndFormatDeltaBytes, + calcAndFormatDeltaPercentInMdTable, + escapeMdTableCell, + formatBytes, +} from 'diagnostics-shared/format'; +import type { CollectedBundleReport, FileEntry } from './manifest'; + +/** + * この差分以下のチャンクは個別に出さず `(other)` にまとめる。 + * ハッシュ文字列の揺れ等でサイズが数バイト動くだけの行がノイズになるため。 + */ +const smallDeltaThreshold = 5; + +/** diff表に個別行として出す上限。これを超えた分は `(other)` に集約する */ +const diffRowLimit = 30; + +function entryDisplayName(entry: FileEntry | undefined) { + if (entry == null) return ''; + return entry.displayName || entry.file; +} + +export function getChunkComparisonRows(keys: string[], base: Partial>, head: Partial>) { + return keys.map(key => { + const baseEntry = base[key]; + const headEntry = head[key]; + const baseSize = baseEntry?.size ?? 0; + const headSize = headEntry?.size ?? 0; + return { + key, + name: entryDisplayName(baseEntry ?? headEntry), + baseFile: baseEntry?.file, + headFile: headEntry?.file, + baseSize, + headSize, + changeType: baseEntry == null ? 'added' : headEntry == null ? 'removed' : baseSize !== headSize ? 'updated' : 'unchanged', + sortSize: Math.max(baseSize, headSize), + }; + }); +} + +export type ChunkComparisonRow = ReturnType[number]; + +export type ChunkAggregate = { + baseSize: number; + headSize: number; + baseCount: number; + headCount: number; +}; + +export function sumChunkSizes(chunks: FileEntry[]) { + return chunks.reduce((sum, chunk) => sum + chunk.size, 0); +} + +/** + * 比較キーを持たない (= base/head で対応付けできない) チャンクの合計。 + */ +export function generatedAggregate(base: FileEntry[], head: FileEntry[]): ChunkAggregate { + const baseGenerated = base.filter(chunk => chunk.comparisonKey == null); + const headGenerated = head.filter(chunk => chunk.comparisonKey == null); + return { + baseSize: sumChunkSizes(baseGenerated), + headSize: sumChunkSizes(headGenerated), + baseCount: baseGenerated.length, + headCount: headGenerated.length, + }; +} + +export function hasSmallDelta(row: ChunkComparisonRow) { + return Math.abs(row.headSize - row.baseSize) <= smallDeltaThreshold; +} + +export function comparisonRowsAggregate(rows: ChunkComparisonRow[]): ChunkAggregate { + return { + baseSize: rows.reduce((sum, row) => sum + row.baseSize, 0), + headSize: rows.reduce((sum, row) => sum + row.headSize, 0), + baseCount: rows.filter(row => row.baseFile != null).length, + headCount: rows.filter(row => row.headFile != null).length, + }; +} + +export function comparableMap(chunks: FileEntry[]) { + const entries: [string, FileEntry][] = []; + for (const chunk of chunks) { + if (chunk.comparisonKey != null) entries.push([chunk.comparisonKey, chunk]); + } + return Object.fromEntries(entries); +} + +export function summarizeChunkChanges(rows: ChunkComparisonRow[]) { + return { + updated: rows.filter((row) => row.changeType === 'updated').length, + added: rows.filter((row) => row.changeType === 'added').length, + removed: rows.filter((row) => row.changeType === 'removed').length, + }; +} + +export function formatChunkChangeSummary(label: string, summary: ReturnType) { + return `${label} (${summary.updated} updated, ${summary.added} added, ${summary.removed} removed)`; +} + +/** + * 差分の絶対値が大きい順。同着は増加側・元サイズ・名前の順で決定的に並べる。 + */ +export function compareChunkComparisonRows(a: ChunkComparisonRow, b: ChunkComparisonRow) { + return Math.abs(b.headSize - b.baseSize) - Math.abs(a.headSize - a.baseSize) + || (b.headSize - b.baseSize) - (a.headSize - a.baseSize) + || b.sortSize - a.sortSize + || a.name.localeCompare(b.name); +} + +export function chunkFileDisplay(row: ChunkComparisonRow) { + if (row.baseFile == null) return row.headFile ?? ''; + if (row.headFile == null || row.baseFile === row.headFile) return row.baseFile; + return `${row.baseFile} → ${row.headFile}`; +} + +export function chunkMarkdownTable( + rows: ChunkComparisonRow[], + total?: { baseSize: number; headSize: number }, + generated?: ChunkAggregate, + other?: ChunkAggregate, +) { + const hasGenerated = generated != null && (generated.baseCount > 0 || generated.headCount > 0); + const hasOther = other != null && (other.baseCount > 0 || other.headCount > 0); + if (rows.length === 0 && total == null && !hasGenerated && !hasOther) return '_No data_'; + + const lines = [ + '| Chunk | Base | Head | Δ | Δ (%) |', // nbspにすること + '| --- | ---: | ---: | ---: | ---: |', + ]; + if (total != null) { + lines.push(`| (total) | ${formatBytes(total.baseSize)} | ${formatBytes(total.headSize)} | ${calcAndFormatDeltaBytes(total.baseSize, total.headSize, 1000)} | ${calcAndFormatDeltaPercentInMdTable(total.baseSize, total.headSize, 0.1)} |`); + lines.push('| | | | | |'); + } + + for (const row of rows) { + const chunkFile = chunkFileDisplay(row); + if (row.changeType === 'added') { + lines.push(`|
\`${escapeMdTableCell(row.name)}\` \`${escapeMdTableCell(chunkFile)}\`
| ${formatBytes(row.baseSize)} | ${formatBytes(row.headSize)} | ${calcAndFormatDeltaBytes(row.baseSize, row.headSize, 1000)} | $\\color{orange}{\\text{( + )}}$ |`); + } else if (row.changeType === 'removed') { + lines.push(`|
\`${escapeMdTableCell(row.name)}\` \`${escapeMdTableCell(chunkFile)}\`
| ${formatBytes(row.baseSize)} | ${formatBytes(row.headSize)} | ${calcAndFormatDeltaBytes(row.baseSize, row.headSize, 1000)} | $\\color{green}{\\text{( - )}}$ |`); + } else { + lines.push(`|
\`${escapeMdTableCell(row.name)}\` \`${escapeMdTableCell(chunkFile)}\`
| ${formatBytes(row.baseSize)} | ${formatBytes(row.headSize)} | ${calcAndFormatDeltaBytes(row.baseSize, row.headSize, 1000)} | ${calcAndFormatDeltaPercentInMdTable(row.baseSize, row.headSize, 0.1)} |`); + } + } + if (hasGenerated) { + // eslint-disable-next-line no-irregular-whitespace + lines.push(`| (other generated chunks) | ${formatBytes(generated.baseSize)} | ${formatBytes(generated.headSize)} | ${calcAndFormatDeltaBytes(generated.baseSize, generated.headSize, 1000)} | ${calcAndFormatDeltaPercentInMdTable(generated.baseSize, generated.headSize, 0.1)} |`); // nbspにすること + } + if (hasOther) { + lines.push(`| (other) | ${formatBytes(other.baseSize)} | ${formatBytes(other.headSize)} | ${calcAndFormatDeltaBytes(other.baseSize, other.headSize, 1000)} | ${calcAndFormatDeltaPercentInMdTable(other.baseSize, other.headSize, 0.1)} |`); + } + return lines.join('\n'); +} + +export function renderFrontendChunkReport(base: CollectedBundleReport, head: CollectedBundleReport) { + const baseComparable = base.comparableChunks; + const headComparable = head.comparableChunks; + const allChunkKeys = [...new Set([...Object.keys(baseComparable), ...Object.keys(headComparable)])]; + const allComparisonRows = getChunkComparisonRows(allChunkKeys, baseComparable, headComparable); + + const changedRows = allComparisonRows.filter((row) => row.changeType !== 'unchanged'); + const diffSummary = summarizeChunkChanges(changedRows); + const diffTotal = { + baseSize: sumChunkSizes(base.chunks), + headSize: sumChunkSizes(head.chunks), + }; + const diffGenerated = generatedAggregate(base.chunks, head.chunks); + const largeDeltaRows = changedRows.filter(row => !hasSmallDelta(row)).sort(compareChunkComparisonRows); + const diffRows = largeDeltaRows.slice(0, diffRowLimit); + // 表示上限で切り捨てた行も `(other)` に含める。落とすと合計が実際の変化量と合わなくなる + const diffOther = comparisonRowsAggregate([ + ...changedRows.filter(hasSmallDelta), + ...largeDeltaRows.slice(diffRowLimit), + ]); + + const baseStartupFiles = new Set(base.startupFiles); + const headStartupFiles = new Set(head.startupFiles); + const baseStartupChunks = base.chunks.filter(chunk => baseStartupFiles.has(chunk.file)); + const headStartupChunks = head.chunks.filter(chunk => headStartupFiles.has(chunk.file)); + const baseStartupComparable = comparableMap(baseStartupChunks); + const headStartupComparable = comparableMap(headStartupChunks); + const startupKeys = [...new Set([...Object.keys(baseStartupComparable), ...Object.keys(headStartupComparable)])]; + const startupComparisonRows = getChunkComparisonRows(startupKeys, baseStartupComparable, headStartupComparable); + const startupSummary = summarizeChunkChanges(startupComparisonRows); + const startupOther = comparisonRowsAggregate(startupComparisonRows.filter(hasSmallDelta)); + const startupRows = startupComparisonRows.filter(row => !hasSmallDelta(row)).sort(compareChunkComparisonRows); + const startupTotal = { + baseSize: sumChunkSizes(baseStartupChunks), + headSize: sumChunkSizes(headStartupChunks), + }; + const startupGenerated = generatedAggregate(baseStartupChunks, headStartupChunks); + + return [ + '
', + `${formatChunkChangeSummary('Chunk size diff', diffSummary)}`, + '', + chunkMarkdownTable(diffRows, diffTotal, diffGenerated, diffOther), + '', + '
', + '', + '
', + `${formatChunkChangeSummary('Startup chunk size', startupSummary)}`, + '', + chunkMarkdownTable(startupRows, startupTotal, startupGenerated, startupOther), + '', + '_Startup chunks are the Vite entry for \`src/_boot_.ts\` and its static imports._', + '', + '
', + '', + ].join('\n'); +} diff --git a/packages-private/diagnostics-frontend/src/bundle/fs-utils.ts b/packages-private/diagnostics-frontend/src/bundle/fs-utils.ts new file mode 100644 index 0000000000..c6b1d3d2e6 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/bundle/fs-utils.ts @@ -0,0 +1,41 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { promises as fs } from 'node:fs'; +import path from 'node:path'; + +/** + * Windows の `\` 区切りを `/` に揃える。manifest 側のキーが常に `/` 区切りのため、 + * 実ファイルパスと突き合わせる前に正規化する必要がある。 + */ +export function normalizePath(filePath: string) { + return filePath.split(path.sep).join('/'); +} + +export async function fileExists(filePath: string) { + try { + await fs.access(filePath); + return true; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return false; + throw error; + } +} + +export async function fileSize(filePath: string) { + const stat = await fs.stat(filePath); + return stat.size; +} + +export async function* traverseDirectory(dir: string): AsyncGenerator { + for (const entry of await fs.readdir(dir, { withFileTypes: true })) { + const fullPath = path.join(dir, entry.name); + if (entry.isDirectory()) { + yield* traverseDirectory(fullPath); + } else if (entry.isFile()) { + yield fullPath; + } + } +} diff --git a/packages-private/diagnostics-frontend/src/bundle/manifest.ts b/packages-private/diagnostics-frontend/src/bundle/manifest.ts new file mode 100644 index 0000000000..05b00c2a09 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/bundle/manifest.ts @@ -0,0 +1,180 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { promises as fs } from 'node:fs'; +import path from 'node:path'; +import { fileExists, fileSize, normalizePath, traverseDirectory } from './fs-utils'; + +export type BundleManifestChunk = { + file: string; + src?: string; + name?: string; + isEntry?: boolean; + imports?: string[]; +}; + +export type BundleManifest = Record; + +/** + * 比較対象とするロケール。ロケール別チャンクは全ロケール分だと数が多すぎるため、 + * 代表として ja-JP のみを見る。 + */ +const locale = 'ja-JP'; + +/** + * `src` を持たないチャンクのうち、名前がビルド間で安定していて比較可能なもの。 + */ +const stableNamedChunks = new Set(['vue', 'i18n']); + +export type FileEntry = { + comparisonKey: string | null; + displayName: string; + file: string; + manifestKeys: string[]; + size: number; +}; + +export type CollectedBundleReport = { + manifest: BundleManifest; + chunks: FileEntry[]; + comparableChunks: Record; + chunksByManifestKey: Record; + startupFiles: string[]; +}; + +export function findEntryKey(manifest: BundleManifest) { + const entries = Object.entries(manifest); + return entries.find(([key, chunk]) => key === 'src/_boot_.ts' || chunk.src === 'src/_boot_.ts')?.[0] + ?? entries.find(([, chunk]) => chunk.name === 'entry' && chunk.isEntry)?.[0] + ?? entries.find(([, chunk]) => chunk.isEntry)?.[0] + ?? null; +} + +/** + * ビルド間で安定するチャンク識別子。出力ファイル名はハッシュ付きで毎回変わるため、 + * これが取れないチャンクは base/head の対応付けができない。 + */ +export function stableChunkKey(chunk: BundleManifestChunk) { + if (chunk.src != null) return `src:${normalizePath(chunk.src)}`; + if (chunk.name != null && stableNamedChunks.has(chunk.name)) return `named:${chunk.name}`; + return null; +} + +/** + * 起動時に必ず読み込まれるチャンク (entry とその静的 import) の manifest キーを集める。 + */ +export function collectStartupManifestKeys(manifest: BundleManifest) { + const entryKey = findEntryKey(manifest); + const keys = new Set(); + if (entryKey == null) throw new Error('Unable to find frontend startup entry in Vite manifest.'); + + function visit(key: string, importedBy?: string) { + if (keys.has(key)) return; + const chunk = manifest[key]; + const importContext = importedBy == null ? '' : ` imported by "${importedBy}"`; + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (chunk == null) throw new Error(`Startup manifest key "${key}"${importContext} is missing.`); + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (chunk.file == null || chunk.file.length === 0) throw new Error(`Startup manifest key "${key}"${importContext} has no output file.`); + if (!chunk.file.endsWith('.js')) throw new Error(`Startup manifest key "${key}"${importContext} resolves to non-JavaScript output "${chunk.file}".`); + keys.add(key); + for (const importKey of chunk.imports ?? []) visit(importKey, key); + } + + visit(entryKey); + return keys; +} + +/** + * manifest 上の出力パスを実ファイルへ解決する。`scripts/` 配下はロケール別に + * 複製されて出力されるため、代表ロケールのものへ読み替える。 + */ +export async function resolveBuiltFile(outDir: string, file: string) { + if (file.startsWith('scripts/')) { + const localizedFile = file.slice('scripts/'.length); + const localizedPath = path.join(outDir, locale, localizedFile); + if (await fileExists(localizedPath)) { + return { + absolutePath: localizedPath, + relativePath: `${locale}/${localizedFile}`, + }; + } + + throw new Error(`Expected ${locale} localized chunk for ${file}, but ${localizedPath} was not found.`); + } + return { + absolutePath: path.join(outDir, file), + relativePath: file, + }; +} + +export async function collectBundleReport(repoDir: string): Promise { + const outDir = path.join(repoDir, 'built/_frontend_vite_'); + const manifestPath = path.join(outDir, 'manifest.json'); + const manifest = JSON.parse(await fs.readFile(manifestPath, 'utf8')) as BundleManifest; + const chunksByFile = new Map(); + const comparableChunks = new Map(); + const chunksByManifestKey = new Map(); + + for (const [manifestKey, chunk] of Object.entries(manifest)) { + if (!chunk.file.endsWith('.js')) continue; + const builtFile = await resolveBuiltFile(outDir, chunk.file); + const comparisonKey = stableChunkKey(chunk); + let entry = chunksByFile.get(builtFile.relativePath); + if (entry == null) { + entry = { + comparisonKey, + displayName: chunk.src ?? chunk.name ?? manifestKey, + file: builtFile.relativePath, + manifestKeys: [manifestKey], + size: await fileSize(builtFile.absolutePath), + }; + chunksByFile.set(entry.file, entry); + } else if (entry.comparisonKey !== comparisonKey) { + throw new Error(`Conflicting identities for ${entry.file}`); + } else { + entry.manifestKeys.push(manifestKey); + } + chunksByManifestKey.set(manifestKey, entry); + if (comparisonKey != null) { + const existing = comparableChunks.get(comparisonKey); + if (existing != null && existing.file !== entry.file) { + throw new Error(`Duplicate stable chunk key "${comparisonKey}": ${existing.file}, ${entry.file}`); + } + comparableChunks.set(comparisonKey, entry); + } + } + + // manifest に載らないロケール別チャンクも合計サイズには含めたいので拾っておく + const localeDir = path.join(outDir, locale); + if (await fileExists(localeDir)) { + for await (const fullPath of traverseDirectory(localeDir)) { + if (!fullPath.endsWith('.js')) continue; + const relativePath = normalizePath(path.relative(outDir, fullPath)); + if (chunksByFile.has(relativePath)) continue; + chunksByFile.set(relativePath, { + comparisonKey: null, + displayName: relativePath, + file: relativePath, + manifestKeys: [], + size: await fileSize(fullPath), + }); + } + } + + const startupFiles = new Set(); + for (const manifestKey of collectStartupManifestKeys(manifest)) { + const entry = chunksByManifestKey.get(manifestKey); + if (entry != null) startupFiles.add(entry.file); + } + + return { + manifest, + chunks: [...chunksByFile.values()], + comparableChunks: Object.fromEntries(comparableChunks), + chunksByManifestKey: Object.fromEntries(chunksByManifestKey), + startupFiles: [...startupFiles], + }; +} diff --git a/packages-private/diagnostics-frontend/src/bundle/visualizer.ts b/packages-private/diagnostics-frontend/src/bundle/visualizer.ts new file mode 100644 index 0000000000..51f7264eed --- /dev/null +++ b/packages-private/diagnostics-frontend/src/bundle/visualizer.ts @@ -0,0 +1,204 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { + calcAndFormatDeltaBytes, + calcAndFormatDeltaNumber, + calcAndFormatDeltaPercent, + formatBytes, + formatNumber, +} from 'diagnostics-shared/format'; + +/** + * rollup-plugin-visualizer が出力する `stats.json` のうち、ここで使う部分のみの型。 + */ +export type VisualizerReport = { + nodeParts?: Record; + nodeMetas?: Record; + renderedLength: number; + gzipLength: number; + brotliLength: number; + }>; + options?: Record; +}; + +type ModuleRow = { + id: string; + bundles: number; + renderedLength: number; + gzipLength: number; + brotliLength: number; + importedByCount: number; + importedCount: number; +}; + +type BundleRow = { + id: string; + modules: number; + renderedLength: number; + gzipLength: number; + brotliLength: number; +}; + +export function collectVisualizerReport(data: VisualizerReport) { + const nodeParts = data.nodeParts ?? {}; + const nodeMetas = Object.values(data.nodeMetas ?? {}); + const moduleRows: ModuleRow[] = []; + const bundleMap = new Map(); + + for (const meta of nodeMetas) { + const row: ModuleRow = { + id: meta.id, + bundles: 0, + renderedLength: 0, + gzipLength: 0, + brotliLength: 0, + importedByCount: meta.importedBy?.length ?? 0, + importedCount: meta.imported?.length ?? 0, + }; + + for (const [bundleId, partUid] of Object.entries(meta.moduleParts ?? {})) { + const part = nodeParts[partUid]; + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (part == null) continue; + + row.bundles += 1; + row.renderedLength += part.renderedLength; + row.gzipLength += part.gzipLength; + row.brotliLength += part.brotliLength; + + const bundle = bundleMap.get(bundleId) ?? { + id: bundleId, + modules: 0, + renderedLength: 0, + gzipLength: 0, + brotliLength: 0, + }; + bundle.modules += 1; + bundle.renderedLength += part.renderedLength; + bundle.gzipLength += part.gzipLength; + bundle.brotliLength += part.brotliLength; + bundleMap.set(bundleId, bundle); + } + + // どのバンドルにも含まれないモジュール (tree-shake 済み等) は集計対象外 + if (row.bundles > 0) { + moduleRows.push(row); + } + } + + let staticImports = 0; + let dynamicImports = 0; + for (const meta of nodeMetas) { + for (const imported of meta.imported ?? []) { + if (imported.dynamic) { + dynamicImports += 1; + } else { + staticImports += 1; + } + } + } + + const bundleRows = [...bundleMap.values()].sort((a, b) => b.renderedLength - a.renderedLength); + const hotModules = [...moduleRows].sort((a, b) => b.renderedLength - a.renderedLength); + const totalRendered = moduleRows.reduce((sum, row) => sum + row.renderedLength, 0); + const totalGzip = moduleRows.reduce((sum, row) => sum + row.gzipLength, 0); + const totalBrotli = moduleRows.reduce((sum, row) => sum + row.brotliLength, 0); + + return { + options: data.options ?? {}, + summary: { + bundles: bundleRows.length, + modules: moduleRows.length, + entries: nodeMetas.filter((meta) => meta.isEntry).length, + externals: nodeMetas.filter((meta) => meta.isExternal).length, + staticImports, + dynamicImports, + }, + metrics: { + renderedLength: totalRendered, + gzipLength: totalGzip, + brotliLength: totalBrotli, + }, + hotModules, + }; +} + +/** + * NOTE: 以前はこの関数が `string[]` を返しており、呼び出し側の `[...].join('\n')` の + * 要素として配列のまま埋め込まれていたため、テーブルがカンマ区切りの1行に潰れていた。 + * 呼び出し側で意識しなくて済むよう、ここで文字列にして返す。 + */ +export function renderVisualizerSummaryTable(base: ReturnType, head: ReturnType) { + const summary = [ + 'bundles', + 'modules', + 'entries', + //'externals', + 'staticImports', + 'dynamicImports', + ] as const; + + const metrics = [ + 'renderedLength', + 'gzipLength', + 'brotliLength', + ] as const; + + return [ + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + ...summary.map((key) => ``), + ...metrics.map((key) => ``), + '', + '', + '', + ...summary.map((key) => ``), + ...metrics.map((key) => ``), + '', + '', + '', + '', + ...summary.map((key) => ``), + ...metrics.map((key) => ``), + '', + '', + '', + ...summary.map((key) => ``), + ...metrics.map((key) => ``), + '', + '', + '
BundlesModulesEntriesImportsSize
StaticDynamicRenderedGzipBrotli
Base${formatNumber(base.summary[key])}${formatBytes(base.metrics[key])}
Head${formatNumber(head.summary[key])}${formatBytes(head.metrics[key])}
Δ${calcAndFormatDeltaNumber(base.summary[key], head.summary[key], 0)}${calcAndFormatDeltaBytes(base.metrics[key], head.metrics[key], 1000)}
Δ (%)${calcAndFormatDeltaPercent(base.summary[key], head.summary[key], 0.1)}${calcAndFormatDeltaPercent(base.metrics[key], head.metrics[key], 0.1)}
', + ].join('\n'); +} diff --git a/packages-private/diagnostics-frontend/src/inspect-browser.ts b/packages-private/diagnostics-frontend/src/inspect-browser.ts new file mode 100644 index 0000000000..25d2703e99 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/inspect-browser.ts @@ -0,0 +1,128 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { copyFile, mkdir, rm, writeFile } from 'node:fs/promises'; +import { join, resolve } from 'node:path'; +import { readIntegerEnv, readOptionalEnv } from 'diagnostics-shared/env'; +import { analyzeHeapSnapshot, defaultHeapSnapshotBreakdownTopN } from 'diagnostics-shared/heap-snapshot'; +import { HeadlessChromeController } from './browser/controller'; +import { summarizeNetwork } from './browser/network'; +import { prepareInstance, runSignupAndPostScenario, scenarioDescription } from './browser/scenario'; +import { startServer, stopServer, waitForServer } from './browser/server'; +import { selectRepresentativeSample, summarizeSamples } from './browser/summarize'; +import type { BrowserMeasurementSample, BrowserMetricsReport } from './browser/types'; + +type Label = 'base' | 'head'; + +const labels = ['base', 'head'] as const satisfies readonly Label[]; + +const baseUrl = process.env.FRONTEND_BROWSER_METRICS_URL ?? 'http://127.0.0.1:61812'; +const sampleCount = readIntegerEnv('FRONTEND_BROWSER_METRICS_SAMPLE_COUNT', 5, 1); +const heapSnapshotBreakdownTopN = readIntegerEnv('FRONTEND_BROWSER_HEAP_SNAPSHOT_BREAKDOWN_TOP_N', defaultHeapSnapshotBreakdownTopN, 1); + +// 成果物 (artifact) としてアップロードされるファイルの出力先。CIではworkspace直下を指す +const heapSnapshotOutputDir = resolve(readOptionalEnv('FRONTEND_BROWSER_HEAP_SNAPSHOT_OUTPUT_DIR') ?? process.cwd()); +const heapSnapshotWorkDirs = { + base: join(heapSnapshotOutputDir, 'frontend-browser-base-heap-snapshots'), + head: join(heapSnapshotOutputDir, 'frontend-browser-head-heap-snapshots'), +} as const; +const heapSnapshotOutputPaths = { + base: join(heapSnapshotOutputDir, 'base-heap-snapshot.heapsnapshot'), + head: join(heapSnapshotOutputDir, 'head-heap-snapshot.heapsnapshot'), +} as const; + +function heapSnapshotPath(label: Label, round: number) { + return join(heapSnapshotWorkDirs[label], `round-${round}.heapsnapshot`); +} + +/** + * ブラウザを毎ラウンド立ち上げ直して1サンプル分を計測する。 + * ブラウザを使い回すとキャッシュや前ラウンドのGC残渣が乗るため、必ず作り直す。 + */ +async function measureSample(label: Label, round: number, heapSnapshotSavePath: string): Promise { + await prepareInstance(baseUrl); + + return await HeadlessChromeController.with(label, { scenarioTimeoutMs: 120000, baseUrl }, async chrome => { + await chrome.enableNetworkTracking(); + + const startedAt = Date.now(); + await runSignupAndPostScenario(chrome, baseUrl); + const durationMs = Date.now() - startedAt; + + await chrome.waitForNetworkDetails(); + const performance = await chrome.collectPerformance(); + const heapSnapshotRaw = await chrome.takeHeapSnapshot(heapSnapshotSavePath); + + return { + label, + round, + timestamp: new Date().toISOString(), + url: baseUrl, + scenario: scenarioDescription, + diagnostics: chrome.collectDiagnostics(), + durationMs, + network: summarizeNetwork(chrome.networkRequests, baseUrl, chrome.webSocketConnections), + networkRequests: chrome.networkRequests, + performance, + heapSnapshot: analyzeHeapSnapshot(heapSnapshotRaw, { breakdownTopN: heapSnapshotBreakdownTopN }), + }; + }); +} + +/** + * 中央値に最も近いラウンドのスナップショットだけを成果物として残し、残りは捨てる。 + */ +async function saveRepresentativeHeapSnapshot(label: Label, report: BrowserMetricsReport) { + const representative = selectRepresentativeSample(report.samples, sample => sample.heapSnapshot.categories.total); + await copyFile(heapSnapshotPath(label, representative.round), heapSnapshotOutputPaths[label]); + process.stderr.write(`[${label}] Selected round ${representative.round} heap snapshot for artifact\n`); + await rm(heapSnapshotWorkDirs[label], { recursive: true, force: true }); +} + +async function genReport(label: Label, repoDir: string, outputPath: string) { + let server: ReturnType | null = null; + + try { + server = startServer(label, repoDir); + await waitForServer(baseUrl, server); + + await rm(heapSnapshotWorkDirs[label], { recursive: true, force: true }); + await mkdir(heapSnapshotWorkDirs[label], { recursive: true }); + + const samples: BrowserMeasurementSample[] = []; + for (let round = 1; round <= sampleCount; round++) { + process.stderr.write(`[${label}] Measuring browser metrics sample ${round}/${sampleCount}\n`); + samples.push(await measureSample(label, round, heapSnapshotPath(label, round))); + } + + const report = summarizeSamples(label, samples, { url: baseUrl, heapSnapshotBreakdownTopN }); + await writeFile(outputPath, JSON.stringify(report, null, '\t')); + process.stderr.write(`[${label}] Wrote browser metrics report to ${outputPath}\n`); + + await saveRepresentativeHeapSnapshot(label, report); + } finally { + if (server != null) await stopServer(server); + } +} + +async function main() { + const [baseDirArg, headDirArg, baseOutputArg, headOutputArg] = process.argv.slice(2); + if (baseDirArg == null || headDirArg == null || baseOutputArg == null || headOutputArg == null) { + throw new Error('Usage: inspect-browser '); + } + + for (const label of labels) { + await rm(heapSnapshotOutputPaths[label], { force: true }); + } + + // base / head は同じポートでサーバーを立てるため、並行させず順番に計測する + await genReport('base', resolve(baseDirArg), resolve(baseOutputArg)); + await genReport('head', resolve(headDirArg), resolve(headOutputArg)); +} + +await main().catch(err => { + console.error(err); + process.exit(1); +}); diff --git a/packages-private/diagnostics-frontend/src/render-browser-html.ts b/packages-private/diagnostics-frontend/src/render-browser-html.ts new file mode 100644 index 0000000000..c6567e088e --- /dev/null +++ b/packages-private/diagnostics-frontend/src/render-browser-html.ts @@ -0,0 +1,26 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { readFile, writeFile } from 'node:fs/promises'; +import { resolve } from 'node:path'; +import { renderBrowserDiagnosticsHtml } from './browser/report/html'; +import type { BrowserMetricsReport } from './browser/types'; + +async function main() { + const [baseFileArg, headFileArg, outputFileArg] = process.argv.slice(2); + if (baseFileArg == null || headFileArg == null || outputFileArg == null) { + throw new Error('Usage: render-browser-html '); + } + + const base = JSON.parse(await readFile(resolve(baseFileArg), 'utf8')) as BrowserMetricsReport; + const head = JSON.parse(await readFile(resolve(headFileArg), 'utf8')) as BrowserMetricsReport; + + await writeFile(resolve(outputFileArg), renderBrowserDiagnosticsHtml(base, head)); +} + +await main().catch(err => { + console.error(err); + process.exit(1); +}); diff --git a/packages-private/diagnostics-frontend/src/render-md.ts b/packages-private/diagnostics-frontend/src/render-md.ts new file mode 100644 index 0000000000..a9cfbc1cd1 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/render-md.ts @@ -0,0 +1,62 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { readFile, writeFile } from 'node:fs/promises'; +import { resolve } from 'node:path'; +import { readOptionalEnv, readRequiredEnv } from 'diagnostics-shared/env'; +import { collectBundleReport } from './bundle/manifest'; +import { renderFrontendDiagnosticsMarkdown } from './report'; +import type { BrowserMetricsReport } from './browser/types'; +import type { VisualizerReport } from './bundle/visualizer'; + +const args = process.argv.slice(2); +if (args.length !== 7) { + throw new Error('Usage: render-md '); +} +const [ + baseDirArg, + headDirArg, + baseBundleStatsFileArg, + headBundleStatsFileArg, + baseBrowserFileArg, + headBrowserFileArg, + outputFileArg, +] = args as [string, string, string, string, string, string, string]; + +const [ + baseBundle, + headBundle, + baseBundleStatsJson, + headBundleStatsJson, + baseBrowserJson, + headBrowserJson, +] = await Promise.all([ + collectBundleReport(resolve(baseDirArg)), + collectBundleReport(resolve(headDirArg)), + readFile(resolve(baseBundleStatsFileArg), 'utf8'), + readFile(resolve(headBundleStatsFileArg), 'utf8'), + readFile(resolve(baseBrowserFileArg), 'utf8'), + readFile(resolve(headBrowserFileArg), 'utf8'), +]); + +await writeFile( + resolve(outputFileArg), + renderFrontendDiagnosticsMarkdown({ + bundle: { + base: baseBundle, + head: headBundle, + baseStats: JSON.parse(baseBundleStatsJson) as VisualizerReport, + headStats: JSON.parse(headBundleStatsJson) as VisualizerReport, + visualizerArtifactUrl: readRequiredEnv('FRONTEND_BUNDLE_REPORT_ARTIFACT_URL'), + }, + browser: { + base: JSON.parse(baseBrowserJson) as BrowserMetricsReport, + head: JSON.parse(headBrowserJson) as BrowserMetricsReport, + baseHeapSnapshotUrl: readRequiredEnv('FRONTEND_BROWSER_BASE_HEAP_SNAPSHOT_ARTIFACT_URL'), + headHeapSnapshotUrl: readRequiredEnv('FRONTEND_BROWSER_HEAD_HEAP_SNAPSHOT_ARTIFACT_URL'), + detailedHtmlUrl: readOptionalEnv('FRONTEND_BROWSER_DETAILED_HTML_ARTIFACT_URL'), + }, + }), +); diff --git a/packages-private/diagnostics-frontend/src/report.ts b/packages-private/diagnostics-frontend/src/report.ts new file mode 100644 index 0000000000..c61985f405 --- /dev/null +++ b/packages-private/diagnostics-frontend/src/report.ts @@ -0,0 +1,237 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { formatBytes, formatColoredDelta, formatNumber } from 'diagnostics-shared/format'; +import { renderHeapSnapshotTable, type HeapSnapshotReport } from 'diagnostics-shared/heap-snapshot'; +import { renderMetricComparisonTable } from 'diagnostics-shared/metric-table'; +import { renderFrontendChunkReport } from './bundle/chunk-report'; +import { collectVisualizerReport, renderVisualizerSummaryTable, type VisualizerReport } from './bundle/visualizer'; +import type { CollectedBundleReport } from './bundle/manifest'; +import type { BrowserMeasurementSample, BrowserMetricsReport } from './browser/types'; + +export type FrontendDiagnosticsMarkdownInput = { + bundle: { + base: CollectedBundleReport; + head: CollectedBundleReport; + baseStats: VisualizerReport; + headStats: VisualizerReport; + /** rollup-plugin-visualizer が出力したtreemap HTMLのartifact URL */ + visualizerArtifactUrl: string; + }; + browser: { + base: BrowserMetricsReport; + head: BrowserMetricsReport; + baseHeapSnapshotUrl: string; + headHeapSnapshotUrl: string; + detailedHtmlUrl?: string | null; + }; +}; + +function resourceTypeSampleBytes(sample: BrowserMeasurementSample, resourceTypes: string[]) { + return resourceTypes.reduce((sum, resourceType) => sum + (sample.network.byResourceType[resourceType]?.encodedBytes ?? 0), 0); +} + +function renderBrowserSummaryTable(base: BrowserMetricsReport, head: BrowserMetricsReport) { + return renderMetricComparisonTable( + base.samples, + head.samples, + [{ + label: '**Requests**', + getValue: sample => sample.network.requestCount, + formatValue: formatNumber, + absoluteThreshold: 1, + }, { + label: '**Encoded network**', + getValue: sample => sample.network.totalEncodedBytes, + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**Decoded body**', + getValue: sample => sample.network.totalDecodedBodyBytes, + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**Same-origin encoded**', + getValue: sample => sample.network.sameOriginEncodedBytes, + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**Third-party encoded**', + getValue: sample => sample.network.thirdPartyEncodedBytes, + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**Script encoded**', + getValue: sample => resourceTypeSampleBytes(sample, ['Script']), + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**Stylesheet encoded**', + getValue: sample => resourceTypeSampleBytes(sample, ['Stylesheet']), + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**Fetch/XHR encoded**', + getValue: sample => resourceTypeSampleBytes(sample, ['Fetch', 'XHR']), + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**Image encoded**', + getValue: sample => resourceTypeSampleBytes(sample, ['Image']), + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**Font encoded**', + getValue: sample => resourceTypeSampleBytes(sample, ['Font']), + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**WebSocket connections**', + getValue: sample => sample.network.webSocketConnectionCount, + formatValue: formatNumber, + absoluteThreshold: 1, + }, { + label: '**WebSocket sent**', + getValue: sample => sample.network.webSocketSentBytes, + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**WebSocket received**', + getValue: sample => sample.network.webSocketReceivedBytes, + formatValue: formatBytes, + absoluteThreshold: 10_000, + }, { + label: '**Page errors**', + getValue: sample => sample.diagnostics.pageErrorCount, + formatValue: formatNumber, + absoluteThreshold: 1, + }, { + label: '**Console log**', + getValue: sample => sample.diagnostics.console.log, + formatValue: formatNumber, + absoluteThreshold: 1, + }, { + label: '**Console warnings**', + getValue: sample => sample.diagnostics.console.warning, + formatValue: formatNumber, + absoluteThreshold: 1, + }, { + label: '**Console errors**', + getValue: sample => sample.diagnostics.console.error, + formatValue: formatNumber, + absoluteThreshold: 1, + }, { + label: '**Console info**', + getValue: sample => sample.diagnostics.console.info, + formatValue: formatNumber, + absoluteThreshold: 1, + }, { + label: '**Page-attributed memory**', + getValue: sample => sample.performance.tabMemory.totalBytes, + formatValue: formatBytes, + absoluteThreshold: 10_000, + }], + { onlySignificantChanges: true }, + ); +} + +function renderResourceTypeTable(base: BrowserMetricsReport, head: BrowserMetricsReport) { + const preferredOrder = ['Document', 'Script', 'Stylesheet', 'Fetch', 'XHR', 'Image', 'Font', 'Media', 'WebSocket', 'EventSource', 'Other']; + const keys = [...new Set([ + ...preferredOrder, + ...Object.keys(base.summary.network.byResourceType), + ...Object.keys(head.summary.network.byResourceType), + ])].filter(key => base.summary.network.byResourceType[key] != null || head.summary.network.byResourceType[key] != null); + + const lines = [ + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + '', + ]; + + for (const key of keys) { + const baseRow = base.summary.network.byResourceType[key] ?? { requests: 0, encodedBytes: 0 }; + const headRow = head.summary.network.byResourceType[key] ?? { requests: 0, encodedBytes: 0 }; + lines.push(''); + lines.push(``); + lines.push(``); + lines.push(``); + lines.push(``); + lines.push(``); + lines.push(``); + lines.push(``); + lines.push(''); + } + + lines.push(''); + lines.push('
TypeRequestsEncoded bytes
BaseHeadΔBaseHeadΔ
${key}${formatNumber(baseRow.requests)}${formatNumber(headRow.requests)}${formatColoredDelta(headRow.requests - baseRow.requests, formatNumber)}${formatBytes(baseRow.encodedBytes)}${formatBytes(headRow.encodedBytes)}${formatColoredDelta(headRow.encodedBytes - baseRow.encodedBytes, formatBytes)}
'); + + return lines.join('\n'); +} + +function toHeapSnapshotReport(report: BrowserMetricsReport): HeapSnapshotReport { + return { + summary: report.summary.heapSnapshot, + samples: report.samples.map(sample => ({ + round: sample.round, + data: sample.heapSnapshot, + })), + }; +} + +export function renderFrontendDiagnosticsMarkdown(input: FrontendDiagnosticsMarkdownInput) { + const { bundle, browser } = input; + const detailedHtmlUrl = browser.detailedHtmlUrl; + const heapSnapshotTable = renderHeapSnapshotTable(toHeapSnapshotReport(browser.base), toHeapSnapshotReport(browser.head)); + const lines = [ + '## 🖥 Frontend Diagnostics Report', + '', + renderBrowserSummaryTable(browser.base, browser.head), + '', + detailedHtmlUrl == null || detailedHtmlUrl === '' ? null : `[View details](${detailedHtmlUrl})`, + detailedHtmlUrl == null || detailedHtmlUrl === '' ? null : '', + '
', + 'Requests by resource type', + '', + renderResourceTypeTable(browser.base, browser.head), + '', + '
', + '', + '
', + 'V8 heap snapshot statistics', + '', + heapSnapshotTable ?? '_No V8 heap snapshot data._', + '', + //renderHeapSnapshotSankey(toHeapSnapshotReport(browser.head), 'Head'), + //'', + `Download representative heap snapshot: [base](${browser.baseHeapSnapshotUrl}) / [head](${browser.headHeapSnapshotUrl})`, + '
', + '', + '## 📦 Bundle Stats', + '', + renderFrontendChunkReport(bundle.base, bundle.head), + '', + renderVisualizerSummaryTable(collectVisualizerReport(bundle.baseStats), collectVisualizerReport(bundle.headStats)), + '', + `[Open treemap HTML](${bundle.visualizerArtifactUrl})`, + '', + ]; + + return lines.filter(line => line != null).join('\n').trimEnd() + '\n'; +} diff --git a/packages-private/diagnostics-frontend/test/__snapshots__/report.md b/packages-private/diagnostics-frontend/test/__snapshots__/report.md new file mode 100644 index 0000000000..84a5a3e90f --- /dev/null +++ b/packages-private/diagnostics-frontend/test/__snapshots__/report.md @@ -0,0 +1,182 @@ +## 🖥 Frontend Diagnostics Report + +| Metric | @ Base | @ Head | Δ | MAD | +| --- | ---: | ---: | ---: | ---: | +| **Encoded network** | 1 MB
± 10 KB | 1.1 MB
± 11 KB | $\color{orange}{\text{+83 KB}}$
$\color{orange}{\text{+8\\%}}$ | 15 KB | +| **Decoded body** | 3.5 MB
± 34 KB | 3.7 MB
± 37 KB | $\color{orange}{\text{+277 KB}}$
$\color{orange}{\text{+8\\%}}$ | 50 KB | +| **Same-origin encoded** | 1 MB
± 10 KB | 1.1 MB
± 11 KB | $\color{orange}{\text{+82 KB}}$
$\color{orange}{\text{+8\\%}}$ | 15 KB | +| **Script encoded** | 918 KB
± 9 KB | 991 KB
± 9.7 KB | $\color{orange}{\text{+73 KB}}$
$\color{orange}{\text{+8\\%}}$ | 13 KB | +| **Page-attributed memory** | 92 MB
± 900 KB | 99 MB
± 972 KB | $\color{orange}{\text{+7.3 MB}}$
$\color{orange}{\text{+8\\%}}$ | 1.3 MB | + +Only metrics showing significant changes are displayed. + +[View details](https://example.invalid/html) + +
+Requests by resource type + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeRequestsEncoded bytes
BaseHeadΔBaseHeadΔ
Script10100918 KB991 KB$\color{orange}{\text{+73 KB}}$
Stylesheet22082 KB88 KB$\color{orange}{\text{+6.5 KB}}$
Fetch66041 KB44 KB$\color{orange}{\text{+3.3 KB}}$
+ +
+ +
+V8 heap snapshot statistics + +| Metric | @ Base | @ Head | Δ | MAD | +| --- | ---: | ---: | ---: | ---: | +| $\color{gray}{\rule{8pt}{8pt}}$ **Total** | 1 MB
± 10 KB | 1.1 MB
± 11 KB | $\text{+82 KB}$
$\text{+8\\%}$ | 15 KB | +| | | | | | +| $\color{orange}{\rule{8pt}{8pt}}$ **Code** | 2 MB | 2.2 MB | $\color{orange}{\text{+163 KB}}$ | 29 KB | +| $\color{red}{\rule{8pt}{8pt}}$ **Strings** | 3.1 MB | 3.3 MB | $\color{orange}{\text{+245 KB}}$ | 44 KB | +| $\color{cyan}{\rule{8pt}{8pt}}$ **JS arrays** | 4.1 MB | 4.4 MB | $\color{orange}{\text{+326 KB}}$ | 59 KB | +| $\color{green}{\rule{8pt}{8pt}}$ **Typed arrays** | 5.1 MB | 5.5 MB | $\color{orange}{\text{+408 KB}}$ | 74 KB | +| $\color{yellow}{\rule{8pt}{8pt}}$ **System objects** | 6.1 MB | 6.6 MB | $\color{orange}{\text{+490 KB}}$ | 88 KB | +| $\color{violet}{\rule{8pt}{8pt}}$ **Other JS objs** | 7.1 MB | 7.7 MB | $\color{orange}{\text{+571 KB}}$ | 103 KB | +| $\color{pink}{\rule{8pt}{8pt}}$ **Other non-JS objs** | 8.2 MB | 8.8 MB | $\color{orange}{\text{+653 KB}}$ | 118 KB | + +Download representative heap snapshot: [base](https://example.invalid/base) / [head](https://example.invalid/head) +
+ +## 📦 Bundle Stats + +
+Chunk size diff (2 updated, 0 added, 0 removed) + +| Chunk | Base | Head | Δ | Δ (%) | +| --- | ---: | ---: | ---: | ---: | +| (total) | 120 KB | 127 KB | $\color{orange}{\text{+6.3 KB}}$ | $\color{orange}{\text{+5.2\\%}}$ | +| | | | | | +|
`vue` `assets/vue-b2.js`
| 90 KB | 96 KB | $\color{orange}{\text{+6 KB}}$ | $\color{orange}{\text{+6.7\\%}}$ | +| (other generated chunks) | 1.2 KB | 1.5 KB | $\text{+300 B}$ | $\color{orange}{\text{+25\\%}}$ | +| (other) | 20 KB | 20 KB | $\text{+3 B}$ | $\text{+0\\%}$ | + +
+ +
+Startup chunk size (2 updated, 0 added, 0 removed) + +| Chunk | Base | Head | Δ | Δ (%) | +| --- | ---: | ---: | ---: | ---: | +| (total) | 114 KB | 120 KB | $\color{orange}{\text{+6 KB}}$ | $\color{orange}{\text{+5.3\\%}}$ | +| | | | | | +|
`vue` `assets/vue-b2.js`
| 90 KB | 96 KB | $\color{orange}{\text{+6 KB}}$ | $\color{orange}{\text{+6.7\\%}}$ | +| (other) | 24 KB | 24 KB | $\text{+3 B}$ | $\text{+0\\%}$ | + +_Startup chunks are the Vite entry for `src/_boot_.ts` and its static imports._ + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
BundlesModulesEntriesImportsSize
StaticDynamicRenderedGzipBrotli
Base2612321 KB6.3 KB5.3 KB
Head2713331 KB8.4 KB7 KB
Δ0$\color{orange}{\text{+1}}$0$\color{orange}{\text{+1}}$0$\color{orange}{\text{+9.8 KB}}$$\color{orange}{\text{+2.1 KB}}$$\color{orange}{\text{+1.8 KB}}$
Δ (%)0%$\color{orange}{\text{+16.7\%}}$0%$\color{orange}{\text{+50\%}}$0%$\color{orange}{\text{+46.7\%}}$$\color{orange}{\text{+33.3\%}}$$\color{orange}{\text{+33.3\%}}$
+ +[Open treemap HTML](https://example.invalid/treemap) diff --git a/packages-private/diagnostics-frontend/test/browser/__snapshots__/render-html.html b/packages-private/diagnostics-frontend/test/browser/__snapshots__/render-html.html new file mode 100644 index 0000000000..4a44b7cafb --- /dev/null +++ b/packages-private/diagnostics-frontend/test/browser/__snapshots__/render-html.html @@ -0,0 +1,351 @@ + + + + + + Frontend Browser Network Request Diff + + + +
+

Frontend Browser Network Request Diff

+

Generated at 2026-07-18T00:00:00.000Z. Requests are compared per paired round by method, resource type, and exact URL. Bodies are shown for added/removed request instances when CDP exposes them.

+ +
+
+ Base samples + 3 +
+
+ Head samples + 3 +
+
+ Added in Head + 3 +
+
+ Removed in Head + 0 +
+
+ +
+

Diffs by Resource Type

+ + + + + + + + +
TypeDiff requests
Fetch3
+
+ +
+

Round 1

+

1 added, 0 removed

+
+ +
+
+ Added in Head + POST + Fetch + 200 +
+ http://127.0.0.1:61812/vite/new-chunk.js +
+
Round
1
+
Base count
0
+
Head count
1
+
Encoded
50 KB
+
Decoded body
120 KB
+
MIME
text/javascript
+
Protocol
h2
+
Remote
-
+
Failed
no
+
+ + +
+ Request body + 
{
+	"i": 1
+}
+
+ + +
+ Response headers + 
{
+	"content-type": "text/javascript"
+}
+
+
+
+
+ +
+

Round 2

+

1 added, 0 removed

+
+ +
+
+ Added in Head + POST + Fetch + 200 +
+ http://127.0.0.1:61812/vite/new-chunk.js +
+
Round
2
+
Base count
0
+
Head count
1
+
Encoded
50 KB
+
Decoded body
120 KB
+
MIME
text/javascript
+
Protocol
h2
+
Remote
-
+
Failed
no
+
+ + +
+ Request body + 
{
+	"i": 2
+}
+
+ + +
+ Response headers + 
{
+	"content-type": "text/javascript"
+}
+
+
+
+
+ +
+

Round 3

+

1 added, 0 removed

+
+ +
+
+ Added in Head + POST + Fetch + 200 +
+ http://127.0.0.1:61812/vite/new-chunk.js +
+
Round
3
+
Base count
0
+
Head count
1
+
Encoded
50 KB
+
Decoded body
120 KB
+
MIME
text/javascript
+
Protocol
h2
+
Remote
-
+
Failed
no
+
+ + +
+ Request body + 
{
+	"i": 3
+}
+
+ + +
+ Response headers + 
{
+	"content-type": "text/javascript"
+}
+
+
+
+
+
+ + diff --git a/packages-private/diagnostics-frontend/test/browser/fixtures/base.json b/packages-private/diagnostics-frontend/test/browser/fixtures/base.json new file mode 100644 index 0000000000..af591f4615 --- /dev/null +++ b/packages-private/diagnostics-frontend/test/browser/fixtures/base.json @@ -0,0 +1,679 @@ +{ + "label": "base", + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "sampleCount": 3, + "aggregation": "median", + "summary": { + "label": "base", + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "diagnostics": { + "pageErrorCount": 0, + "console": { + "log": 5, + "warning": 3, + "error": 0, + "info": 2 + } + }, + "durationMs": 20400, + "network": { + "requestCount": 20, + "webSocketConnectionCount": 1, + "webSocketSentBytes": 2040, + "webSocketReceivedBytes": 9180, + "finishedRequestCount": 18, + "failedRequestCount": 0, + "cachedRequestCount": 0, + "serviceWorkerRequestCount": 0, + "totalEncodedBytes": 1040400, + "totalDecodedBodyBytes": 3457800, + "sameOriginEncodedBytes": 1020000, + "thirdPartyEncodedBytes": 20400, + "byResourceType": { + "Script": { + "requests": 10, + "encodedBytes": 918000, + "decodedBodyBytes": 3060000 + }, + "Stylesheet": { + "requests": 2, + "encodedBytes": 81600, + "decodedBodyBytes": 306000 + }, + "Fetch": { + "requests": 6, + "encodedBytes": 40800, + "decodedBodyBytes": 91800 + } + }, + "largestRequests": [ + { + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "status": 200, + "encodedBytes": 300000, + "decodedBodyBytes": 900000 + } + ], + "failedRequests": [] + }, + "performance": { + "cdpMetrics": { + "Nodes": 5000, + "JSEventListeners": 400, + "LayoutCount": 30 + }, + "runtimeHeap": { + "usedSize": 30600000, + "totalSize": 51000000 + }, + "tabMemory": { + "totalBytes": 91800000 + }, + "webVitals": { + "firstPaintMs": 400, + "firstContentfulPaintMs": 450, + "domContentLoadedEventEndMs": 800, + "loadEventEndMs": 1200, + "longTaskCount": 3, + "longTaskDurationMs": 300, + "maxLongTaskDurationMs": 150, + "resourceEntryCount": 18, + "domElements": 2500 + } + }, + "heapSnapshot": { + "categories": { + "total": 1020000, + "code": 2040000, + "strings": 3060000, + "jsArrays": 4080000, + "typedArrays": 5100000, + "systemObjects": 6120000, + "otherJsObjects": 7140000, + "otherNonJsObjects": 8160000 + }, + "nodeCounts": { + "total": 100, + "code": 200, + "strings": 300, + "jsArrays": 400, + "typedArrays": 500, + "systemObjects": 600, + "otherJsObjects": 700, + "otherNonJsObjects": 800 + }, + "breakdowns": { + "code": { + "code: alpha": 1224000, + "Other": 816000 + }, + "strings": { + "strings: alpha": 1836000, + "Other": 1224000 + }, + "jsArrays": { + "jsArrays: alpha": 2448000, + "Other": 1632000 + }, + "typedArrays": { + "typedArrays: alpha": 3060000, + "Other": 2040000 + }, + "systemObjects": { + "systemObjects: alpha": 3672000, + "Other": 2448000 + }, + "otherJsObjects": { + "otherJsObjects: alpha": 4284000, + "Other": 2856000 + }, + "otherNonJsObjects": { + "otherNonJsObjects: alpha": 4896000, + "Other": 3264000 + } + } + } + }, + "samples": [ + { + "label": "base", + "round": 1, + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "diagnostics": { + "pageErrorCount": 0, + "console": { + "log": 5, + "warning": 2, + "error": 0, + "info": 2 + } + }, + "durationMs": 20200, + "network": { + "requestCount": 19, + "webSocketConnectionCount": 1, + "webSocketSentBytes": 2020, + "webSocketReceivedBytes": 9090, + "finishedRequestCount": 18, + "failedRequestCount": 0, + "cachedRequestCount": 0, + "serviceWorkerRequestCount": 0, + "totalEncodedBytes": 1030200, + "totalDecodedBodyBytes": 3423900, + "sameOriginEncodedBytes": 1010000, + "thirdPartyEncodedBytes": 20200, + "byResourceType": { + "Script": { + "requests": 10, + "encodedBytes": 909000, + "decodedBodyBytes": 3030000 + }, + "Stylesheet": { + "requests": 2, + "encodedBytes": 80800, + "decodedBodyBytes": 303000 + }, + "Fetch": { + "requests": 6, + "encodedBytes": 40400, + "decodedBodyBytes": 90900 + } + }, + "largestRequests": [ + { + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "status": 200, + "encodedBytes": 300000, + "decodedBodyBytes": 900000 + } + ], + "failedRequests": [] + }, + "networkRequests": [ + { + "requestId": "1-http://127.0.0.1:61812/vite/a.js", + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + }, + { + "requestId": "1-http://127.0.0.1:61812/vite/b.js", + "url": "http://127.0.0.1:61812/vite/b.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + } + ], + "performance": { + "cdpMetrics": { + "Nodes": 5000, + "JSEventListeners": 400, + "LayoutCount": 30 + }, + "runtimeHeap": { + "usedSize": 30300000, + "totalSize": 50500000 + }, + "tabMemory": { + "totalBytes": 90900000 + }, + "webVitals": { + "firstPaintMs": 400, + "firstContentfulPaintMs": 450, + "domContentLoadedEventEndMs": 800, + "loadEventEndMs": 1200, + "longTaskCount": 3, + "longTaskDurationMs": 300, + "maxLongTaskDurationMs": 150, + "resourceEntryCount": 18, + "domElements": 2500 + } + }, + "heapSnapshot": { + "categories": { + "total": 1010000, + "code": 2020000, + "strings": 3030000, + "jsArrays": 4040000, + "typedArrays": 5050000, + "systemObjects": 6060000, + "otherJsObjects": 7070000, + "otherNonJsObjects": 8080000 + }, + "nodeCounts": { + "total": 100, + "code": 200, + "strings": 300, + "jsArrays": 400, + "typedArrays": 500, + "systemObjects": 600, + "otherJsObjects": 700, + "otherNonJsObjects": 800 + }, + "breakdowns": { + "code": { + "code: alpha": 1212000, + "Other": 808000 + }, + "strings": { + "strings: alpha": 1818000, + "Other": 1212000 + }, + "jsArrays": { + "jsArrays: alpha": 2424000, + "Other": 1616000 + }, + "typedArrays": { + "typedArrays: alpha": 3030000, + "Other": 2020000 + }, + "systemObjects": { + "systemObjects: alpha": 3636000, + "Other": 2424000 + }, + "otherJsObjects": { + "otherJsObjects: alpha": 4242000, + "Other": 2828000 + }, + "otherNonJsObjects": { + "otherNonJsObjects: alpha": 4848000, + "Other": 3232000 + } + } + } + }, + { + "label": "base", + "round": 2, + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "diagnostics": { + "pageErrorCount": 0, + "console": { + "log": 5, + "warning": 3, + "error": 0, + "info": 2 + } + }, + "durationMs": 20400, + "network": { + "requestCount": 20, + "webSocketConnectionCount": 1, + "webSocketSentBytes": 2040, + "webSocketReceivedBytes": 9180, + "finishedRequestCount": 18, + "failedRequestCount": 0, + "cachedRequestCount": 0, + "serviceWorkerRequestCount": 0, + "totalEncodedBytes": 1040400, + "totalDecodedBodyBytes": 3457800, + "sameOriginEncodedBytes": 1020000, + "thirdPartyEncodedBytes": 20400, + "byResourceType": { + "Script": { + "requests": 10, + "encodedBytes": 918000, + "decodedBodyBytes": 3060000 + }, + "Stylesheet": { + "requests": 2, + "encodedBytes": 81600, + "decodedBodyBytes": 306000 + }, + "Fetch": { + "requests": 6, + "encodedBytes": 40800, + "decodedBodyBytes": 91800 + } + }, + "largestRequests": [ + { + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "status": 200, + "encodedBytes": 300000, + "decodedBodyBytes": 900000 + } + ], + "failedRequests": [] + }, + "networkRequests": [ + { + "requestId": "2-http://127.0.0.1:61812/vite/a.js", + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + }, + { + "requestId": "2-http://127.0.0.1:61812/vite/b.js", + "url": "http://127.0.0.1:61812/vite/b.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + } + ], + "performance": { + "cdpMetrics": { + "Nodes": 5000, + "JSEventListeners": 400, + "LayoutCount": 30 + }, + "runtimeHeap": { + "usedSize": 30600000, + "totalSize": 51000000 + }, + "tabMemory": { + "totalBytes": 91800000 + }, + "webVitals": { + "firstPaintMs": 400, + "firstContentfulPaintMs": 450, + "domContentLoadedEventEndMs": 800, + "loadEventEndMs": 1200, + "longTaskCount": 3, + "longTaskDurationMs": 300, + "maxLongTaskDurationMs": 150, + "resourceEntryCount": 18, + "domElements": 2500 + } + }, + "heapSnapshot": { + "categories": { + "total": 1020000, + "code": 2040000, + "strings": 3060000, + "jsArrays": 4080000, + "typedArrays": 5100000, + "systemObjects": 6120000, + "otherJsObjects": 7140000, + "otherNonJsObjects": 8160000 + }, + "nodeCounts": { + "total": 100, + "code": 200, + "strings": 300, + "jsArrays": 400, + "typedArrays": 500, + "systemObjects": 600, + "otherJsObjects": 700, + "otherNonJsObjects": 800 + }, + "breakdowns": { + "code": { + "code: alpha": 1224000, + "Other": 816000 + }, + "strings": { + "strings: alpha": 1836000, + "Other": 1224000 + }, + "jsArrays": { + "jsArrays: alpha": 2448000, + "Other": 1632000 + }, + "typedArrays": { + "typedArrays: alpha": 3060000, + "Other": 2040000 + }, + "systemObjects": { + "systemObjects: alpha": 3672000, + "Other": 2448000 + }, + "otherJsObjects": { + "otherJsObjects: alpha": 4284000, + "Other": 2856000 + }, + "otherNonJsObjects": { + "otherNonJsObjects: alpha": 4896000, + "Other": 3264000 + } + } + } + }, + { + "label": "base", + "round": 3, + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "diagnostics": { + "pageErrorCount": 0, + "console": { + "log": 5, + "warning": 4, + "error": 0, + "info": 2 + } + }, + "durationMs": 20600, + "network": { + "requestCount": 21, + "webSocketConnectionCount": 1, + "webSocketSentBytes": 2060, + "webSocketReceivedBytes": 9270, + "finishedRequestCount": 18, + "failedRequestCount": 0, + "cachedRequestCount": 0, + "serviceWorkerRequestCount": 0, + "totalEncodedBytes": 1050600, + "totalDecodedBodyBytes": 3491700, + "sameOriginEncodedBytes": 1030000, + "thirdPartyEncodedBytes": 20600, + "byResourceType": { + "Script": { + "requests": 10, + "encodedBytes": 927000, + "decodedBodyBytes": 3090000 + }, + "Stylesheet": { + "requests": 2, + "encodedBytes": 82400, + "decodedBodyBytes": 309000 + }, + "Fetch": { + "requests": 6, + "encodedBytes": 41200, + "decodedBodyBytes": 92700 + } + }, + "largestRequests": [ + { + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "status": 200, + "encodedBytes": 300000, + "decodedBodyBytes": 900000 + } + ], + "failedRequests": [] + }, + "networkRequests": [ + { + "requestId": "3-http://127.0.0.1:61812/vite/a.js", + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + }, + { + "requestId": "3-http://127.0.0.1:61812/vite/b.js", + "url": "http://127.0.0.1:61812/vite/b.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + } + ], + "performance": { + "cdpMetrics": { + "Nodes": 5000, + "JSEventListeners": 400, + "LayoutCount": 30 + }, + "runtimeHeap": { + "usedSize": 30900000, + "totalSize": 51500000 + }, + "tabMemory": { + "totalBytes": 92700000 + }, + "webVitals": { + "firstPaintMs": 400, + "firstContentfulPaintMs": 450, + "domContentLoadedEventEndMs": 800, + "loadEventEndMs": 1200, + "longTaskCount": 3, + "longTaskDurationMs": 300, + "maxLongTaskDurationMs": 150, + "resourceEntryCount": 18, + "domElements": 2500 + } + }, + "heapSnapshot": { + "categories": { + "total": 1030000, + "code": 2060000, + "strings": 3090000, + "jsArrays": 4120000, + "typedArrays": 5150000, + "systemObjects": 6180000, + "otherJsObjects": 7210000, + "otherNonJsObjects": 8240000 + }, + "nodeCounts": { + "total": 100, + "code": 200, + "strings": 300, + "jsArrays": 400, + "typedArrays": 500, + "systemObjects": 600, + "otherJsObjects": 700, + "otherNonJsObjects": 800 + }, + "breakdowns": { + "code": { + "code: alpha": 1236000, + "Other": 824000 + }, + "strings": { + "strings: alpha": 1854000, + "Other": 1236000 + }, + "jsArrays": { + "jsArrays: alpha": 2472000, + "Other": 1648000 + }, + "typedArrays": { + "typedArrays: alpha": 3090000, + "Other": 2060000 + }, + "systemObjects": { + "systemObjects: alpha": 3708000, + "Other": 2472000 + }, + "otherJsObjects": { + "otherJsObjects: alpha": 4326000, + "Other": 2884000 + }, + "otherNonJsObjects": { + "otherNonJsObjects: alpha": 4944000, + "Other": 3296000 + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages-private/diagnostics-frontend/test/browser/fixtures/head.json b/packages-private/diagnostics-frontend/test/browser/fixtures/head.json new file mode 100644 index 0000000000..9832e65b31 --- /dev/null +++ b/packages-private/diagnostics-frontend/test/browser/fixtures/head.json @@ -0,0 +1,742 @@ +{ + "label": "head", + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "sampleCount": 3, + "aggregation": "median", + "summary": { + "label": "head", + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "diagnostics": { + "pageErrorCount": 0, + "console": { + "log": 5, + "warning": 3, + "error": 0, + "info": 2 + } + }, + "durationMs": 22032, + "network": { + "requestCount": 20, + "webSocketConnectionCount": 1, + "webSocketSentBytes": 2203, + "webSocketReceivedBytes": 9914, + "finishedRequestCount": 18, + "failedRequestCount": 0, + "cachedRequestCount": 0, + "serviceWorkerRequestCount": 0, + "totalEncodedBytes": 1123632, + "totalDecodedBodyBytes": 3734424, + "sameOriginEncodedBytes": 1101600, + "thirdPartyEncodedBytes": 22032, + "byResourceType": { + "Script": { + "requests": 10, + "encodedBytes": 991440, + "decodedBodyBytes": 3304800 + }, + "Stylesheet": { + "requests": 2, + "encodedBytes": 88128, + "decodedBodyBytes": 330480 + }, + "Fetch": { + "requests": 6, + "encodedBytes": 44064, + "decodedBodyBytes": 99144 + } + }, + "largestRequests": [ + { + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "status": 200, + "encodedBytes": 300000, + "decodedBodyBytes": 900000 + } + ], + "failedRequests": [] + }, + "performance": { + "cdpMetrics": { + "Nodes": 5000, + "JSEventListeners": 400, + "LayoutCount": 30 + }, + "runtimeHeap": { + "usedSize": 33048000, + "totalSize": 55080000 + }, + "tabMemory": { + "totalBytes": 99144000 + }, + "webVitals": { + "firstPaintMs": 400, + "firstContentfulPaintMs": 450, + "domContentLoadedEventEndMs": 800, + "loadEventEndMs": 1200, + "longTaskCount": 3, + "longTaskDurationMs": 300, + "maxLongTaskDurationMs": 150, + "resourceEntryCount": 18, + "domElements": 2500 + } + }, + "heapSnapshot": { + "categories": { + "total": 1101600, + "code": 2203200, + "strings": 3304800, + "jsArrays": 4406400, + "typedArrays": 5508000, + "systemObjects": 6609600, + "otherJsObjects": 7711200, + "otherNonJsObjects": 8812800 + }, + "nodeCounts": { + "total": 100, + "code": 200, + "strings": 300, + "jsArrays": 400, + "typedArrays": 500, + "systemObjects": 600, + "otherJsObjects": 700, + "otherNonJsObjects": 800 + }, + "breakdowns": { + "code": { + "code: alpha": 1321920, + "Other": 881280 + }, + "strings": { + "strings: alpha": 1982880, + "Other": 1321920 + }, + "jsArrays": { + "jsArrays: alpha": 2643840, + "Other": 1762560 + }, + "typedArrays": { + "typedArrays: alpha": 3304800, + "Other": 2203200 + }, + "systemObjects": { + "systemObjects: alpha": 3965760, + "Other": 2643840 + }, + "otherJsObjects": { + "otherJsObjects: alpha": 4626720, + "Other": 3084480 + }, + "otherNonJsObjects": { + "otherNonJsObjects: alpha": 5287680, + "Other": 3525120 + } + } + } + }, + "samples": [ + { + "label": "head", + "round": 1, + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "diagnostics": { + "pageErrorCount": 0, + "console": { + "log": 5, + "warning": 2, + "error": 0, + "info": 2 + } + }, + "durationMs": 21816, + "network": { + "requestCount": 19, + "webSocketConnectionCount": 1, + "webSocketSentBytes": 2182, + "webSocketReceivedBytes": 9817, + "finishedRequestCount": 18, + "failedRequestCount": 0, + "cachedRequestCount": 0, + "serviceWorkerRequestCount": 0, + "totalEncodedBytes": 1112616, + "totalDecodedBodyBytes": 3697812, + "sameOriginEncodedBytes": 1090800, + "thirdPartyEncodedBytes": 21816, + "byResourceType": { + "Script": { + "requests": 10, + "encodedBytes": 981720, + "decodedBodyBytes": 3272400 + }, + "Stylesheet": { + "requests": 2, + "encodedBytes": 87264, + "decodedBodyBytes": 327240 + }, + "Fetch": { + "requests": 6, + "encodedBytes": 43632, + "decodedBodyBytes": 98172 + } + }, + "largestRequests": [ + { + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "status": 200, + "encodedBytes": 300000, + "decodedBodyBytes": 900000 + } + ], + "failedRequests": [] + }, + "networkRequests": [ + { + "requestId": "1-http://127.0.0.1:61812/vite/a.js", + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + }, + { + "requestId": "1-http://127.0.0.1:61812/vite/b.js", + "url": "http://127.0.0.1:61812/vite/b.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + }, + { + "requestId": "1-http://127.0.0.1:61812/vite/new-chunk.js", + "url": "http://127.0.0.1:61812/vite/new-chunk.js", + "method": "POST", + "resourceType": "Fetch", + "startedAt": 1, + "hasRequestBody": true, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200, + "requestBody": "{\"i\":1}" + } + ], + "performance": { + "cdpMetrics": { + "Nodes": 5000, + "JSEventListeners": 400, + "LayoutCount": 30 + }, + "runtimeHeap": { + "usedSize": 32724000, + "totalSize": 54540000 + }, + "tabMemory": { + "totalBytes": 98172000 + }, + "webVitals": { + "firstPaintMs": 400, + "firstContentfulPaintMs": 450, + "domContentLoadedEventEndMs": 800, + "loadEventEndMs": 1200, + "longTaskCount": 3, + "longTaskDurationMs": 300, + "maxLongTaskDurationMs": 150, + "resourceEntryCount": 18, + "domElements": 2500 + } + }, + "heapSnapshot": { + "categories": { + "total": 1090800, + "code": 2181600, + "strings": 3272400, + "jsArrays": 4363200, + "typedArrays": 5454000, + "systemObjects": 6544800, + "otherJsObjects": 7635600, + "otherNonJsObjects": 8726400 + }, + "nodeCounts": { + "total": 100, + "code": 200, + "strings": 300, + "jsArrays": 400, + "typedArrays": 500, + "systemObjects": 600, + "otherJsObjects": 700, + "otherNonJsObjects": 800 + }, + "breakdowns": { + "code": { + "code: alpha": 1308960, + "Other": 872640 + }, + "strings": { + "strings: alpha": 1963440, + "Other": 1308960 + }, + "jsArrays": { + "jsArrays: alpha": 2617920, + "Other": 1745280 + }, + "typedArrays": { + "typedArrays: alpha": 3272400, + "Other": 2181600 + }, + "systemObjects": { + "systemObjects: alpha": 3926880, + "Other": 2617920 + }, + "otherJsObjects": { + "otherJsObjects: alpha": 4581360, + "Other": 3054240 + }, + "otherNonJsObjects": { + "otherNonJsObjects: alpha": 5235840, + "Other": 3490560 + } + } + } + }, + { + "label": "head", + "round": 2, + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "diagnostics": { + "pageErrorCount": 0, + "console": { + "log": 5, + "warning": 3, + "error": 0, + "info": 2 + } + }, + "durationMs": 22032, + "network": { + "requestCount": 20, + "webSocketConnectionCount": 1, + "webSocketSentBytes": 2203, + "webSocketReceivedBytes": 9914, + "finishedRequestCount": 18, + "failedRequestCount": 0, + "cachedRequestCount": 0, + "serviceWorkerRequestCount": 0, + "totalEncodedBytes": 1123632, + "totalDecodedBodyBytes": 3734424, + "sameOriginEncodedBytes": 1101600, + "thirdPartyEncodedBytes": 22032, + "byResourceType": { + "Script": { + "requests": 10, + "encodedBytes": 991440, + "decodedBodyBytes": 3304800 + }, + "Stylesheet": { + "requests": 2, + "encodedBytes": 88128, + "decodedBodyBytes": 330480 + }, + "Fetch": { + "requests": 6, + "encodedBytes": 44064, + "decodedBodyBytes": 99144 + } + }, + "largestRequests": [ + { + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "status": 200, + "encodedBytes": 300000, + "decodedBodyBytes": 900000 + } + ], + "failedRequests": [] + }, + "networkRequests": [ + { + "requestId": "2-http://127.0.0.1:61812/vite/a.js", + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + }, + { + "requestId": "2-http://127.0.0.1:61812/vite/b.js", + "url": "http://127.0.0.1:61812/vite/b.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + }, + { + "requestId": "2-http://127.0.0.1:61812/vite/new-chunk.js", + "url": "http://127.0.0.1:61812/vite/new-chunk.js", + "method": "POST", + "resourceType": "Fetch", + "startedAt": 1, + "hasRequestBody": true, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200, + "requestBody": "{\"i\":2}" + } + ], + "performance": { + "cdpMetrics": { + "Nodes": 5000, + "JSEventListeners": 400, + "LayoutCount": 30 + }, + "runtimeHeap": { + "usedSize": 33048000, + "totalSize": 55080000 + }, + "tabMemory": { + "totalBytes": 99144000 + }, + "webVitals": { + "firstPaintMs": 400, + "firstContentfulPaintMs": 450, + "domContentLoadedEventEndMs": 800, + "loadEventEndMs": 1200, + "longTaskCount": 3, + "longTaskDurationMs": 300, + "maxLongTaskDurationMs": 150, + "resourceEntryCount": 18, + "domElements": 2500 + } + }, + "heapSnapshot": { + "categories": { + "total": 1101600, + "code": 2203200, + "strings": 3304800, + "jsArrays": 4406400, + "typedArrays": 5508000, + "systemObjects": 6609600, + "otherJsObjects": 7711200, + "otherNonJsObjects": 8812800 + }, + "nodeCounts": { + "total": 100, + "code": 200, + "strings": 300, + "jsArrays": 400, + "typedArrays": 500, + "systemObjects": 600, + "otherJsObjects": 700, + "otherNonJsObjects": 800 + }, + "breakdowns": { + "code": { + "code: alpha": 1321920, + "Other": 881280 + }, + "strings": { + "strings: alpha": 1982880, + "Other": 1321920 + }, + "jsArrays": { + "jsArrays: alpha": 2643840, + "Other": 1762560 + }, + "typedArrays": { + "typedArrays: alpha": 3304800, + "Other": 2203200 + }, + "systemObjects": { + "systemObjects: alpha": 3965760, + "Other": 2643840 + }, + "otherJsObjects": { + "otherJsObjects: alpha": 4626720, + "Other": 3084480 + }, + "otherNonJsObjects": { + "otherNonJsObjects: alpha": 5287680, + "Other": 3525120 + } + } + } + }, + { + "label": "head", + "round": 3, + "timestamp": "2026-07-18T00:00:00.000Z", + "url": "http://127.0.0.1:61812", + "scenario": "fresh browser signup, first timeline note, after the note becomes visible", + "diagnostics": { + "pageErrorCount": 0, + "console": { + "log": 5, + "warning": 4, + "error": 0, + "info": 2 + } + }, + "durationMs": 22248, + "network": { + "requestCount": 21, + "webSocketConnectionCount": 1, + "webSocketSentBytes": 2225, + "webSocketReceivedBytes": 10012, + "finishedRequestCount": 18, + "failedRequestCount": 0, + "cachedRequestCount": 0, + "serviceWorkerRequestCount": 0, + "totalEncodedBytes": 1134648, + "totalDecodedBodyBytes": 3771036, + "sameOriginEncodedBytes": 1112400, + "thirdPartyEncodedBytes": 22248, + "byResourceType": { + "Script": { + "requests": 10, + "encodedBytes": 1001160, + "decodedBodyBytes": 3337200 + }, + "Stylesheet": { + "requests": 2, + "encodedBytes": 88992, + "decodedBodyBytes": 333720 + }, + "Fetch": { + "requests": 6, + "encodedBytes": 44496, + "decodedBodyBytes": 100116 + } + }, + "largestRequests": [ + { + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "status": 200, + "encodedBytes": 300000, + "decodedBodyBytes": 900000 + } + ], + "failedRequests": [] + }, + "networkRequests": [ + { + "requestId": "3-http://127.0.0.1:61812/vite/a.js", + "url": "http://127.0.0.1:61812/vite/a.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + }, + { + "requestId": "3-http://127.0.0.1:61812/vite/b.js", + "url": "http://127.0.0.1:61812/vite/b.js", + "method": "GET", + "resourceType": "Script", + "startedAt": 1, + "hasRequestBody": false, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200 + }, + { + "requestId": "3-http://127.0.0.1:61812/vite/new-chunk.js", + "url": "http://127.0.0.1:61812/vite/new-chunk.js", + "method": "POST", + "resourceType": "Fetch", + "startedAt": 1, + "hasRequestBody": true, + "encodedDataLength": 50000, + "decodedBodyLength": 120000, + "fromDiskCache": false, + "fromServiceWorker": false, + "finished": true, + "failed": false, + "mimeType": "text/javascript", + "protocol": "h2", + "responseHeaders": { + "content-type": "text/javascript" + }, + "status": 200, + "requestBody": "{\"i\":3}" + } + ], + "performance": { + "cdpMetrics": { + "Nodes": 5000, + "JSEventListeners": 400, + "LayoutCount": 30 + }, + "runtimeHeap": { + "usedSize": 33372000, + "totalSize": 55620000 + }, + "tabMemory": { + "totalBytes": 100116000 + }, + "webVitals": { + "firstPaintMs": 400, + "firstContentfulPaintMs": 450, + "domContentLoadedEventEndMs": 800, + "loadEventEndMs": 1200, + "longTaskCount": 3, + "longTaskDurationMs": 300, + "maxLongTaskDurationMs": 150, + "resourceEntryCount": 18, + "domElements": 2500 + } + }, + "heapSnapshot": { + "categories": { + "total": 1112400, + "code": 2224800, + "strings": 3337200, + "jsArrays": 4449600, + "typedArrays": 5562000, + "systemObjects": 6674400, + "otherJsObjects": 7786800, + "otherNonJsObjects": 8899200 + }, + "nodeCounts": { + "total": 100, + "code": 200, + "strings": 300, + "jsArrays": 400, + "typedArrays": 500, + "systemObjects": 600, + "otherJsObjects": 700, + "otherNonJsObjects": 800 + }, + "breakdowns": { + "code": { + "code: alpha": 1334880, + "Other": 889920 + }, + "strings": { + "strings: alpha": 2002320, + "Other": 1334880 + }, + "jsArrays": { + "jsArrays: alpha": 2669760, + "Other": 1779840 + }, + "typedArrays": { + "typedArrays: alpha": 3337200, + "Other": 2224800 + }, + "systemObjects": { + "systemObjects: alpha": 4004640, + "Other": 2669760 + }, + "otherJsObjects": { + "otherJsObjects: alpha": 4672080, + "Other": 3114720 + }, + "otherNonJsObjects": { + "otherNonJsObjects: alpha": 5339520, + "Other": 3559680 + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages-private/diagnostics-frontend/test/browser/render.test.ts b/packages-private/diagnostics-frontend/test/browser/render.test.ts new file mode 100644 index 0000000000..9def362ff6 --- /dev/null +++ b/packages-private/diagnostics-frontend/test/browser/render.test.ts @@ -0,0 +1,44 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { readFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { afterEach, beforeEach, expect, test, vi } from 'vitest'; +import { renderBrowserDiagnosticsHtml } from '../../src/browser/report/html'; +import type { BrowserMetricsReport } from '../../src/browser/types'; + +const fixturesDir = join(import.meta.dirname, 'fixtures'); + +async function loadFixture(name: string) { + return JSON.parse(await readFile(join(fixturesDir, `${name}.json`), 'utf8')) as BrowserMetricsReport; +} + +beforeEach(() => { + // renderBrowserDiagnosticsHtml() が生成時刻を埋め込むので、スナップショットが揺れないよう時刻を固定する + vi.useFakeTimers(); + vi.setSystemTime(new Date('2026-07-18T00:00:00.000Z')); +}); + +afterEach(() => { + vi.useRealTimers(); +}); + +test('renders the network request diff html report', async () => { + const html = renderBrowserDiagnosticsHtml(await loadFixture('base'), await loadFixture('head')); + + await expect(html).toMatchFileSnapshot('./__snapshots__/render-html.html'); +}); + +test('escapes html metacharacters coming from the browser session', async () => { + const base = await loadFixture('base'); + const head = await loadFixture('head'); + // CDP由来の値は原理的には任意の文字列になりうるので、生HTMLとして出ないことを確かめる + head.samples[0].networkRequests[0].url = 'http://127.0.0.1:61812/">'; + + const html = renderBrowserDiagnosticsHtml(base, head); + + expect(html).not.toContain(''); + expect(html).toContain('<script>alert(1)</script>'); +}); diff --git a/packages-private/diagnostics-frontend/test/browser/server.test.ts b/packages-private/diagnostics-frontend/test/browser/server.test.ts new file mode 100644 index 0000000000..24f3e62444 --- /dev/null +++ b/packages-private/diagnostics-frontend/test/browser/server.test.ts @@ -0,0 +1,47 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { EventEmitter } from 'node:events'; +import { afterEach, expect, test, vi } from 'vitest'; +import { stopServer, waitForServer } from '../../src/browser/server'; +import type { ChildProcess } from 'node:child_process'; + +vi.mock('node:child_process', async () => { + const actual = await vi.importActual('node:child_process'); + return { + ...actual, + spawnSync: vi.fn(), + }; +}); + +function signaledChild() { + return Object.assign(new EventEmitter(), { + exitCode: null, + signalCode: 'SIGTERM' as NodeJS.Signals, + pid: undefined, + kill: vi.fn(), + }) as unknown as ChildProcess; +} + +afterEach(() => { + vi.unstubAllGlobals(); +}); + +test('waitForServer fails immediately when the server exited by signal', async () => { + const fetchMock = vi.fn().mockResolvedValue({ status: 200 }); + vi.stubGlobal('fetch', fetchMock); + + await expect(waitForServer('http://127.0.0.1:61812', signaledChild())) + .rejects.toThrow('Misskey server exited early with signal SIGTERM'); + expect(fetchMock).not.toHaveBeenCalled(); +}); + +test('stopServer returns immediately when the server already exited by signal', async () => { + const child = signaledChild(); + const stopPromise = stopServer(child); + + expect(child.listenerCount('exit')).toBe(0); + await stopPromise; +}); diff --git a/packages-private/diagnostics-frontend/test/bundle/fixtures/base-stats.json b/packages-private/diagnostics-frontend/test/bundle/fixtures/base-stats.json new file mode 100644 index 0000000000..27f6b2d6b4 --- /dev/null +++ b/packages-private/diagnostics-frontend/test/bundle/fixtures/base-stats.json @@ -0,0 +1 @@ +{"nodeParts": {"p0": {"renderedLength": 1000, "gzipLength": 300, "brotliLength": 250}, "p1": {"renderedLength": 2000, "gzipLength": 600, "brotliLength": 500}, "p2": {"renderedLength": 3000, "gzipLength": 900, "brotliLength": 750}, "p3": {"renderedLength": 4000, "gzipLength": 1200, "brotliLength": 1000}, "p4": {"renderedLength": 5000, "gzipLength": 1500, "brotliLength": 1250}, "p5": {"renderedLength": 6000, "gzipLength": 1800, "brotliLength": 1500}}, "nodeMetas": {"m0": {"id": "/src/mod0.ts", "isEntry": true, "importedBy": [], "imported": [{"id": "m1", "dynamic": true}], "moduleParts": {"assets/boot-a1.js": "p0"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m1": {"id": "/src/mod1.ts", "isEntry": false, "importedBy": ["m0"], "imported": [{"id": "m2", "dynamic": false}], "moduleParts": {"assets/vue-b2.js": "p1"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m2": {"id": "/src/mod2.ts", "isEntry": false, "importedBy": ["m1"], "imported": [{"id": "m3", "dynamic": true}], "moduleParts": {"assets/boot-a1.js": "p2"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m3": {"id": "/src/mod3.ts", "isEntry": false, "importedBy": ["m2"], "imported": [{"id": "m4", "dynamic": false}], "moduleParts": {"assets/vue-b2.js": "p3"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m4": {"id": "/src/mod4.ts", "isEntry": false, "importedBy": ["m3"], "imported": [{"id": "m5", "dynamic": true}], "moduleParts": {"assets/boot-a1.js": "p4"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m5": {"id": "/src/mod5.ts", "isEntry": false, "importedBy": ["m4"], "imported": [], "moduleParts": {"assets/vue-b2.js": "p5"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}}, "options": {}} diff --git a/packages-private/diagnostics-frontend/test/bundle/fixtures/head-stats.json b/packages-private/diagnostics-frontend/test/bundle/fixtures/head-stats.json new file mode 100644 index 0000000000..ecddcae812 --- /dev/null +++ b/packages-private/diagnostics-frontend/test/bundle/fixtures/head-stats.json @@ -0,0 +1 @@ +{"nodeParts": {"p0": {"renderedLength": 1100.0, "gzipLength": 300, "brotliLength": 250}, "p1": {"renderedLength": 2200.0, "gzipLength": 600, "brotliLength": 500}, "p2": {"renderedLength": 3300.0000000000005, "gzipLength": 900, "brotliLength": 750}, "p3": {"renderedLength": 4400.0, "gzipLength": 1200, "brotliLength": 1000}, "p4": {"renderedLength": 5500.0, "gzipLength": 1500, "brotliLength": 1250}, "p5": {"renderedLength": 6600.000000000001, "gzipLength": 1800, "brotliLength": 1500}, "p6": {"renderedLength": 7700.000000000001, "gzipLength": 2100, "brotliLength": 1750}}, "nodeMetas": {"m0": {"id": "/src/mod0.ts", "isEntry": true, "importedBy": [], "imported": [{"id": "m1", "dynamic": true}], "moduleParts": {"assets/boot-a1.js": "p0"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m1": {"id": "/src/mod1.ts", "isEntry": false, "importedBy": ["m0"], "imported": [{"id": "m2", "dynamic": false}], "moduleParts": {"assets/vue-b2.js": "p1"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m2": {"id": "/src/mod2.ts", "isEntry": false, "importedBy": ["m1"], "imported": [{"id": "m3", "dynamic": true}], "moduleParts": {"assets/boot-a1.js": "p2"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m3": {"id": "/src/mod3.ts", "isEntry": false, "importedBy": ["m2"], "imported": [{"id": "m4", "dynamic": false}], "moduleParts": {"assets/vue-b2.js": "p3"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m4": {"id": "/src/mod4.ts", "isEntry": false, "importedBy": ["m3"], "imported": [{"id": "m5", "dynamic": true}], "moduleParts": {"assets/boot-a1.js": "p4"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m5": {"id": "/src/mod5.ts", "isEntry": false, "importedBy": ["m4"], "imported": [{"id": "m6", "dynamic": false}], "moduleParts": {"assets/vue-b2.js": "p5"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}, "m6": {"id": "/src/mod6.ts", "isEntry": false, "importedBy": ["m5"], "imported": [], "moduleParts": {"assets/boot-a1.js": "p6"}, "renderedLength": 0, "gzipLength": 0, "brotliLength": 0}}, "options": {}} diff --git a/packages-private/diagnostics-frontend/test/bundle/fs-utils.test.ts b/packages-private/diagnostics-frontend/test/bundle/fs-utils.test.ts new file mode 100644 index 0000000000..12ec8d7bb1 --- /dev/null +++ b/packages-private/diagnostics-frontend/test/bundle/fs-utils.test.ts @@ -0,0 +1,29 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { promises as fs } from 'node:fs'; +import { afterEach, expect, test, vi } from 'vitest'; +import { fileExists } from '../../src/bundle/fs-utils'; + +function fsError(code: string) { + return Object.assign(new Error(`fs error: ${code}`), { code }) as NodeJS.ErrnoException; +} + +afterEach(() => { + vi.restoreAllMocks(); +}); + +test('fileExists returns false for ENOENT', async () => { + vi.spyOn(fs, 'access').mockRejectedValueOnce(fsError('ENOENT')); + + await expect(fileExists('missing')).resolves.toBe(false); +}); + +test('fileExists rethrows errors other than ENOENT', async () => { + const error = fsError('EACCES'); + vi.spyOn(fs, 'access').mockRejectedValueOnce(error); + + await expect(fileExists('restricted')).rejects.toBe(error); +}); diff --git a/packages-private/diagnostics-frontend/test/bundle/manifest.test.ts b/packages-private/diagnostics-frontend/test/bundle/manifest.test.ts new file mode 100644 index 0000000000..ed91120f5f --- /dev/null +++ b/packages-private/diagnostics-frontend/test/bundle/manifest.test.ts @@ -0,0 +1,24 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterAll, beforeAll, expect, test } from 'vitest'; +import { collectBundleReport } from '../../src/bundle/manifest'; + +let workDir: string; + +beforeAll(async () => { + workDir = await mkdtemp(join(tmpdir(), 'diagnostics-frontend-test-')); +}); + +afterAll(async () => { + await rm(workDir, { recursive: true, force: true }); +}); + +test('fails loudly when the built output is missing', async () => { + await expect(collectBundleReport(join(workDir, 'nonexistent'))).rejects.toThrow(); +}); diff --git a/packages-private/diagnostics-frontend/test/report.test.ts b/packages-private/diagnostics-frontend/test/report.test.ts new file mode 100644 index 0000000000..8b8fb459cd --- /dev/null +++ b/packages-private/diagnostics-frontend/test/report.test.ts @@ -0,0 +1,309 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { mkdtemp, mkdir, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { afterAll, beforeAll, expect, test } from 'vitest'; +import { collectBundleReport } from '../src/bundle/manifest'; +import { renderFrontendDiagnosticsMarkdown } from '../src/report'; +import type { BrowserMeasurementSample, BrowserMetricsReport } from '../src/browser/types'; +import type { VisualizerReport } from '../src/bundle/visualizer'; + +const bundleFixturesDir = join(import.meta.dirname, 'bundle/fixtures'); +const browserFixturesDir = join(import.meta.dirname, 'browser/fixtures'); + +/** + * ビルド成果物のfixture。 + * + * `collectBundleReport` はファイルの中身を見ずサイズしか使わないので、実体は指定バイト数の + * 詰め物でよい。ディレクトリ名が `built` になるためリポジトリにはコミットできず + * (ルートの .gitignore がビルド成果物として除外する)、テスト実行時に組み立てている。 + */ +const manifest = { + 'src/_boot_.ts': { file: 'assets/boot-a1.js', src: 'src/_boot_.ts', name: 'boot', isEntry: true, imports: ['_vue.js', '_i18n.js'] }, + '_vue.js': { file: 'assets/vue-b2.js', name: 'vue' }, + // `scripts/` 配下はロケール別に出力されるので ja-JP/ に解決される + '_i18n.js': { file: 'scripts/i18n-c3.js', name: 'i18n' }, + 'src/pages/foo.vue': { file: 'assets/foo-d4.js', src: 'src/pages/foo.vue', name: 'foo' }, + // .js 以外はチャンクとして数えない + 'src/pages/style.css': { file: 'assets/style-e5.css', src: 'src/pages/style.css' }, +}; + +const fileSizes = { + base: { + 'assets/boot-a1.js': 20_000, + 'assets/vue-b2.js': 90_000, + 'assets/foo-d4.js': 5_000, + 'assets/style-e5.css': 100, + 'ja-JP/i18n-c3.js': 4_000, + 'ja-JP/orphan.js': 1_200, + }, + head: { + // 差が小さすぎる (閾値5バイト以下) ので「(other)」に集約される + 'assets/boot-a1.js': 20_003, + // 明確に増えるので diff表に行として出る + 'assets/vue-b2.js': 96_000, + 'assets/foo-d4.js': 5_000, + 'assets/style-e5.css': 100, + 'ja-JP/i18n-c3.js': 4_000, + // manifestに載らない出力なので「(other generated chunks)」に集約される + 'ja-JP/orphan.js': 1_500, + }, +} as const satisfies Record<'base' | 'head', Record>; + +let repoDirs: { base: string; head: string }; +let workDir: string; + +beforeAll(async () => { + workDir = await mkdtemp(join(tmpdir(), 'diagnostics-frontend-report-test-')); + + for (const label of ['base', 'head'] as const) { + const outDir = join(workDir, label, 'built/_frontend_vite_'); + await mkdir(outDir, { recursive: true }); + await writeFile(join(outDir, 'manifest.json'), JSON.stringify(manifest)); + + for (const [file, size] of Object.entries(fileSizes[label])) { + const path = join(outDir, file); + await mkdir(dirname(path), { recursive: true }); + await writeFile(path, 'x'.repeat(size)); + } + } + + repoDirs = { + base: join(workDir, 'base'), + head: join(workDir, 'head'), + }; +}); + +afterAll(async () => { + await rm(workDir, { recursive: true, force: true }); +}); + +async function loadBundleStats(name: 'base' | 'head') { + return JSON.parse(await readFile(join(bundleFixturesDir, `${name}-stats.json`), 'utf8')) as VisualizerReport; +} + +async function loadBrowserReport(name: 'base' | 'head') { + return JSON.parse(await readFile(join(browserFixturesDir, `${name}.json`), 'utf8')) as BrowserMetricsReport; +} + +type BrowserReports = { + base: BrowserMetricsReport; + head: BrowserMetricsReport; +}; + +async function renderReport(detailedHtmlUrl: string | null, reports?: BrowserReports) { + const base = reports?.base ?? await loadBrowserReport('base'); + const head = reports?.head ?? await loadBrowserReport('head'); + + return renderFrontendDiagnosticsMarkdown({ + bundle: { + base: await collectBundleReport(repoDirs.base), + head: await collectBundleReport(repoDirs.head), + baseStats: await loadBundleStats('base'), + headStats: await loadBundleStats('head'), + visualizerArtifactUrl: 'https://example.invalid/treemap', + }, + browser: { + base, + head, + baseHeapSnapshotUrl: 'https://example.invalid/base', + headHeapSnapshotUrl: 'https://example.invalid/head', + detailedHtmlUrl, + }, + }); +} + +function withMetricSamples( + report: BrowserMetricsReport, + values: number[], + setValue: (sample: BrowserMeasurementSample, value: number) => void, +): BrowserMetricsReport { + const cloned = structuredClone(report); + const samples = values.map((value, index) => { + const sample = structuredClone(cloned.samples[index % cloned.samples.length]); + sample.round = index + 1; + setValue(sample, value); + return sample; + }); + + return { + ...cloned, + sampleCount: samples.length, + samples, + }; +} + +function requireMetricRow(markdown: string, label: string) { + const row = markdown.split('\n').find(line => line.startsWith(`| **${label}** |`)); + if (row == null) throw new Error(`Metric row not found: ${label}`); + return row; +} + +/** + * 出力をゴールデンファイルで固定する。 + * 意図的に変更したときは `vitest -u` で更新し、__snapshots__ の差分もレビューすること。 + */ +test('renders one frontend diagnostics markdown report from bundle and browser data', async () => { + const markdown = await renderReport('https://example.invalid/html'); + + await expect(markdown).toMatchFileSnapshot('./__snapshots__/report.md'); + expect(markdown).toContain('| Metric | @ Base | @ Head | Δ | MAD |'); + expect(markdown).not.toContain('| Metric | @ Base | @ Head | Δ | MAD | Result |'); + expect(markdown).toContain('Requests by resource type'); + expect(markdown).toContain('## 📦 Bundle Stats'); +}); + +test('omits the browser details link when no detailed html artifact was uploaded', async () => { + const markdown = await renderReport(null); + + expect(markdown).not.toContain('View details'); +}); + +test('renders the difference of independent medians instead of the paired median delta', async () => { + const base = withMetricSamples( + await loadBrowserReport('base'), + [100, 100, 100, 1_000, 1_000], + (sample, value) => { sample.network.requestCount = value; }, + ); + const head = withMetricSamples( + await loadBrowserReport('head'), + [0, 0, 200, 200, 200], + (sample, value) => { sample.network.requestCount = value; }, + ); + + const row = requireMetricRow(await renderReport(null, { base, head }), 'Requests'); + + expect(row).toContain('100
± 0'); + expect(row).toContain('200
± 0'); + expect(row).toContain('$\\color{orange}{\\text{+100}}$
$\\color{orange}{\\text{+100\\\\%}}$'); + expect(row).toContain('| 0 |'); + expect(row.split('|')).toHaveLength(7); + expect(row).not.toMatch(/increase|decrease|within noise|inconclusive/); + expect(row).not.toContain('\\color{green}'); +}); + +test('hides a threshold-sized change that remains within observed noise', async () => { + const base = withMetricSamples( + await loadBrowserReport('base'), + [100, 110, 120], + (sample, value) => { sample.network.requestCount = value; }, + ); + const head = withMetricSamples( + await loadBrowserReport('head'), + [110, 120, 130], + (sample, value) => { sample.network.requestCount = value; }, + ); + + const markdown = await renderReport(null, { base, head }); + + expect(markdown).not.toContain('| **Requests** |'); +}); + +test('renders a directional request count delta at the absolute threshold', async () => { + const base = withMetricSamples( + await loadBrowserReport('base'), + [100, 100, 100], + (sample, value) => { sample.network.requestCount = value; }, + ); + const head = withMetricSamples( + await loadBrowserReport('head'), + [101, 101, 101], + (sample, value) => { sample.network.requestCount = value; }, + ); + + const row = requireMetricRow(await renderReport(null, { base, head }), 'Requests'); + + expect(row).toContain('$\\color{orange}{\\text{+1}}$'); +}); + +test('hides a directional byte change below the existing absolute threshold', async () => { + const base = withMetricSamples( + await loadBrowserReport('base'), + [1_000_000, 1_000_000, 1_000_000], + (sample, value) => { sample.network.totalEncodedBytes = value; }, + ); + const head = withMetricSamples( + await loadBrowserReport('head'), + [1_005_000, 1_005_000, 1_005_000], + (sample, value) => { sample.network.totalEncodedBytes = value; }, + ); + + const markdown = await renderReport(null, { base, head }); + + expect(markdown).not.toContain('| **Encoded network** |'); +}); + +test('renders a directional encoded byte delta at the absolute threshold', async () => { + const base = withMetricSamples( + await loadBrowserReport('base'), + [1_000_000, 1_000_000, 1_000_000], + (sample, value) => { sample.network.totalEncodedBytes = value; }, + ); + const head = withMetricSamples( + await loadBrowserReport('head'), + [1_010_000, 1_010_000, 1_010_000], + (sample, value) => { sample.network.totalEncodedBytes = value; }, + ); + + const row = requireMetricRow(await renderReport(null, { base, head }), 'Encoded network'); + + expect(row).toContain('$\\color{orange}{\\text{+10 KB}}$'); +}); + +test('colours absolute and relative deltas together when the row is significant', async () => { + const base = withMetricSamples( + await loadBrowserReport('base'), + [100_000_000, 100_000_000, 100_000_000], + (sample, value) => { sample.network.totalEncodedBytes = value; }, + ); + const head = withMetricSamples( + await loadBrowserReport('head'), + [100_020_000, 100_020_000, 100_020_000], + (sample, value) => { sample.network.totalEncodedBytes = value; }, + ); + + const row = requireMetricRow(await renderReport(null, { base, head }), 'Encoded network'); + + expect(row).toContain('100 MB
± 0 B'); + expect(row).toContain('$\\color{orange}{\\text{+20 KB}}$
$\\color{orange}{\\text{+0\\\\%}}$'); + expect(row).toContain('| 0 B |'); + expect(row.split('|')).toHaveLength(7); +}); + +test('renders an unavailable relative delta when the base median is zero', async () => { + const base = withMetricSamples( + await loadBrowserReport('base'), + [0, 0, 0], + (sample, value) => { sample.network.requestCount = value; }, + ); + const head = withMetricSamples( + await loadBrowserReport('head'), + [2, 2, 2], + (sample, value) => { sample.network.requestCount = value; }, + ); + + const row = requireMetricRow(await renderReport(null, { base, head }), 'Requests'); + + expect(row).toContain('$\\color{orange}{\\text{+2}}$
-'); + expect(row).not.toContain('increase'); +}); + +test('throws for a metric with fewer than two samples on one side', async () => { + const base = withMetricSamples( + await loadBrowserReport('base'), + [100], + (sample, value) => { sample.network.requestCount = value; }, + ); + const head = withMetricSamples( + await loadBrowserReport('head'), + [102, 102], + (sample, value) => { sample.network.requestCount = value; }, + ); + + await expect(renderReport(null, { base, head })) + .rejects.toThrow('At least two samples per side are required'); +}); diff --git a/packages-private/diagnostics-frontend/tsconfig.json b/packages-private/diagnostics-frontend/tsconfig.json new file mode 100644 index 0000000000..6672c40acb --- /dev/null +++ b/packages-private/diagnostics-frontend/tsconfig.json @@ -0,0 +1,20 @@ +{ + "$schema": "https://json.schemastore.org/tsconfig", + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "bundler", + "strict": true, + "strictFunctionTypes": true, + "strictNullChecks": true, + "esModuleInterop": true, + "skipLibCheck": true, + "noEmit": true, + "types": ["node"] + }, + "include": [ + "src/**/*.ts", + "test/**/*.ts" + ], + "exclude": [] +} diff --git a/packages-private/diagnostics-shared/eslint.config.js b/packages-private/diagnostics-shared/eslint.config.js new file mode 100644 index 0000000000..11bc648243 --- /dev/null +++ b/packages-private/diagnostics-shared/eslint.config.js @@ -0,0 +1,25 @@ +import tsParser from '@typescript-eslint/parser'; +import sharedConfig from '../../packages/shared/eslint.config.js'; + +// eslint-disable-next-line import/no-default-export +export default [ + ...sharedConfig, + { + ignores: [ + '**/node_modules', + '**/__snapshots__', + 'test/fixtures', + ], + }, + { + files: ['**/*.ts', '**/*.tsx'], + languageOptions: { + parserOptions: { + parser: tsParser, + project: ['./tsconfig.json'], + sourceType: 'module', + tsconfigRootDir: import.meta.dirname, + }, + }, + }, +]; diff --git a/packages-private/diagnostics-shared/package.json b/packages-private/diagnostics-shared/package.json new file mode 100644 index 0000000000..7a0268b2df --- /dev/null +++ b/packages-private/diagnostics-shared/package.json @@ -0,0 +1,24 @@ +{ + "name": "diagnostics-shared", + "private": true, + "type": "module", + "exports": { + "./env": "./src/env.ts", + "./format": "./src/format.ts", + "./html": "./src/html.ts", + "./stats": "./src/stats.ts", + "./metric-table": "./src/metric-table.ts", + "./heap-snapshot": "./src/heap-snapshot/index.ts" + }, + "scripts": { + "eslint": "eslint './**/*.{js,jsx,ts,tsx}'", + "lint": "pnpm typecheck && pnpm eslint", + "test": "vitest run", + "typecheck": "tsc --noEmit" + }, + "devDependencies": { + "@types/node": "26.1.1", + "typescript": "5.9.3", + "vitest": "4.1.10" + } +} diff --git a/packages-private/diagnostics-shared/src/env.ts b/packages-private/diagnostics-shared/src/env.ts new file mode 100644 index 0000000000..0875cb2373 --- /dev/null +++ b/packages-private/diagnostics-shared/src/env.ts @@ -0,0 +1,40 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +export function readIntegerEnv(name: string, defaultValue: number, min: number) { + const rawValue = process.env[name]; + if (rawValue == null || rawValue === '') return defaultValue; + if (!/^\d+$/.test(rawValue)) throw new Error(`${name} must be an integer`); + + const value = Number(rawValue); + if (!Number.isSafeInteger(value) || value < min) throw new Error(`${name} must be >= ${min}`); + return value; +} + +export function readBooleanEnv(name: string, defaultValue: boolean) { + const rawValue = process.env[name]; + if (rawValue == null || rawValue === '') return defaultValue; + if (rawValue === '1' || rawValue === 'true') return true; + if (rawValue === '0' || rawValue === 'false') return false; + throw new Error(`${name} must be one of: 1, 0, true, false`); +} + +/** + * 必須の環境変数を読む。未設定・空文字ならエラーにする。 + */ +export function readRequiredEnv(name: string) { + const rawValue = process.env[name]?.trim(); + if (rawValue == null || rawValue === '') throw new Error(`${name} must be set`); + return rawValue; +} + +/** + * 任意の環境変数を読む。未設定・空文字なら null を返す。 + */ +export function readOptionalEnv(name: string) { + const rawValue = process.env[name]?.trim(); + if (rawValue == null || rawValue === '') return null; + return rawValue; +} diff --git a/packages-private/diagnostics-shared/src/format.ts b/packages-private/diagnostics-shared/src/format.ts new file mode 100644 index 0000000000..9ec2a8a749 --- /dev/null +++ b/packages-private/diagnostics-shared/src/format.ts @@ -0,0 +1,117 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +const numberFormatter = new Intl.NumberFormat('en-US', { + maximumFractionDigits: 1, +}); + +export function escapeLatex(text: string) { + return text + .replaceAll('\\', '\\\\') + .replaceAll('{', '\\{') + .replaceAll('}', '\\}') + .replaceAll('%', '\\%'); +} + +export function escapeMdTableCell(value: string) { + return String(value).replaceAll('|', '\\|').replaceAll('\n', '
'); +} + +export function escapeHtml(value: unknown) { + return String(value ?? '') + .replaceAll('&', '&') + .replaceAll('<', '<') + .replaceAll('>', '>') + .replaceAll('"', '"') + .replaceAll('\'', '''); +} + +export function formatNumber(value: number) { + return numberFormatter.format(value); +} + +export function formatBytes(value: number) { + if (value === 0) return '0 B'; // nbspにすること + const units = ['B', 'KB', 'MB', 'GB']; + let unitIndex = 0; + let size = value; + while (size >= 1000 && unitIndex < units.length - 1) { + size /= 1000; + unitIndex += 1; + } + + const maximumFractionDigits = size >= 10 || unitIndex === 0 ? 0 : 1; + // eslint-disable-next-line no-irregular-whitespace + return `${numberFormatter.format(Number(size.toFixed(maximumFractionDigits)))} ${units[unitIndex]}`; // nbspにすること +} + +/** + * KiB単位の値をMB表記にする。/proc 由来の値の表示に使う。 + */ +export function formatKiBAsMb(valueKiB: number | null | undefined) { + if (valueKiB == null || !Number.isFinite(valueKiB)) return '-'; + return `${formatNumber(valueKiB / 1000)} MB`; +} + +export function formatPercent(value: number) { + return `${formatNumber(value)}%`; +} + +export function formatMs(value: number | null | undefined) { + if (value == null || !Number.isFinite(value)) return '-'; + if (value >= 1_000) return `${formatNumber(value / 1_000)} s`; + return `${formatNumber(value)} ms`; +} + +export function formatSecondsAsMs(value: number | null | undefined) { + if (value == null || !Number.isFinite(value)) return '-'; + return formatMs(value * 1_000); +} + +/** + * 差分値を符号付きで整形する。`colorThreshold` 以上の変化があるときだけ色を付ける。 + */ +export function formatColoredDelta(delta: number, text: (value: number) => string, colorThreshold = 0) { + if (delta === 0) return text(0); + const sign = delta > 0 ? '+' : '-'; + if (Math.abs(delta) < colorThreshold) return `$\\text{${sign}${escapeLatex(text(Math.abs(delta))).replaceAll(' ', '~')}}$`; // "1 B"とかで1とBの間で自動改行させないためnbsp(~)にする + const color = delta > 0 ? 'orange' : 'green'; + return `$\\color{${color}}{\\text{${sign}${escapeLatex(text(Math.abs(delta))).replaceAll(' ', '~')}}}$`; // "1 B"とかで1とBの間で自動改行させないためnbsp(~)にする +} + +export function formatDeltaBytes(deltaBytes: number, colorThreshold = 0) { + return formatColoredDelta(deltaBytes, formatBytes, colorThreshold); +} + +export function formatDeltaPercent(deltaPercent: number, colorThreshold = 0) { + return formatColoredDelta(deltaPercent, formatPercent, colorThreshold); +} + +/** + * Markdownのテーブルセル内に置く差分パーセント。 + * LaTeX由来の `\%` がMarkdownのエスケープで食われるため二重エスケープする。 + */ +export function formatDeltaPercentInMdTable(deltaPercent: number, colorThreshold = 0) { + return formatDeltaPercent(deltaPercent, colorThreshold).replaceAll('\\%', '\\\\%'); +} + +export function calcAndFormatDeltaNumber(before: number | null | undefined, after: number | null | undefined, colorThreshold = 0) { + if (before == null || after == null) return '-'; + return formatColoredDelta(after - before, formatNumber, colorThreshold); +} + +export function calcAndFormatDeltaBytes(before: number | null | undefined, after: number | null | undefined, colorThreshold = 0) { + if (before == null || after == null) return '-'; + return formatDeltaBytes(after - before, colorThreshold); +} + +export function calcAndFormatDeltaPercent(before: number | null | undefined, after: number | null | undefined, colorThreshold = 0) { + if (before == null || before === 0 || after == null) return '-'; + return formatDeltaPercent((after - before) / before * 100, colorThreshold); +} + +export function calcAndFormatDeltaPercentInMdTable(before: number | null | undefined, after: number | null | undefined, colorThreshold = 0) { + return calcAndFormatDeltaPercent(before, after, colorThreshold).replaceAll('\\%', '\\\\%'); +} diff --git a/packages-private/diagnostics-shared/src/heap-snapshot/analyze.ts b/packages-private/diagnostics-shared/src/heap-snapshot/analyze.ts new file mode 100644 index 0000000000..b3443e1889 --- /dev/null +++ b/packages-private/diagnostics-shared/src/heap-snapshot/analyze.ts @@ -0,0 +1,198 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { classifyHeapSnapshotBreakdown, collapseHeapSnapshotBreakdowns } from './breakdown'; +import { + createEmptyHeapSnapshotData, + heapSnapshotBreakdownCategories, + type HeapSnapshotCategory, + type HeapSnapshotData, +} from './categories'; + +/** + * `.heapsnapshot` のJSONを、フィールドオフセットを解決した状態で扱うためのビュー。 + */ +type HeapSnapshotView = { + nodes: number[]; + edges: number[]; + strings: string[]; + nodeFieldCount: number; + edgeFieldCount: number; + nodeTypeNames: string[]; + edgeTypeNames: string[]; + typeOffset: number; + nameOffset: number; + selfSizeOffset: number; + edgeCountOffset: number; + edgeTypeOffset: number; + edgeNameOffset: number; + edgeToNodeOffset: number; + extraNativeBytes: number; +}; + +function requireOffsets(fields: string[], names: string[], what: string) { + const offsets = names.map(name => fields.indexOf(name)); + if (offsets.some(offset => offset < 0)) throw new Error(`Heap snapshot is missing required ${what} fields`); + return offsets; +} + +function parseHeapSnapshot(snapshot: any): HeapSnapshotView { + const meta = snapshot?.snapshot?.meta; + const { nodes, edges, strings } = snapshot ?? {}; + if (meta == null || !Array.isArray(nodes) || !Array.isArray(edges) || !Array.isArray(strings)) { + throw new Error('Invalid heap snapshot format'); + } + + const nodeFields = meta.node_fields; + if (!Array.isArray(nodeFields)) throw new Error('Invalid heap snapshot node fields'); + const edgeFields = meta.edge_fields; + if (!Array.isArray(edgeFields)) throw new Error('Invalid heap snapshot edge fields'); + + const [typeOffset, nameOffset, selfSizeOffset, edgeCountOffset] = requireOffsets(nodeFields, ['type', 'name', 'self_size', 'edge_count'], 'node'); + const [edgeTypeOffset, edgeNameOffset, edgeToNodeOffset] = requireOffsets(edgeFields, ['type', 'name_or_index', 'to_node'], 'edge'); + + const nodeTypeNames = meta.node_types?.[typeOffset]; + if (!Array.isArray(nodeTypeNames)) throw new Error('Invalid heap snapshot node types'); + const edgeTypeNames = meta.edge_types?.[edgeTypeOffset]; + if (!Array.isArray(edgeTypeNames)) throw new Error('Invalid heap snapshot edge types'); + + return { + nodes, + edges, + strings, + nodeFieldCount: nodeFields.length, + edgeFieldCount: edgeFields.length, + nodeTypeNames, + edgeTypeNames, + typeOffset, + nameOffset, + selfSizeOffset, + edgeCountOffset, + edgeTypeOffset, + edgeNameOffset, + edgeToNodeOffset, + extraNativeBytes: Number.isFinite(snapshot.snapshot.extra_native_bytes) ? snapshot.snapshot.extra_native_bytes : 0, + }; +} + +/** + * ノードごとのedge開始位置と、各ノードが何本のedgeから参照されているかを求める。 + * JS配列のelementsストアが専有されているか判定するために使う。 + */ +function indexEdges(view: HeapSnapshotView) { + const edgeStartIndexes = new Map(); + const retainerCounts = new Map(); + let edgeIndex = 0; + + for (let nodeIndex = 0; nodeIndex < view.nodes.length; nodeIndex += view.nodeFieldCount) { + edgeStartIndexes.set(nodeIndex, edgeIndex); + const edgeCount = view.nodes[nodeIndex + view.edgeCountOffset] ?? 0; + for (let i = 0; i < edgeCount; i++, edgeIndex += view.edgeFieldCount) { + const toNodeIndex = view.edges[edgeIndex + view.edgeToNodeOffset]; + retainerCounts.set(toNodeIndex, (retainerCounts.get(toNodeIndex) ?? 0) + 1); + } + } + + return { edgeStartIndexes, retainerCounts }; +} + +export function analyzeHeapSnapshot(snapshot: any, options: { breakdownTopN?: number } = {}): HeapSnapshotData { + const view = parseHeapSnapshot(snapshot); + const { nodes, edges, strings, nodeFieldCount, edgeFieldCount, nodeTypeNames, edgeTypeNames } = view; + + const nativeType = nodeTypeNames.indexOf('native'); + const codeType = nodeTypeNames.indexOf('code'); + const hiddenType = nodeTypeNames.indexOf('hidden'); + const stringTypes = new Set([ + nodeTypeNames.indexOf('string'), + nodeTypeNames.indexOf('concatenated string'), + nodeTypeNames.indexOf('sliced string'), + ]); + const internalEdgeType = edgeTypeNames.indexOf('internal'); + + const { categories, nodeCounts } = createEmptyHeapSnapshotData(); + const breakdowns = {} as Record>; + for (const category of heapSnapshotBreakdownCategories) { + breakdowns[category] = {}; + } + + const { edgeStartIndexes, retainerCounts } = indexEdges(view); + const jsArrayElementNodeIndexes = new Set(); + + function addCategoryValue(category: HeapSnapshotCategory, value: number, type: string, name: string, counted = true) { + if (value <= 0) return; + categories[category] += value; + const label = classifyHeapSnapshotBreakdown(category, type, name); + breakdowns[category][label] = (breakdowns[category][label] ?? 0) + value; + if (counted) nodeCounts[category]++; + } + + /** + * 配列オブジェクト自身のself_sizeには要素ストアが含まれないため、 + * その配列だけが参照しているelementsノードの分を JS arrays に加算する。 + */ + function addJsArrayElementSize(nodeIndex: number) { + const beginEdgeIndex = edgeStartIndexes.get(nodeIndex) ?? 0; + const edgeCount = nodes[nodeIndex + view.edgeCountOffset] ?? 0; + for (let i = 0, currentEdgeIndex = beginEdgeIndex; i < edgeCount; i++, currentEdgeIndex += edgeFieldCount) { + if (edges[currentEdgeIndex + view.edgeTypeOffset] !== internalEdgeType) continue; + if (strings[edges[currentEdgeIndex + view.edgeNameOffset]] !== 'elements') continue; + + const elementsNodeIndex = edges[currentEdgeIndex + view.edgeToNodeOffset]; + if ((retainerCounts.get(elementsNodeIndex) ?? 0) === 1) { + addCategoryValue('jsArrays', nodes[elementsNodeIndex + view.selfSizeOffset] ?? 0, 'array elements', 'Array elements'); + jsArrayElementNodeIndexes.add(elementsNodeIndex); + } + break; + } + } + + if (view.extraNativeBytes > 0) { + addCategoryValue('otherNonJsObjects', view.extraNativeBytes, 'extra native bytes', 'extra native bytes', false); + } + + for (let nodeIndex = 0; nodeIndex < nodes.length; nodeIndex += nodeFieldCount) { + const typeId = nodes[nodeIndex + view.typeOffset]; + const type = nodeTypeNames[typeId] ?? 'unknown'; + const name = strings[nodes[nodeIndex + view.nameOffset]] ?? ''; + const selfSize = nodes[nodeIndex + view.selfSizeOffset] ?? 0; + categories.total += selfSize; + nodeCounts.total++; + + if (typeId === hiddenType) { + addCategoryValue('systemObjects', selfSize, type, name); + } else if (typeId === nativeType) { + addCategoryValue(name === 'system / JSArrayBufferData' ? 'typedArrays' : 'otherNonJsObjects', selfSize, type, name); + } else if (typeId === codeType) { + addCategoryValue('code', selfSize, type, name); + } else if (stringTypes.has(typeId)) { + addCategoryValue('strings', selfSize, type, name); + } else if (name === 'Array') { + addCategoryValue('jsArrays', selfSize, type, name); + addJsArrayElementSize(nodeIndex); + } + } + + categories.total += view.extraNativeBytes; + + // 上のループで JS arrays に計上したelementsノードが確定してから、残りを Other JS objects に振り分ける + for (let nodeIndex = 0; nodeIndex < nodes.length; nodeIndex += nodeFieldCount) { + if (jsArrayElementNodeIndexes.has(nodeIndex)) continue; + + const typeId = nodes[nodeIndex + view.typeOffset]; + if (typeId === hiddenType || typeId === nativeType || typeId === codeType || stringTypes.has(typeId)) continue; + + const name = strings[nodes[nodeIndex + view.nameOffset]] ?? ''; + if (name === 'Array') continue; + + addCategoryValue('otherJsObjects', nodes[nodeIndex + view.selfSizeOffset] ?? 0, nodeTypeNames[typeId] ?? 'unknown', name); + } + + return { + categories, + nodeCounts, + breakdowns: collapseHeapSnapshotBreakdowns(breakdowns, options.breakdownTopN), + }; +} diff --git a/packages-private/diagnostics-shared/src/heap-snapshot/breakdown.ts b/packages-private/diagnostics-shared/src/heap-snapshot/breakdown.ts new file mode 100644 index 0000000000..0dd5ffe324 --- /dev/null +++ b/packages-private/diagnostics-shared/src/heap-snapshot/breakdown.ts @@ -0,0 +1,94 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { + defaultHeapSnapshotBreakdownTopN, + heapSnapshotBreakdownCategories, + type HeapSnapshotCategory, + type HeapSnapshotData, +} from './categories'; + +function sanitizeLabel(value: unknown, fallback = 'unknown') { + const label = String(value ?? '').replace(/\s+/g, ' ').trim(); + if (label === '') return fallback; + if (label.length <= 80) return label; + return `${label.slice(0, 77)}...`; +} + +/** + * ノードの type / name から、内訳テーブルに出す粒度のラベルを決める。 + */ +export function classifyHeapSnapshotBreakdown(category: HeapSnapshotCategory, type: string, name: string) { + switch (category) { + case 'strings': + return type; + + case 'jsArrays': + if (type === 'array elements') return 'Array elements'; + if (type === 'object' && name === 'Array') return 'Array objects'; + return sanitizeLabel(`${type}: ${name}`); + + case 'typedArrays': + if (name === 'system / JSArrayBufferData') return 'ArrayBuffer data'; + return sanitizeLabel(`${type}: ${name}`); + + case 'systemObjects': + if (name.startsWith('system /') || name.startsWith('(system ')) return sanitizeLabel(name); + return sanitizeLabel(`${type}: ${name}`, type); + + case 'otherJsObjects': + if (type === 'object') return sanitizeLabel(`object: ${name}`, 'object: unknown'); + return type; + + case 'otherNonJsObjects': + if (type === 'extra native bytes') return 'Extra native bytes'; + if (type === 'native') return sanitizeLabel(`native: ${name}`, 'native: unknown'); + return sanitizeLabel(`${type}: ${name}`, type); + + case 'code': { + const lowerName = name.toLowerCase(); + if (lowerName.includes('bytecode')) return 'bytecode'; + if (lowerName.includes('builtin')) return 'builtins'; + if (lowerName.includes('regexp')) return 'regexp code'; + if (lowerName.includes('stub')) return 'stubs'; + return sanitizeLabel(`code: ${name}`, 'code: unknown'); + } + + default: + return sanitizeLabel(`${type}: ${name}`, type); + } +} + +/** + * 内訳を上位N件に丸め、残りを `Other` にまとめる。 + */ +export function collapseHeapSnapshotBreakdown(breakdown: Record, topN = defaultHeapSnapshotBreakdownTopN) { + const entries = Object.entries(breakdown) + .filter(([, value]) => value > 0) + .toSorted((a, b) => b[1] - a[1]); + + const collapsed = Object.fromEntries(entries.slice(0, topN)); + const otherValue = entries.slice(topN).reduce((sum, [, value]) => sum + value, 0); + if (otherValue > 0) collapsed.Other = otherValue; + return collapsed; +} + +export function collapseHeapSnapshotBreakdowns( + breakdowns: Partial>>, + topN = defaultHeapSnapshotBreakdownTopN, +) { + const collapsed: NonNullable = {}; + for (const category of heapSnapshotBreakdownCategories) { + const categoryBreakdown = breakdowns[category]; + if (categoryBreakdown == null) continue; + + const collapsedCategory = collapseHeapSnapshotBreakdown(categoryBreakdown, topN); + if (Object.keys(collapsedCategory).length > 0) { + collapsed[category] = collapsedCategory; + } + } + + return collapsed; +} diff --git a/packages-private/diagnostics-shared/src/heap-snapshot/categories.ts b/packages-private/diagnostics-shared/src/heap-snapshot/categories.ts new file mode 100644 index 0000000000..4e7caed6d8 --- /dev/null +++ b/packages-private/diagnostics-shared/src/heap-snapshot/categories.ts @@ -0,0 +1,54 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +// Chrome DevTools の heap snapshot Statistics ビューと同じ分類になるように保つこと。 +export const heapSnapshotCategory = { + total: { label: 'Total', color: 'gray', colorHex: '#888888' }, + code: { label: 'Code', color: 'orange', colorHex: '#f28e2c' }, + strings: { label: 'Strings', color: 'red', colorHex: '#e15759' }, + jsArrays: { label: 'JS arrays', color: 'cyan', colorHex: '#76b7b2' }, + typedArrays: { label: 'Typed arrays', color: 'green', colorHex: '#59a14f' }, + systemObjects: { label: 'System objects', color: 'yellow', colorHex: '#edc949' }, + otherJsObjects: { label: 'Other JS objs', color: 'violet', colorHex: '#af7aa1' }, + otherNonJsObjects: { label: 'Other non-JS objs', color: 'pink', colorHex: '#ff9da7' }, +} as const satisfies Record; + +export type HeapSnapshotCategory = keyof typeof heapSnapshotCategory; + +export const heapSnapshotCategories = Object.keys(heapSnapshotCategory) as HeapSnapshotCategory[]; + +/** `total` は他カテゴリの合算ではなく全体量なので、内訳を扱うときは除外する */ +export const heapSnapshotBreakdownCategories = heapSnapshotCategories.filter(category => category !== 'total'); + +export type HeapSnapshotData = { + categories: Record; + nodeCounts: Record; + /** 内訳が空でないカテゴリだけが入る (`total` は内訳を持たない) */ + breakdowns?: Partial>>; +}; + +export type HeapSnapshotReport = { + summary: HeapSnapshotData; + samples: { + round: number; + data: HeapSnapshotData; + }[]; +}; + +export const defaultHeapSnapshotBreakdownTopN = 6; + +export function createEmptyHeapSnapshotData(): HeapSnapshotData { + const categories = {} as HeapSnapshotData['categories']; + const nodeCounts = {} as HeapSnapshotData['nodeCounts']; + for (const category of heapSnapshotCategories) { + categories[category] = 0; + nodeCounts[category] = 0; + } + return { + categories, + nodeCounts, + breakdowns: {}, + }; +} diff --git a/packages-private/diagnostics-shared/src/heap-snapshot/index.ts b/packages-private/diagnostics-shared/src/heap-snapshot/index.ts new file mode 100644 index 0000000000..0634efac87 --- /dev/null +++ b/packages-private/diagnostics-shared/src/heap-snapshot/index.ts @@ -0,0 +1,10 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +export * from './analyze'; +export * from './breakdown'; +export * from './categories'; +export * from './render'; +export * from './summarize'; diff --git a/packages-private/diagnostics-shared/src/heap-snapshot/render.ts b/packages-private/diagnostics-shared/src/heap-snapshot/render.ts new file mode 100644 index 0000000000..f1f1ea30f5 --- /dev/null +++ b/packages-private/diagnostics-shared/src/heap-snapshot/render.ts @@ -0,0 +1,145 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { formatBytes } from '../format'; +import { renderMetricComparisonTable } from '../metric-table'; +import { + heapSnapshotCategories, + heapSnapshotCategory, + type HeapSnapshotCategory, + type HeapSnapshotReport, +} from './categories'; + +/** これ未満のバイト差分には色を付けない (0.1 MB) */ +const byteColorThreshold = 100_000; + +function categoryValue(report: HeapSnapshotReport, category: HeapSnapshotCategory) { + return report.summary.categories[category]; +} + +function swatch(category: HeapSnapshotCategory) { + // eslint-disable-next-line no-irregular-whitespace + return `$\\color{${heapSnapshotCategory[category].color}}{\\rule{8pt}{8pt}}$ **${heapSnapshotCategory[category].label.replaceAll(' ', ' ')}**`; // nbspにすること +} + +/** + * base / head のheap snapshotをカテゴリ別に比較するMarkdownテーブルを描画する。 + */ +export function renderHeapSnapshotTable(base: HeapSnapshotReport, head: HeapSnapshotReport) { + return renderMetricComparisonTable( + base.samples, + head.samples, + heapSnapshotCategories.map(category => ({ + label: swatch(category), + getValue: sample => sample.data.categories[category], + formatValue: formatBytes, + absoluteThreshold: byteColorThreshold, + showMedianMad: category === 'total', + showDeltaPercentage: category === 'total', + separatorAfter: category === 'total', + })), + ); +} + +const sankeyChildMinRatio = 0.3; +const sankeyParentMinPercent = 10; + +function escapeCsvValue(value: string) { + return `"${String(value).replaceAll('"', '""')}"`; +} + +function formatSankeyPercentValue(value: number) { + const rounded = Math.round(value * 100) / 100; + if (rounded === 0 && value > 0) return '0.01'; + if (Number.isInteger(rounded)) return String(rounded); + return rounded.toFixed(2).replace(/0+$/, '').replace(/\.$/, ''); +} + +/** + * heap snapshotの構成比をmermaidのsankey図として描画する。 + * 全体に占める割合が小さいカテゴリ・内訳は `Other` にまとめる。 + */ +export function renderHeapSnapshotSankey(report: HeapSnapshotReport, title: string) { + const total = categoryValue(report, 'total'); + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (total == null || total <= 0) return null; + + const categories = heapSnapshotCategories + .filter(category => category !== 'total') + .map(category => { + const value = categoryValue(report, category); + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (value == null || value <= 0) return null; + + const breakdownEntries = Object.entries(report.summary.breakdowns?.[category] ?? {}) + .filter(([, childValue]) => Number.isFinite(childValue) && childValue > 0) + .toSorted((a, b) => b[1] - a[1]); + const breakdownTotal = breakdownEntries.reduce((sum, [, childValue]) => sum + childValue, 0); + const percent = (value * 100) / total; + const childEntries: [string, number][] = []; + let otherPercent = 0; + + if (breakdownTotal > 0 && percent > sankeyParentMinPercent) { + for (const [childName, childValue] of breakdownEntries) { + const childRatio = childValue / breakdownTotal; + if (childRatio >= sankeyChildMinRatio) { + childEntries.push([childName.replace(/^[^:]+:\s*/, ''), percent * childRatio]); + } else { + otherPercent += percent * childRatio; + } + } + + if (childEntries.length > 0 && otherPercent > 0) { + childEntries.push(['Other', otherPercent]); + } + } + + return { category, percent, childEntries }; + }) + .filter(value => value != null); + + if (categories.length === 0) return null; + + const nodeColors: Record = { + [title]: heapSnapshotCategory.total.colorHex, + Other: '#888888', + }; + for (const { category, childEntries } of categories) { + nodeColors[category] = heapSnapshotCategory[category].colorHex; + for (const [childName] of childEntries) { + nodeColors[childName] = heapSnapshotCategory[category].colorHex; + } + } + + const lines = [ + `
${title} heap snapshot composition`, + '', + '```mermaid', + `%%{init: ${JSON.stringify({ + sankey: { + showValues: false, + linkColor: 'target', + labelStyle: 'outlined', + nodeAlignment: 'center', + nodePadding: 10, + nodeColors, + }, + })}}%%`, + 'sankey-beta', + ]; + + for (const { category, percent, childEntries } of categories) { + const categoryLabel = heapSnapshotCategory[category].label; + lines.push(`${escapeCsvValue(title)},${escapeCsvValue(categoryLabel)},${formatSankeyPercentValue(percent)}`); + + for (const [childName, childPercent] of childEntries) { + lines.push(`${escapeCsvValue(categoryLabel)},${escapeCsvValue(childName)},${formatSankeyPercentValue(childPercent)}`); + } + } + + lines.push('```', '', '
'); + + return lines.join('\n'); +} diff --git a/packages-private/diagnostics-shared/src/heap-snapshot/summarize.ts b/packages-private/diagnostics-shared/src/heap-snapshot/summarize.ts new file mode 100644 index 0000000000..bbc2a6d203 --- /dev/null +++ b/packages-private/diagnostics-shared/src/heap-snapshot/summarize.ts @@ -0,0 +1,63 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { finiteMedian } from '../stats'; +import { collapseHeapSnapshotBreakdown } from './breakdown'; +import { + heapSnapshotBreakdownCategories, + heapSnapshotCategories, + type HeapSnapshotCategory, + type HeapSnapshotData, +} from './categories'; + +function isComplete(values: Partial>): values is Record { + return heapSnapshotCategories.every(category => values[category] != null); +} + +/** + * 複数ラウンド分のheap snapshotを、カテゴリ・内訳ごとの中央値にまとめる。 + * 全カテゴリ分の値が揃わなければ null を返す。 + */ +export function summarizeHeapSnapshotDataSamples( + samples: T[], + getData: (sample: T) => HeapSnapshotData | null | undefined, + options: { breakdownTopN?: number } = {}, +) { + const data = samples.map(getData); + + const categories: Partial = {}; + const nodeCounts: Partial = {}; + for (const category of heapSnapshotCategories) { + const categoryValue = finiteMedian(data.map(snapshot => snapshot?.categories?.[category])); + if (categoryValue != null) categories[category] = categoryValue; + + const nodeCountValue = finiteMedian(data.map(snapshot => snapshot?.nodeCounts?.[category])); + if (nodeCountValue != null) nodeCounts[category] = nodeCountValue; + } + + // 一部のカテゴリだけ欠けた状態で返すと、呼び出し側が完全な値として扱って + // undefined を描画してしまう。全カテゴリ揃っていなければサマリ自体を無しとする + if (!isComplete(categories) || !isComplete(nodeCounts)) return null; + + const breakdowns: NonNullable = {}; + for (const category of heapSnapshotBreakdownCategories) { + const childKeys = new Set(data.flatMap(snapshot => Object.keys(snapshot?.breakdowns?.[category] ?? {}))); + + const categoryBreakdown = {} as Record; + for (const childKey of childKeys) { + const value = finiteMedian(data.map(snapshot => snapshot?.breakdowns?.[category]?.[childKey])); + if (value != null) categoryBreakdown[childKey] = value; + } + + const collapsed = collapseHeapSnapshotBreakdown(categoryBreakdown, options.breakdownTopN); + if (Object.keys(collapsed).length > 0) breakdowns[category] = collapsed; + } + + return { + categories, + nodeCounts, + ...(Object.keys(breakdowns).length > 0 ? { breakdowns } : {}), + } satisfies HeapSnapshotData; +} diff --git a/packages-private/diagnostics-shared/src/html.ts b/packages-private/diagnostics-shared/src/html.ts new file mode 100644 index 0000000000..907c089656 --- /dev/null +++ b/packages-private/diagnostics-shared/src/html.ts @@ -0,0 +1,58 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { escapeHtml } from './format'; + +/** + * エスケープ済み、あるいは意図的にエスケープしない生のHTML断片。 + * + * ただの文字列と型で区別するためだけの存在ではなく、`html` が実行時に + * 「この値はもうエスケープしなくてよい」と判定するためのマーカーでもある。 + * 型だけのブランドにすると実行時に判定できず、結局エスケープ漏れを防げない。 + */ +export class Raw { + constructor(private readonly value: string) {} + + toString() { + return this.value; + } +} + +/** + * 文字列をエスケープせずそのまま埋め込む。 + * 呼び出しが差分に残るので、レビューで「なぜ生で入れてよいのか」を確認できる。 + */ +export function raw(value: string) { + return new Raw(value); +} + +function interpolate(value: unknown): string { + if (value instanceof Raw) return value.toString(); + // 配列をそのまま文字列化するとカンマ区切りで潰れるので、要素ごとに処理する + if (Array.isArray(value)) return value.map(interpolate).join(''); + if (value == null) return ''; + return escapeHtml(value); +} + +/** + * HTMLを組み立てるタグ付きテンプレート。補間値は既定でエスケープされる。 + * エスケープしたくない場合は `raw()` で包む必要があるため、 + * 「うっかり生のまま埋め込む」ことが起きない。 + */ +export function html(strings: TemplateStringsArray, ...values: unknown[]) { + let result = strings[0]; + for (let i = 0; i < values.length; i++) { + result += interpolate(values[i]) + strings[i + 1]; + } + return new Raw(result); +} + +/** + * 断片を指定の区切り文字で連結する。 + * `html` の配列補間は区切り無しで繋ぐので、改行などを挟みたいときはこちらを使う。 + */ +export function joinHtml(parts: readonly Raw[], separator: string) { + return new Raw(parts.map(part => part.toString()).join(separator)); +} diff --git a/packages-private/diagnostics-shared/src/metric-table.ts b/packages-private/diagnostics-shared/src/metric-table.ts new file mode 100644 index 0000000000..e9c55d48b2 --- /dev/null +++ b/packages-private/diagnostics-shared/src/metric-table.ts @@ -0,0 +1,86 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { formatColoredDelta, formatDeltaPercentInMdTable } from './format'; +import { + independentDeltaSummary, + isOutsideObservedNoise, + type IndependentDeltaSummary, +} from './stats'; + +export type MetricComparisonRow = { + label: string; + getValue: (sample: T) => number; + formatValue: (value: number) => string; + absoluteThreshold: number; + showMedianMad?: boolean; + showDeltaPercentage?: boolean; + separatorAfter?: boolean; +}; + +export type MetricComparisonTableOptions = { + onlySignificantChanges?: boolean; +}; + +function isSignificant(summary: IndependentDeltaSummary, absoluteThreshold: number) { + return isOutsideObservedNoise(summary) && Math.abs(summary.delta) >= absoluteThreshold; +} + +function formatMedian( + value: number, + spread: number, + row: MetricComparisonRow, +) { + const formatted = row.formatValue(value); + if (row.showMedianMad === false) return formatted; + // eslint-disable-next-line no-irregular-whitespace + return `${formatted}
± ${row.formatValue(spread)}`; // nbspにすること +} + +function formatDelta( + summary: IndependentDeltaSummary, + row: MetricComparisonRow, + significant: boolean, +) { + const colorThreshold = significant ? 0 : Number.POSITIVE_INFINITY; + const absolute = formatColoredDelta(summary.delta, row.formatValue, colorThreshold); + if (row.showDeltaPercentage === false) return absolute; + + const percentage = summary.baseMedian === 0 + ? '-' + : formatDeltaPercentInMdTable(summary.delta * 100 / summary.baseMedian, colorThreshold); + return `${absolute}
${percentage}`; +} + +export function renderMetricComparisonTable( + baseSamples: T[], + headSamples: T[], + rows: MetricComparisonRow[], + options: MetricComparisonTableOptions = {}, +): string { + const lines: string[] = []; + let omitted = false; + + for (const row of rows) { + const summary = independentDeltaSummary(baseSamples, headSamples, row.getValue); + const significant = isSignificant(summary, row.absoluteThreshold); + if (options.onlySignificantChanges === true && !significant) { + omitted = true; + continue; + } + + lines.push(`| ${row.label} | ${formatMedian(summary.baseMedian, summary.baseMad, row)} | ${formatMedian(summary.headMedian, summary.headMad, row)} | ${formatDelta(summary, row, significant)} | ${row.formatValue(summary.combinedMad)} |`); + if (row.separatorAfter === true) lines.push('| | | | | |'); + } + + if (lines.length === 0) return '**(No significant changes)**'; + + return [ + '| Metric | @ Base | @ Head | Δ | MAD |', + '| --- | ---: | ---: | ---: | ---: |', + ...lines, + ...(omitted ? ['', 'Only metrics showing significant changes are displayed.'] : []), + ].join('\n'); +} diff --git a/packages-private/diagnostics-shared/src/stats.ts b/packages-private/diagnostics-shared/src/stats.ts new file mode 100644 index 0000000000..20fd7b1347 --- /dev/null +++ b/packages-private/diagnostics-shared/src/stats.ts @@ -0,0 +1,127 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +export function median(values: number[]) { + const sorted = values.toSorted((a, b) => a - b); + const center = Math.floor(sorted.length / 2); + if (sorted.length % 2 === 1) return sorted[center]; + return Math.round((sorted[center - 1] + sorted[center]) / 2); +} + +export function mad(values: number[]) { + if (values.length < 2) throw new Error('Not enough samples to calculate MAD'); + + const center = median(values); + return median(values.map(value => Math.abs(value - center))); +} + +/** + * 有限値のみを対象に中央値を求める。有限値が1つも無い場合は `defaultValue` を返す。 + */ +export function finiteMedian(values: (number | null | undefined)[]): number | null; +export function finiteMedian(values: (number | null | undefined)[], defaultValue: number): number; +export function finiteMedian(values: (number | null | undefined)[], defaultValue: number | null = null) { + const finiteValues = values.filter(value => Number.isFinite(value)) as number[]; + if (finiteValues.length === 0) return defaultValue; + return median(finiteValues); +} + +/** + * サンプルのばらつき (MAD) を求める。サンプルが2つ未満で求められない場合は null を返す。 + */ +export function sampleSpread(values: (number | null | undefined)[]) { + const finiteValues = values.filter(value => Number.isFinite(value)) as number[]; + if (finiteValues.length < 2) return null; + return mad(finiteValues); +} + +export type IndependentDeltaSummary = { + baseMedian: number; + headMedian: number; + delta: number; + baseMad: number; + headMad: number; + combinedMad: number; + baseSamples: number; + headSamples: number; +}; + +export function independentDeltaSummary( + baseSamples: T[], + headSamples: T[], + getValue: (sample: T) => number, +): IndependentDeltaSummary { + const baseValues = baseSamples.map(getValue); + const headValues = headSamples.map(getValue); + if (baseValues.length < 2 || headValues.length < 2) { + throw new Error('At least two samples per side are required'); + } + + const baseMedian = median(baseValues); + const headMedian = median(headValues); + const baseMad = mad(baseValues); + const headMad = mad(headValues); + + return { + baseMedian, + headMedian, + delta: headMedian - baseMedian, + baseMad, + headMad, + combinedMad: Math.hypot(baseMad, headMad), + baseSamples: baseValues.length, + headSamples: headValues.length, + }; +} + +export function isOutsideObservedNoise(summary: IndependentDeltaSummary) { + return Math.abs(summary.delta) > summary.combinedMad * 3; +} + +type RoundedSample = { round: number }; + +function indexByRound(samples: T[]) { + const samplesByRound = new Map(); + for (const sample of samples) { + // 負のroundはwarmupを表すため対象外 + if (sample.round <= 0) continue; + samplesByRound.set(sample.round, sample); + } + return samplesByRound; +} + +/** + * base / head を同じroundどうしで突き合わせ、その差分の分布を要約する。 + * 実行順による揺らぎの影響を抑えるため、単純な集計値どうしの引き算ではなくペア差分を使う。 + */ +export function pairedDeltaSummary(baseSamples: T[], headSamples: T[], getValue: (sample: T) => number | null | undefined) { + const baseSamplesByRound = indexByRound(baseSamples); + const headSamplesByRound = indexByRound(headSamples); + const values: number[] = []; + + for (const [round, baseSample] of baseSamplesByRound) { + const headSample = headSamplesByRound.get(round); + if (headSample == null) continue; + + const baseValue = getValue(baseSample); + const headValue = getValue(headSample); + if (baseValue == null || headValue == null) continue; + + values.push(headValue - baseValue); + } + + // 対応するroundが1つも無いと中央値も最小/最大も定義できない。 + // 静かにNaNやInfinityをレポートに載せるより、比較が成立していないと分かる形で落とす + if (values.length === 0) throw new Error('No paired samples to compare: base and head have no rounds in common'); + + return { + median: median(values), + // 1サンプルでは中央値からの偏差が常に0になる (mad() は統計として無意味なので拒否する) + mad: values.length < 2 ? 0 : mad(values), + min: Math.min(...values), + max: Math.max(...values), + samples: values.length, + }; +} diff --git a/packages-private/diagnostics-shared/test/format.test.ts b/packages-private/diagnostics-shared/test/format.test.ts new file mode 100644 index 0000000000..51cb5940f1 --- /dev/null +++ b/packages-private/diagnostics-shared/test/format.test.ts @@ -0,0 +1,102 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test } from 'vitest'; +import { + calcAndFormatDeltaPercent, + calcAndFormatDeltaPercentInMdTable, + escapeHtml, + escapeLatex, + escapeMdTableCell, + formatBytes, + formatColoredDelta, + formatKiBAsMb, + formatMs, + formatNumber, +} from '../src/format'; + +describe('formatBytes', () => { + // 1024ではなく1000区切り。単位が2桁以上なら小数を落とす + test.each([ + [0, '0 B'], + [999, '999 B'], + [1_000, '1 KB'], + [1_500, '1.5 KB'], + [15_000, '15 KB'], + [1_234_567, '1.2 MB'], + [12_345_678, '12 MB'], + [1_500_000_000, '1.5 GB'], + [1_500_000_000_000, '1,500 GB'], + ])('formats %i as %s', (input, expected) => { + expect(formatBytes(input)).toBe(expected); + }); +}); + +describe('formatColoredDelta', () => { + test('leaves zero uncoloured and unsigned', () => { + expect(formatColoredDelta(0, formatNumber)).toBe('0'); + }); + + test('colours growth orange and shrinkage green', () => { + expect(formatColoredDelta(5, formatNumber)).toBe('$\\color{orange}{\\text{+5}}$'); + expect(formatColoredDelta(-5, formatNumber)).toBe('$\\color{green}{\\text{-5}}$'); + }); + + test('omits colour below the threshold but keeps the sign', () => { + expect(formatColoredDelta(5, formatNumber, 10)).toBe('$\\text{+5}$'); + }); +}); + +describe('escaping', () => { + test('escapeHtml covers the five metacharacters', () => { + expect(escapeHtml(`&<>"'`)).toBe('&<>"''); + }); + + test('escapeHtml maps nullish to an empty string', () => { + expect(escapeHtml(null)).toBe(''); + expect(escapeHtml(undefined)).toBe(''); + }); + + test('escapeLatex escapes braces and percent', () => { + expect(escapeLatex('100% {x}')).toBe('100\\% \\{x\\}'); + }); + + test('escapeMdTableCell keeps pipes and newlines from breaking the table', () => { + expect(escapeMdTableCell('a|b\nc')).toBe('a\\|b
c'); + }); +}); + +describe('percent helpers', () => { + // before が0だと変化率そのものが定義できない + test('returns a placeholder when the baseline is zero or missing', () => { + expect(calcAndFormatDeltaPercent(0, 10)).toBe('-'); + expect(calcAndFormatDeltaPercent(null, 10)).toBe('-'); + expect(calcAndFormatDeltaPercent(10, null)).toBe('-'); + }); + + // 0になったのは「消えた」という有効な結果なので、隠さず -100% として出す + test('formats a drop to zero as -100%', () => { + expect(calcAndFormatDeltaPercent(10, 0)).toBe('$\\color{green}{\\text{-100\\%}}$'); + }); + + // Markdownのテーブルセル内ではLaTeXの \% がさらに食われるため二重にする + test('doubles the percent escape for markdown table cells', () => { + expect(calcAndFormatDeltaPercentInMdTable(100, 110)).toContain('\\\\%'); + expect(calcAndFormatDeltaPercent(100, 110)).not.toContain('\\\\%'); + }); +}); + +describe('nullish handling', () => { + test('formatKiBAsMb and formatMs render a dash for missing values', () => { + expect(formatKiBAsMb(null)).toBe('-'); + expect(formatKiBAsMb(Number.NaN)).toBe('-'); + expect(formatMs(undefined)).toBe('-'); + }); + + test('formatMs switches to seconds past 1000ms', () => { + expect(formatMs(999)).toBe('999 ms'); + expect(formatMs(1_500)).toBe('1.5 s'); + }); +}); diff --git a/packages-private/diagnostics-shared/test/heap-snapshot-render.test.ts b/packages-private/diagnostics-shared/test/heap-snapshot-render.test.ts new file mode 100644 index 0000000000..9fd9bdad14 --- /dev/null +++ b/packages-private/diagnostics-shared/test/heap-snapshot-render.test.ts @@ -0,0 +1,103 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test } from 'vitest'; +import { + createEmptyHeapSnapshotData, + renderHeapSnapshotTable, + summarizeHeapSnapshotDataSamples, + type HeapSnapshotData, + type HeapSnapshotReport, +} from '../src/heap-snapshot'; + +function snapshot(total: number): HeapSnapshotData { + const data = createEmptyHeapSnapshotData(); + data.categories.total = total; + return data; +} + +function report(totals: number[]): HeapSnapshotReport { + const samples = totals.map((total, index) => ({ + round: index + 1, + data: snapshot(total), + })); + const summary = summarizeHeapSnapshotDataSamples(samples, sample => sample.data); + if (summary == null) throw new Error('expected heap snapshot samples to produce a summary'); + + return { + summary, + samples, + }; +} + +function totalRow(markdown: string) { + const row = markdown.split('\n').find(line => line.includes('**Total**')); + if (row === undefined) throw new Error('expected heap snapshot table to contain a Total row'); + return row; +} + +describe('renderHeapSnapshotTable', () => { + test('leaves a threshold-sized delta uncoloured when it is within observed noise', () => { + const row = totalRow(renderHeapSnapshotTable( + report([1_000_000, 1_100_000, 1_200_000]), + report([1_100_000, 1_200_000, 1_300_000]), + )); + + expect(row).toContain('$\\text{+100 KB}$'); + expect(row).not.toContain('within noise'); + expect(row).not.toContain('\\color{orange}'); + }); + + test('colours both delta lines for a clear increase at or above the absolute threshold', () => { + const row = totalRow(renderHeapSnapshotTable( + report([1_000_000, 1_000_000, 1_000_000]), + report([1_200_000, 1_200_000, 1_200_000]), + )); + + expect(row).toContain('$\\color{orange}{\\text{+200 KB}}$'); + expect(row).toContain('$\\color{orange}{\\text{+20\\\\%}}$'); + expect(row).not.toContain('increase'); + }); + + test('leaves both delta lines uncoloured below the absolute threshold', () => { + const row = totalRow(renderHeapSnapshotTable( + report([1_000_000, 1_000_000, 1_000_000]), + report([1_050_000, 1_050_000, 1_050_000]), + )); + + expect(row).toContain('$\\text{+50 KB}$
$\\text{+5\\\\%}$'); + expect(row.split('|')[4]).not.toContain('\\color{'); + }); + + test('throws when only one snapshot per side reaches the renderer', () => { + expect(() => renderHeapSnapshotTable( + report([1_000_000]), + report([1_200_000]), + )).toThrow('At least two samples per side are required'); + }); + + test('uses two-line formatting only for Total and puts a five-column separator after it', () => { + const table = renderHeapSnapshotTable( + report([1_000_000, 1_000_000, 1_000_000]), + report([1_200_000, 1_200_000, 1_200_000]), + ); + const lines = table.split('\n'); + const totalIndex = lines.findIndex(line => line.includes('**Total**')); + const categoryRow = lines.find(line => line.includes('**Code**')); + + expect(lines.slice(0, 2)).toStrictEqual([ + '| Metric | @ Base | @ Head | Δ | MAD |', + '| --- | ---: | ---: | ---: | ---: |', + ]); + expect(table).not.toContain('Result'); + expect(totalIndex).toBeGreaterThan(1); + expect(lines[totalIndex].match(/
/g)).toHaveLength(3); + expect(lines[totalIndex + 1]).toBe('| | | | | |'); + expect(categoryRow).toBeDefined(); + expect(categoryRow).not.toContain('
'); + expect(categoryRow).not.toContain('
'); + expect(categoryRow).not.toContain('→'); + }); +}); diff --git a/packages-private/diagnostics-shared/test/html.test.ts b/packages-private/diagnostics-shared/test/html.test.ts new file mode 100644 index 0000000000..6e9dbc48b0 --- /dev/null +++ b/packages-private/diagnostics-shared/test/html.test.ts @@ -0,0 +1,63 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test } from 'vitest'; +import { html, joinHtml, raw, Raw } from '../src/html'; + +describe('html', () => { + test('escapes interpolated values by default', () => { + const value = ''; + expect(String(html`

${value}

`)).toBe('

<script>alert(1)</script>

'); + }); + + test('escapes values used in attribute position', () => { + const url = 'x">'; + expect(String(html``)).toBe(''); + }); + + test('leaves the static parts of the template untouched', () => { + expect(String(html`

x

`)).toBe('

x

'); + }); + + test('renders nullish values as an empty string', () => { + expect(String(html`

${null}${undefined}

`)).toBe('

'); + }); + + test('does not double-escape nested fragments', () => { + const inner = html`${'a&b'}`; + expect(String(html`

${inner}

`)).toBe('

a&b

'); + }); + + // 配列をそのまま文字列化するとカンマ区切りで潰れる。実際に過去これで表が壊れた + test('joins arrays without separators instead of stringifying them', () => { + const items = [html`
  • 1
    • `, html`
  • 2
    • `]; + expect(String(html`
      ${items}
    `)).toBe('
    • 1
    • 2
    '); + }); + + test('escapes array elements that are not fragments', () => { + expect(String(html`

    ${['', '']}

    `)).toBe('

    <a><b>

    '); + }); +}); + +describe('raw', () => { + test('embeds trusted markup without escaping', () => { + expect(String(html``)).toBe(''); + }); + + test('produces a Raw fragment', () => { + expect(raw('x')).toBeInstanceOf(Raw); + expect(html`x`).toBeInstanceOf(Raw); + }); +}); + +describe('joinHtml', () => { + test('joins fragments with the given separator', () => { + expect(String(joinHtml([html`
  • 1
    • `, html`
  • 2
    • `], '\n'))).toBe('
  • 1
    • \n
  • 2
    • '); + }); + + test('returns an empty fragment for an empty list', () => { + expect(String(joinHtml([], '\n'))).toBe(''); + }); +}); diff --git a/packages-private/diagnostics-shared/test/metric-table.test.ts b/packages-private/diagnostics-shared/test/metric-table.test.ts new file mode 100644 index 0000000000..c414d5111f --- /dev/null +++ b/packages-private/diagnostics-shared/test/metric-table.test.ts @@ -0,0 +1,145 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test } from 'vitest'; +import { + renderMetricComparisonTable, + type MetricComparisonRow, +} from '../src/metric-table'; + +type Sample = { value: number }; + +const defaultRow: MetricComparisonRow = { + label: 'Metric', + getValue: sample => sample.value, + formatValue: value => `${value} units`, + absoluteThreshold: 10, +}; + +function samples(...values: number[]): Sample[] { + return values.map(value => ({ value })); +} + +describe('renderMetricComparisonTable', () => { + test('renders the fixed five-column layout, raw label, MAD, and percentage by default', () => { + const table = renderMetricComparisonTable( + samples(100, 100, 100), + samples(120, 120, 120), + [defaultRow], + ); + + expect(table.split('\n')).toStrictEqual([ + '| Metric | @ Base | @ Head | Δ | MAD |', + '| --- | ---: | ---: | ---: | ---: |', + '| Metric | 100 units
    ± 0 units | 120 units
    ± 0 units | $\\color{orange}{\\text{+20~units}}$
    $\\color{orange}{\\text{+20\\\\%}}$ | 0 units |', + ]); + }); + + test('can hide second lines and insert a five-column separator row', () => { + const table = renderMetricComparisonTable( + samples(100, 100, 100), + samples(120, 120, 120), + [{ + ...defaultRow, + showMedianMad: false, + showDeltaPercentage: false, + separatorAfter: true, + }], + ); + + expect(table.split('\n')).toStrictEqual([ + '| Metric | @ Base | @ Head | Δ | MAD |', + '| --- | ---: | ---: | ---: | ---: |', + '| Metric | 100 units | 120 units | $\\color{orange}{\\text{+20~units}}$ | 0 units |', + '| | | | | |', + ]); + }); + + test('leaves both delta lines uncolored below the absolute threshold and can filter the row', () => { + const base = samples(100, 100, 100); + const head = samples(109, 109, 109); + const table = renderMetricComparisonTable(base, head, [defaultRow]); + + expect(table).toContain('$\\text{+9~units}$
    $\\text{+9\\\\%}$'); + expect(table).not.toContain('\\color{'); + expect(renderMetricComparisonTable(base, head, [defaultRow], { + onlySignificantChanges: true, + })).toBe('**(No significant changes)**'); + }); + + test('renders a no-data marker when no rows are configured', () => { + expect(renderMetricComparisonTable( + samples(100, 100, 100), + samples(120, 120, 120), + [], + )).toBe('**(No significant changes)**'); + }); + + test('treats the absolute threshold itself as significant', () => { + const table = renderMetricComparisonTable( + samples(100, 100, 100), + samples(110, 110, 110), + [defaultRow], + { onlySignificantChanges: true }, + ); + + expect(table).toContain('$\\color{orange}{\\text{+10~units}}$'); + expect(table).toContain('$\\color{orange}{\\text{+10\\\\%}}$'); + }); + + test('requires the delta to be strictly outside three combined MADs', () => { + const inside = renderMetricComparisonTable( + samples(100, 110, 120), + samples(109, 119, 129), + [{ ...defaultRow, absoluteThreshold: 1 }], + ); + const boundary = renderMetricComparisonTable( + samples(100, 100, 100), + samples(120, 130, 140), + [defaultRow], + ); + const outside = renderMetricComparisonTable( + samples(100, 100, 100), + samples(121, 131, 141), + [defaultRow], + ); + + expect(inside).toContain('$\\text{+9~units}$'); + expect(inside).not.toContain('\\color{'); + expect(boundary).toContain('$\\text{+30~units}$'); + expect(boundary).not.toContain('\\color{'); + expect(outside).toContain('$\\color{orange}{\\text{+31~units}}$'); + expect(outside).toContain('$\\color{orange}{\\text{+31\\\\%}}$'); + }); + + test('colours a significant decrease green on both delta lines', () => { + const table = renderMetricComparisonTable( + samples(120, 120, 120), + samples(100, 100, 100), + [defaultRow], + ); + + expect(table).toContain('$\\color{green}{\\text{-20~units}}$'); + expect(table).toContain('$\\color{green}{\\text{-16.7\\\\%}}$'); + }); + + test('shows a dash for percentage when the base median is zero', () => { + const table = renderMetricComparisonTable( + samples(0, 0, 0), + samples(20, 20, 20), + [defaultRow], + ); + + expect(table).toContain('$\\color{orange}{\\text{+20~units}}$
    -'); + }); + + test('propagates the minimum sample contract', () => { + expect(() => renderMetricComparisonTable( + samples(100), + samples(120, 120), + [defaultRow], + )).toThrow('At least two samples per side are required'); + }); +}); diff --git a/packages-private/diagnostics-shared/test/stats.test.ts b/packages-private/diagnostics-shared/test/stats.test.ts new file mode 100644 index 0000000000..22da6a6531 --- /dev/null +++ b/packages-private/diagnostics-shared/test/stats.test.ts @@ -0,0 +1,194 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test } from 'vitest'; +import { + finiteMedian, + independentDeltaSummary, + isOutsideObservedNoise, + mad, + median, + pairedDeltaSummary, + sampleSpread, + type IndependentDeltaSummary, +} from '../src/stats'; + +describe('median', () => { + test('takes the middle value of an odd-length sample', () => { + expect(median([3, 1, 2])).toBe(2); + }); + + // 偶数長では平均を整数に丸める。KiB単位の整数値を扱う前提のため + test('rounds the average of an even-length sample', () => { + expect(median([1, 2])).toBe(2); + expect(median([1, 4])).toBe(3); + }); +}); + +describe('mad', () => { + test('measures the median absolute deviation', () => { + expect(mad([1, 1, 1])).toBe(0); + expect(mad([1, 2, 3])).toBe(1); + }); + + test('refuses to compute from a single sample', () => { + expect(() => mad([1])).toThrow(); + }); +}); + +describe('finiteMedian', () => { + test('ignores non-finite entries', () => { + expect(finiteMedian([1, null, undefined, Number.NaN, 3])).toBe(2); + }); + + test('returns null by default when nothing is finite', () => { + expect(finiteMedian([null, undefined])).toBeNull(); + }); + + test('returns the supplied default when nothing is finite', () => { + expect(finiteMedian([null, undefined], 0)).toBe(0); + }); +}); + +describe('sampleSpread', () => { + test('needs at least two finite samples', () => { + expect(sampleSpread([1])).toBeNull(); + expect(sampleSpread([1, null])).toBeNull(); + expect(sampleSpread([1, 3])).toBe(1); + }); +}); + +describe('pairedDeltaSummary', () => { + const base = [ + { round: 1, value: 100 }, + { round: 2, value: 200 }, + { round: 3, value: 300 }, + ]; + + test('compares base and head within the same round', () => { + const head = [ + { round: 1, value: 110 }, + { round: 2, value: 230 }, + { round: 3, value: 320 }, + ]; + + expect(pairedDeltaSummary(base, head, sample => sample.value)).toStrictEqual({ + median: 20, + mad: 10, + min: 10, + max: 30, + samples: 3, + }); + }); + + test('drops rounds that only one side has', () => { + const head = [ + { round: 1, value: 110 }, + { round: 2, value: 230 }, + { round: 9, value: 999 }, + ]; + + expect(pairedDeltaSummary(base, head, sample => sample.value).samples).toBe(2); + }); + + // 1サンプルでも中央値・最小・最大は定まる (偏差は常に0) + test('summarizes a single paired round without treating MAD as an error', () => { + expect(pairedDeltaSummary([base[0]], [{ round: 1, value: 130 }], sample => sample.value)).toStrictEqual({ + median: 30, + mad: 0, + min: 30, + max: 30, + samples: 1, + }); + }); + + test('fails loudly when no round is shared', () => { + expect(() => pairedDeltaSummary(base, [{ round: 9, value: 1 }], sample => sample.value)).toThrow(/no rounds in common/); + }); + + // 負のroundはwarmupを表すので集計に混ぜない + test('ignores warmup rounds', () => { + const warmupBase = [{ round: -1, value: 0 }, ...base]; + const warmupHead = [{ round: -1, value: 9999 }, { round: 1, value: 110 }, { round: 2, value: 230 }, { round: 3, value: 320 }]; + + expect(pairedDeltaSummary(warmupBase, warmupHead, sample => sample.value).samples).toBe(3); + }); +}); + +describe('independentDeltaSummary', () => { + const base = [290_000, 292_900, 295_800, 298_700, 301_600] + .map((value, index) => ({ round: index + 1, value })); + const head = [292_900, 296_300, 298_700, 301_600, 290_000] + .map((value, index) => ({ round: index + 1, value })); + + test('uses the difference of independent medians instead of the paired median', () => { + expect(pairedDeltaSummary(base, head, sample => sample.value).median).toBe(2_900); + + const summary = independentDeltaSummary(base, head, sample => sample.value); + expect(summary).toMatchObject({ + baseMedian: 295_800, + headMedian: 296_300, + delta: 500, + baseMad: 2_900, + headMad: 3_400, + baseSamples: 5, + headSamples: 5, + }); + expect(summary.combinedMad).toBeCloseTo(Math.hypot(2_900, 3_400)); + }); + + test('allows unequal sample counts', () => { + const summary = independentDeltaSummary( + [{ value: 10 }, { value: 20 }], + [{ value: 20 }, { value: 30 }, { value: 40 }], + sample => sample.value, + ); + + expect(summary).toStrictEqual({ + baseMedian: 15, + headMedian: 30, + delta: 15, + baseMad: 5, + headMad: 10, + combinedMad: Math.hypot(5, 10), + baseSamples: 2, + headSamples: 3, + }); + }); + + test('throws when either side has fewer than two samples', () => { + expect(() => independentDeltaSummary( + [{ value: 10 }], + [{ value: 20 }, { value: 20 }], + sample => sample.value, + )).toThrow('At least two samples per side are required'); + + expect(() => independentDeltaSummary( + [{ value: 10 }, { value: 10 }], + [{ value: 20 }], + sample => sample.value, + )).toThrow('At least two samples per side are required'); + }); + + test('treats exactly three combined MADs as noise and only larger deltas as outside noise', () => { + function summary(delta: number, combinedMad: number): IndependentDeltaSummary { + return { + baseMedian: 100, + headMedian: 100 + delta, + delta, + baseMad: 0, + headMad: combinedMad, + combinedMad, + baseSamples: 3, + headSamples: 3, + }; + } + + expect(isOutsideObservedNoise(summary(29, 10))).toBe(false); + expect(isOutsideObservedNoise(summary(30, 10))).toBe(false); + expect(isOutsideObservedNoise(summary(31, 10))).toBe(true); + expect(isOutsideObservedNoise(summary(-31, 10))).toBe(true); + }); +}); diff --git a/packages-private/diagnostics-shared/tsconfig.json b/packages-private/diagnostics-shared/tsconfig.json new file mode 100644 index 0000000000..6672c40acb --- /dev/null +++ b/packages-private/diagnostics-shared/tsconfig.json @@ -0,0 +1,20 @@ +{ + "$schema": "https://json.schemastore.org/tsconfig", + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "bundler", + "strict": true, + "strictFunctionTypes": true, + "strictNullChecks": true, + "esModuleInterop": true, + "skipLibCheck": true, + "noEmit": true, + "types": ["node"] + }, + "include": [ + "src/**/*.ts", + "test/**/*.ts" + ], + "exclude": [] +} diff --git a/packages/backend/migration/1782581064131-urlPreviewSensitiveList.js b/packages/backend/migration/1782581064131-urlPreviewSensitiveList.js new file mode 100644 index 0000000000..8c0bdedeb9 --- /dev/null +++ b/packages/backend/migration/1782581064131-urlPreviewSensitiveList.js @@ -0,0 +1,16 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +export class UrlPreviewSensitiveList1782581064131 { + name = 'UrlPreviewSensitiveList1782581064131' + + async up(queryRunner) { + await queryRunner.query(`ALTER TABLE "meta" ADD "urlPreviewSensitiveList" character varying(3072) array NOT NULL DEFAULT '{}'`); + } + + async down(queryRunner) { + await queryRunner.query(`ALTER TABLE "meta" DROP COLUMN "urlPreviewSensitiveList"`); + } +} diff --git a/packages/backend/migration/1784899839024-HashtagTableDefaults.js b/packages/backend/migration/1784899839024-HashtagTableDefaults.js new file mode 100644 index 0000000000..4b890fa849 --- /dev/null +++ b/packages/backend/migration/1784899839024-HashtagTableDefaults.js @@ -0,0 +1,26 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +export class HashtagTableDefaults1784899839024 { + name = 'HashtagTableDefaults1784899839024' + + async up(queryRunner) { + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "mentionedUserIds" SET DEFAULT '{}'`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "mentionedLocalUserIds" SET DEFAULT '{}'`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "mentionedRemoteUserIds" SET DEFAULT '{}'`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "attachedUserIds" SET DEFAULT '{}'`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "attachedLocalUserIds" SET DEFAULT '{}'`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "attachedRemoteUserIds" SET DEFAULT '{}'`); + } + + async down(queryRunner) { + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "attachedRemoteUserIds" DROP DEFAULT`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "attachedLocalUserIds" DROP DEFAULT`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "attachedUserIds" DROP DEFAULT`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "mentionedRemoteUserIds" DROP DEFAULT`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "mentionedLocalUserIds" DROP DEFAULT`); + await queryRunner.query(`ALTER TABLE "hashtag" ALTER COLUMN "mentionedUserIds" DROP DEFAULT`); + } +} diff --git a/packages/backend/package.json b/packages/backend/package.json index 58b4df7158..1adbde0827 100644 --- a/packages/backend/package.json +++ b/packages/backend/package.json @@ -18,11 +18,11 @@ "build": "rolldown -c", "build:unit": "rolldown -c --sourcemap", "build:e2e": "rolldown -c --e2e", - "build:tsc": "tsgo -p tsconfig.json && tsc-alias -p tsconfig.json", + "build:tsc": "tsc -p tsconfig.json && tsc-alias -p tsconfig.json", "watch": "pnpm compile-config && node ./scripts/watch.mjs", "restart": "pnpm build && pnpm start", "dev": "pnpm compile-config && rolldown -c --watch", - "typecheck": "tsgo --noEmit && tsgo -p test --noEmit && tsgo -p test-federation --noEmit", + "typecheck": "tsc --noEmit && tsc -p test --noEmit && tsc -p test-federation --noEmit", "eslint": "eslint --quiet \"{src,test-federation}/**/*.ts\"", "lint": "pnpm typecheck && pnpm eslint", "test": "pnpm build:unit && cross-env NODE_ENV=test pnpm compile-config && vitest --config vitest.config.unit.ts", @@ -51,35 +51,44 @@ "utf-8-validate": "6.0.6" }, "dependencies": { - "@aws-sdk/client-s3": "3.1073.0", - "@aws-sdk/lib-storage": "3.1073.0", + "@aws-sdk/client-s3": "3.1085.0", + "@aws-sdk/lib-storage": "3.1085.0", "@fastify/accepts": "5.0.4", - "@fastify/cors": "11.2.0", + "@fastify/cors": "11.3.0", "@fastify/http-proxy": "11.5.0", - "@fastify/multipart": "10.0.0", - "@fastify/static": "9.1.3", + "@fastify/multipart": "10.1.0", + "@fastify/otel": "0.20.1", + "@fastify/static": "10.1.2", "@kitajs/html": "4.2.13", "@misskey-dev/emoji-assets": "17.0.3", "@misskey-dev/emoji-data": "17.0.3", "@misskey-dev/sharp-read-bmp": "1.3.1", "@misskey-dev/summaly": "5.5.1", - "@napi-rs/canvas": "1.0.0", - "@nestjs/common": "11.1.27", - "@nestjs/core": "11.1.27", - "@nestjs/testing": "11.1.27", - "@oxc-project/runtime": "0.137.0", + "@napi-rs/canvas": "1.0.2", + "@nestjs/common": "11.1.28", + "@nestjs/core": "11.1.28", + "@nestjs/testing": "11.1.28", + "@opentelemetry/api": "1.9.1", + "@opentelemetry/core": "2.9.0", + "@opentelemetry/exporter-trace-otlp-proto": "0.220.0", + "@opentelemetry/instrumentation": "0.219.0", + "@opentelemetry/instrumentation-pg": "0.72.0", + "@opentelemetry/resources": "2.9.0", + "@opentelemetry/sdk-trace-base": "2.9.0", + "@opentelemetry/sdk-trace-node": "2.9.0", + "@opentelemetry/semantic-conventions": "1.43.0", + "@oxc-project/runtime": "0.139.0", "@peertube/http-signature": "1.7.0", - "@sentry/node": "10.59.0", - "@sentry/profiling-node": "10.59.0", - "@simplewebauthn/server": "13.3.1", - "@sinonjs/fake-timers": "15.4.0", - "@smithy/node-http-handler": "4.8.1", + "@sentry/node": "10.65.0", + "@sentry/profiling-node": "10.65.0", + "@simplewebauthn/server": "13.3.2", + "@smithy/node-http-handler": "4.9.5", "accepts": "1.3.8", "ajv": "8.20.0", "archiver": "8.0.0", "bcryptjs": "3.0.3", "blurhash": "2.0.5", - "bullmq": "5.79.0", + "bullmq": "5.80.2", "cacheable-lookup": "7.0.0", "chalk": "5.6.2", "chalk-template": "1.1.2", @@ -88,12 +97,12 @@ "content-disposition": "2.0.1", "date-fns": "4.4.0", "deep-email-validator": "0.1.27", - "fastify": "5.8.5", + "fastify": "5.10.0", "fastify-raw-body": "5.0.0", - "feed": "5.2.1", + "feed": "6.0.0", "file-type": "22.0.1", "fluent-ffmpeg": "2.1.3", - "got": "15.0.5", + "got": "15.1.0", "hpagent": "1.2.0", "http-link-header": "1.1.3", "i18n": "workspace:*", @@ -104,17 +113,17 @@ "json5": "2.2.3", "jsonld": "9.0.0", "juice": "12.1.1", - "meilisearch": "0.58.0", + "meilisearch": "0.59.0", "mfm-js": "0.26.0", "mime-types": "3.0.2", "misskey-js": "workspace:*", "misskey-reversi": "workspace:*", "ms": "3.0.0-canary.202508261828", - "nanoid": "5.1.14", + "nanoid": "6.0.0", "nested-property": "4.0.0", "node-fetch": "3.3.2", - "node-html-parser": "7.1.0", - "nodemailer": "9.0.1", + "node-html-parser": "9.0.0", + "nodemailer": "9.0.3", "os-utils": "0.0.14", "otpauth": "9.5.1", "pg": "8.22.0", @@ -124,22 +133,22 @@ "qrcode": "1.5.4", "random-seed": "0.3.0", "ratelimiter": "3.4.1", - "re2": "1.25.0", + "re2": "1.26.0", "reflect-metadata": "0.2.2", "rename": "1.0.4", "rss-parser": "3.13.0", - "sanitize-html": "2.17.5", + "sanitize-html": "2.17.6", "secure-json-parse": "4.1.0", "semver": "7.8.5", - "sharp": "0.35.2", + "sharp": "0.35.3", "slacc": "0.1.5", "strict-event-emitter-types": "2.0.0", "stringz": "2.1.0", - "systeminformation": "5.31.7", + "systeminformation": "5.31.16", "tinycolor2": "1.6.0", "tmp": "0.2.7", - "tsc-alias": "1.8.17", - "typeorm": "1.0.0", + "tsc-alias": "1.9.0", + "typeorm": "1.1.0", "ulid": "3.0.2", "vary": "1.1.2", "web-push": "3.6.7", @@ -148,18 +157,18 @@ }, "devDependencies": { "@kitajs/ts-html-plugin": "4.1.4", - "@nestjs/platform-express": "11.1.27", + "@nestjs/platform-express": "11.1.28", "@rollup/plugin-esm-shim": "0.1.8", - "@sentry/vue": "10.59.0", + "@sentry/vue": "10.65.0", + "@sinonjs/fake-timers": "15.4.0", "@types/accepts": "1.3.7", "@types/archiver": "8.0.0", "@types/fluent-ffmpeg": "2.1.28", "@types/http-link-header": "1.0.7", - "@types/js-yaml": "4.0.9", "@types/jsonld": "1.5.15", "@types/mime-types": "3.0.1", "@types/ms": "2.1.0", - "@types/node": "26.0.0", + "@types/node": "26.1.1", "@types/nodemailer": "8.0.1", "@types/pg": "8.20.0", "@types/qrcode": "1.5.6", @@ -170,28 +179,26 @@ "@types/semver": "7.7.1", "@types/simple-oauth2": "5.0.8", "@types/sinonjs__fake-timers": "15.0.1", - "@types/supertest": "7.2.0", "@types/tinycolor2": "1.4.6", "@types/tmp": "0.2.6", "@types/vary": "1.1.3", "@types/web-push": "3.6.4", "@types/ws": "8.18.1", - "@typescript-eslint/eslint-plugin": "8.61.1", - "@typescript-eslint/parser": "8.61.1", - "@vitest/coverage-v8": "4.1.9", + "@typescript-eslint/eslint-plugin": "8.63.0", + "@typescript-eslint/parser": "8.63.0", + "@vitest/coverage-v8": "4.1.10", "aws-sdk-client-mock": "4.1.0", "cbor2": "2.3.0", "cross-env": "10.1.0", "eslint-plugin-import": "2.32.0", "execa": "9.6.1", "fkill": "10.0.3", - "js-yaml": "4.2.0", + "js-yaml": "5.2.2", "pid-port": "2.1.1", - "rolldown": "1.1.2", + "rolldown": "1.1.5", "simple-oauth2": "5.1.0", - "supertest": "7.2.2", - "vite": "8.1.0", - "vitest": "4.1.9", + "vite": "8.1.4", + "vitest": "4.1.10", "vitest-mock-extended": "4.0.0" } } diff --git a/packages/backend/rolldown.config.ts b/packages/backend/rolldown.config.ts index 187c444b58..768c3afc03 100644 --- a/packages/backend/rolldown.config.ts +++ b/packages/backend/rolldown.config.ts @@ -3,6 +3,7 @@ import { version as summalyVersion } from '@misskey-dev/summaly'; import type { Plugin, ExternalOption } from 'rolldown'; import { execa, execaNode } from 'execa'; import type { ResultPromise } from 'execa'; +import fkill from 'fkill'; import esmShim from '@rollup/plugin-esm-shim'; /** @@ -10,6 +11,7 @@ import esmShim from '@rollup/plugin-esm-shim'; */ function backendDevServerPlugin(): Plugin { let backendProcess: ResultPromise | null = null; + let backendShutdownPromise: Promise | null = null; async function runBuildAssets() { await execa('pnpm', ['run', 'build-assets'], { @@ -20,12 +22,31 @@ function backendDevServerPlugin(): Plugin { } async function killBackendProcess() { - if (backendProcess) { - backendProcess.catch(() => {}); // backendProcess.kill()によって発生する例外を無視するためにcatch()を呼び出す - backendProcess.kill(); - await new Promise(resolve => backendProcess!.on('exit', resolve)); - backendProcess = null; - } + if (backendShutdownPromise) return backendShutdownPromise; + if (!backendProcess) return; + + const processToKill = backendProcess; + backendProcess = null; + processToKill.catch(() => {}); // プロセスの終了によって発生する例外を無視するためにcatch()を呼び出す + + backendShutdownPromise = (async () => { + if (process.platform === 'win32' && processToKill.pid != null) { + await fkill(processToKill.pid, { + force: true, + tree: true, + silent: true, + waitForExit: 5000, + }); + } else { + processToKill.kill(); + } + + await processToKill.catch(() => {}); + })().finally(() => { + backendShutdownPromise = null; + }); + + return backendShutdownPromise; } return { @@ -49,6 +70,9 @@ function backendDevServerPlugin(): Plugin { await runBuildAssets(); } }, + async closeWatcher() { + await killBackendProcess(); + }, }; } @@ -63,6 +87,7 @@ export default defineConfig((args) => { 'class-validator', /^@sentry\/.*/, /^@sentry-internal\/.*/, + /^@opentelemetry\/.*/, '@nestjs/websockets/socket-module', '@nestjs/microservices/microservices-module', '@nestjs/microservices', @@ -75,6 +100,8 @@ export default defineConfig((args) => { 're2', 'ipaddr.js', 'file-type', + // バンドルするとSentryの自動計装が正しく行われなくなるため外しておく + 'pg', ]; const define: Record = { @@ -133,7 +160,7 @@ export default defineConfig((args) => { clearScreen: false, }, // ビルドの高速化のために、watchモードのときは外部モジュールは全てバンドルしないようにする - external: isWatchMode ? /^(?!@\/)[^.\/](?!:[\/\\])/ : externalModules, + external: isWatchMode ? /^(?!@\/|\0)[^.\/](?!:[\/\\])/ : externalModules, }; } }); diff --git a/packages/backend/scripts/compile_config.js b/packages/backend/scripts/compile_config.js index e78fa3dc9f..2eefbe3cfb 100644 --- a/packages/backend/scripts/compile_config.js +++ b/packages/backend/scripts/compile_config.js @@ -11,7 +11,7 @@ import fs from 'node:fs'; import { resolve, dirname } from 'node:path'; import { fileURLToPath } from 'node:url'; -import yaml from 'js-yaml'; +import { load as loadYaml } from 'js-yaml'; const _filename = fileURLToPath(import.meta.url); const _dirname = dirname(_filename); @@ -34,7 +34,7 @@ function yamlToJson(ymlPath) { console.log(`${ymlPath} → ${OUTPUT_PATH}`); const yamlContent = fs.readFileSync(ymlPath, 'utf-8'); - const jsonContent = yaml.load(yamlContent); + const jsonContent = loadYaml(yamlContent); if (!fs.existsSync(dirname(OUTPUT_PATH))) { fs.mkdirSync(dirname(OUTPUT_PATH), { recursive: true }); } diff --git a/packages/backend/scripts/measure-memory.mts b/packages/backend/scripts/measure-memory.mts deleted file mode 100644 index 74b84d998f..0000000000 --- a/packages/backend/scripts/measure-memory.mts +++ /dev/null @@ -1,631 +0,0 @@ -/* - * SPDX-FileCopyrightText: syuilo and misskey-project - * SPDX-License-Identifier: AGPL-3.0-only - */ - -import { ChildProcess, fork } from 'node:child_process'; -import { setTimeout } from 'node:timers/promises'; -import { fileURLToPath } from 'node:url'; -import { dirname, join } from 'node:path'; -import { tmpdir } from 'node:os'; -//import * as http from 'node:http'; -import * as fs from 'node:fs/promises'; -import { heapSnapshotCategory, type HeapSnapshotData } from '../../../.github/scripts/heap-snapshot-util.mts'; - -const __filename = fileURLToPath(import.meta.url); -const __dirname = dirname(__filename); - -function readIntegerEnv(name, defaultValue, min) { - const rawValue = process.env[name]; - if (rawValue == null || rawValue === '') return defaultValue; - if (!/^\d+$/.test(rawValue)) throw new Error(`${name} must be an integer`); - - const value = Number(rawValue); - if (!Number.isSafeInteger(value) || value < min) throw new Error(`${name} must be >= ${min}`); - return value; -} - -function readBooleanEnv(name, defaultValue) { - const rawValue = process.env[name]; - if (rawValue == null || rawValue === '') return defaultValue; - if (rawValue === '1' || rawValue === 'true') return true; - if (rawValue === '0' || rawValue === 'false') return false; - throw new Error(`${name} must be one of: 1, 0, true, false`); -} - -const SAMPLE_COUNT = readIntegerEnv('MK_MEMORY_SAMPLE_COUNT', 3, 1); // Number of samples to measure -const STARTUP_TIMEOUT = readIntegerEnv('MK_MEMORY_STARTUP_TIMEOUT_MS', 120000, 1); // Timeout for server startup -const MEMORY_SETTLE_TIME = readIntegerEnv('MK_MEMORY_SETTLE_TIME_MS', 10000, 0); // Wait after startup for memory to settle -const IPC_TIMEOUT = readIntegerEnv('MK_MEMORY_IPC_TIMEOUT_MS', 30000, 1); // Timeout for IPC responses -const REQUEST_COUNT = readIntegerEnv('MK_MEMORY_REQUEST_COUNT', 10, 0); -const HEAP_SNAPSHOT = readBooleanEnv('MK_MEMORY_HEAP_SNAPSHOT', false); -const HEAP_SNAPSHOT_TIMEOUT = readIntegerEnv('MK_MEMORY_HEAP_SNAPSHOT_TIMEOUT_MS', 120000, 1); -const HEAP_SNAPSHOT_BREAKDOWN_TOP_N = readIntegerEnv('MK_MEMORY_HEAP_SNAPSHOT_BREAKDOWN_TOP_N', 6, 1); -const HEAP_SNAPSHOT_SAVE_PATH = process.env.MK_MEMORY_HEAP_SNAPSHOT_SAVE_PATH; - -const procStatusKeys = ['VmPeak', 'VmSize', 'VmHWM', 'VmRSS', 'VmData', 'VmStk', 'VmExe', 'VmLib', 'VmPTE', 'VmSwap'] as const; -const smapsRollupKeys = ['Pss', 'Shared_Clean', 'Shared_Dirty', 'Private_Clean', 'Private_Dirty', 'Swap', 'SwapPss'] as const; - -type GcMessage = 'gc ok' | 'gc unavailable'; -type RuntimeMemoryUsageMessage = { - type: 'memory usage'; - value: NodeJS.MemoryUsage; -}; -type HeapSnapshotMessage = { - type: 'heap snapshot'; - path?: string; -}; -type HeapSnapshotErrorMessage = { - type: 'heap snapshot error'; - message: string; -}; -type HeapSnapshotResponseMessage = HeapSnapshotMessage | HeapSnapshotErrorMessage; - -function parseMemoryFile(content: string, keys: KS, path: string, required: boolean): Record { - const result = {} as Record; - for (const _key of keys) { - const key = _key as KS[number]; - const match = content.match(new RegExp(`${key}:\\s+(\\d+)\\s+kB`)); - if (match) { - result[key] = parseInt(match[1], 10); - } else if (required) { - throw new Error(`Failed to parse ${key} from ${path}`); - } - } - return result; -} - -function bytesToKiB(value: number) { - return Math.round(value / 1024); -} - -function sanitizeHeapSnapshotBreakdownLabel(value, fallback = 'unknown') { - const label = String(value ?? '').replace(/\s+/g, ' ').trim(); - if (label === '') return fallback; - if (label.length <= 80) return label; - return `${label.slice(0, 77)}...`; -} - -function classifyHeapSnapshotBreakdown(category: keyof typeof heapSnapshotCategory, type, name) { - if (category === 'strings') return type; - - if (category === 'jsArrays') { - if (type === 'array elements') return 'Array elements'; - if (type === 'object' && name === 'Array') return 'Array objects'; - return sanitizeHeapSnapshotBreakdownLabel(`${type}: ${name}`); - } - - if (category === 'typedArrays') { - if (name === 'system / JSArrayBufferData') return 'ArrayBuffer data'; - return sanitizeHeapSnapshotBreakdownLabel(`${type}: ${name}`); - } - - if (category === 'systemObjects') { - if (name.startsWith('system /')) return sanitizeHeapSnapshotBreakdownLabel(name); - if (name.startsWith('(system ')) return sanitizeHeapSnapshotBreakdownLabel(name); - return sanitizeHeapSnapshotBreakdownLabel(`${type}: ${name}`, type); - } - - if (category === 'otherJsObjects') { - if (type === 'object') return sanitizeHeapSnapshotBreakdownLabel(`object: ${name}`, 'object: unknown'); - return type; - } - - if (category === 'otherNonJsObjects') { - if (type === 'extra native bytes') return 'Extra native bytes'; - if (type === 'native') return sanitizeHeapSnapshotBreakdownLabel(`native: ${name}`, 'native: unknown'); - return sanitizeHeapSnapshotBreakdownLabel(`${type}: ${name}`, type); - } - - if (category === 'code') { - const lowerName = name.toLowerCase(); - if (lowerName.includes('bytecode')) return 'bytecode'; - if (lowerName.includes('builtin')) return 'builtins'; - if (lowerName.includes('regexp')) return 'regexp code'; - if (lowerName.includes('stub')) return 'stubs'; - return sanitizeHeapSnapshotBreakdownLabel(`code: ${name}`, 'code: unknown'); - } - - return sanitizeHeapSnapshotBreakdownLabel(`${type}: ${name}`, type); -} - -function collapseHeapSnapshotBreakdown(breakdowns: Record>) { - const collapsed = {} as Record>; - - for (const [category, children] of Object.entries(breakdowns)) { - const entries = Object.entries(children) - .filter(([, value]) => value > 0) - .toSorted((a, b) => b[1] - a[1]); - - const topEntries = entries.slice(0, HEAP_SNAPSHOT_BREAKDOWN_TOP_N); - const otherValue = entries - .slice(HEAP_SNAPSHOT_BREAKDOWN_TOP_N) - .reduce((sum, [, value]) => sum + value, 0); - - const categoryBreakdown = Object.fromEntries(topEntries); - if (otherValue > 0) categoryBreakdown.Other = otherValue; - if (Object.keys(categoryBreakdown).length > 0) collapsed[category] = categoryBreakdown; - } - - return collapsed; -} - -// Keep these buckets aligned with Chrome DevTools' heap snapshot Statistics view. -function analyzeHeapSnapshot(snapshot) { - const meta = snapshot?.snapshot?.meta; - const nodes = snapshot?.nodes; - const edges = snapshot?.edges; - const strings = snapshot?.strings; - if (meta == null || !Array.isArray(nodes) || !Array.isArray(edges) || !Array.isArray(strings)) { - throw new Error('Invalid heap snapshot format'); - } - - const nodeFields = meta.node_fields; - if (!Array.isArray(nodeFields)) throw new Error('Invalid heap snapshot node fields'); - const edgeFields = meta.edge_fields; - if (!Array.isArray(edgeFields)) throw new Error('Invalid heap snapshot edge fields'); - - const typeOffset = nodeFields.indexOf('type'); - const nameOffset = nodeFields.indexOf('name'); - const selfSizeOffset = nodeFields.indexOf('self_size'); - const edgeCountOffset = nodeFields.indexOf('edge_count'); - if (typeOffset < 0 || nameOffset < 0 || selfSizeOffset < 0 || edgeCountOffset < 0) { - throw new Error('Heap snapshot is missing required node fields'); - } - const edgeTypeOffset = edgeFields.indexOf('type'); - const edgeNameOffset = edgeFields.indexOf('name_or_index'); - const edgeToNodeOffset = edgeFields.indexOf('to_node'); - if (edgeTypeOffset < 0 || edgeNameOffset < 0 || edgeToNodeOffset < 0) { - throw new Error('Heap snapshot is missing required edge fields'); - } - - const nodeTypeNames = meta.node_types?.[typeOffset]; - if (!Array.isArray(nodeTypeNames)) throw new Error('Invalid heap snapshot node types'); - const edgeTypeNames = meta.edge_types?.[edgeTypeOffset]; - if (!Array.isArray(edgeTypeNames)) throw new Error('Invalid heap snapshot edge types'); - - function createEmptyHeapSnapshotCategoryMap() { - return Object.fromEntries(Object.keys(heapSnapshotCategory).map(category => [category, 0])) as Record; - } - - const nodeFieldCount = nodeFields.length; - const edgeFieldCount = edgeFields.length; - const nativeType = nodeTypeNames.indexOf('native'); - const codeType = nodeTypeNames.indexOf('code'); - const hiddenType = nodeTypeNames.indexOf('hidden'); - const stringTypes = new Set([ - nodeTypeNames.indexOf('string'), - nodeTypeNames.indexOf('concatenated string'), - nodeTypeNames.indexOf('sliced string'), - ]); - const internalEdgeType = edgeTypeNames.indexOf('internal'); - const extraNativeBytes = Number.isFinite(snapshot.snapshot.extra_native_bytes) ? snapshot.snapshot.extra_native_bytes : 0; - const categories = createEmptyHeapSnapshotCategoryMap(); - const nodeCounts = createEmptyHeapSnapshotCategoryMap(); - const breakdowns = Object.fromEntries( - (Object.keys(heapSnapshotCategory) as (keyof typeof heapSnapshotCategory)[]) - .filter(category => category !== 'total') - .map(category => [category, {}]), - ); - - function addValue(map: Record, key: string, value: number) { - map[key] = (map[key] ?? 0) + value; - } - - const edgeStartIndexes = new Map(); - const retainerCounts = new Map(); - let edgeIndex = 0; - for (let nodeIndex = 0; nodeIndex < nodes.length; nodeIndex += nodeFieldCount) { - edgeStartIndexes.set(nodeIndex, edgeIndex); - const edgeCount = nodes[nodeIndex + edgeCountOffset] ?? 0; - for (let i = 0; i < edgeCount; i++, edgeIndex += edgeFieldCount) { - const toNodeIndex = edges[edgeIndex + edgeToNodeOffset]; - retainerCounts.set(toNodeIndex, (retainerCounts.get(toNodeIndex) ?? 0) + 1); - } - } - - const jsArrayElementNodeIndexes = new Set(); - - function addCategoryValue(category: keyof typeof heapSnapshotCategory, value: number, type: string, name: string, nodeIndex: number | null = null) { - if (value <= 0) return; - categories[category] += value; - addValue(breakdowns[category], classifyHeapSnapshotBreakdown(category, type, name), value); - if (nodeIndex != null) nodeCounts[category]++; - } - - function addJsArrayElementSize(nodeIndex: number) { - const beginEdgeIndex = edgeStartIndexes.get(nodeIndex) ?? 0; - const edgeCount = nodes[nodeIndex + edgeCountOffset] ?? 0; - for (let i = 0, currentEdgeIndex = beginEdgeIndex; i < edgeCount; i++, currentEdgeIndex += edgeFieldCount) { - const edgeType = edges[currentEdgeIndex + edgeTypeOffset]; - if (edgeType !== internalEdgeType) continue; - - const edgeName = strings[edges[currentEdgeIndex + edgeNameOffset]]; - if (edgeName !== 'elements') continue; - - const elementsNodeIndex = edges[currentEdgeIndex + edgeToNodeOffset]; - if ((retainerCounts.get(elementsNodeIndex) ?? 0) === 1) { - const elementsSize = nodes[elementsNodeIndex + selfSizeOffset] ?? 0; - addCategoryValue('jsArrays', elementsSize, 'array elements', 'Array elements', elementsNodeIndex); - jsArrayElementNodeIndexes.add(elementsNodeIndex); - } - break; - } - } - - if (extraNativeBytes > 0) { - addCategoryValue('otherNonJsObjects', extraNativeBytes, 'extra native bytes', 'extra native bytes'); - } - - for (let nodeIndex = 0; nodeIndex < nodes.length; nodeIndex += nodeFieldCount) { - const typeId = nodes[nodeIndex + typeOffset]; - const type = nodeTypeNames[typeId] ?? 'unknown'; - const name = strings[nodes[nodeIndex + nameOffset]] ?? ''; - const selfSize = nodes[nodeIndex + selfSizeOffset] ?? 0; - categories.total += selfSize; - nodeCounts.total++; - - if (typeId === hiddenType) { - addCategoryValue('systemObjects', selfSize, type, name, nodeIndex); - continue; - } - - if (typeId === nativeType) { - if (name === 'system / JSArrayBufferData') { - addCategoryValue('typedArrays', selfSize, type, name, nodeIndex); - } else { - addCategoryValue('otherNonJsObjects', selfSize, type, name, nodeIndex); - } - continue; - } - - if (typeId === codeType) { - addCategoryValue('code', selfSize, type, name, nodeIndex); - continue; - } - - if (stringTypes.has(typeId)) { - addCategoryValue('strings', selfSize, type, name, nodeIndex); - continue; - } - - if (name === 'Array') { - addCategoryValue('jsArrays', selfSize, type, name, nodeIndex); - addJsArrayElementSize(nodeIndex); - continue; - } - } - - categories.total += extraNativeBytes; - - for (let nodeIndex = 0; nodeIndex < nodes.length; nodeIndex += nodeFieldCount) { - if (jsArrayElementNodeIndexes.has(nodeIndex)) continue; - - const typeId = nodes[nodeIndex + typeOffset]; - if (typeId === hiddenType || typeId === nativeType || typeId === codeType || stringTypes.has(typeId)) continue; - - const name = strings[nodes[nodeIndex + nameOffset]] ?? ''; - if (name === 'Array') continue; - - const type = nodeTypeNames[typeId] ?? 'unknown'; - const selfSize = nodes[nodeIndex + selfSizeOffset] ?? 0; - addCategoryValue('otherJsObjects', selfSize, type, name, nodeIndex); - } - - return { - categories, - nodeCounts, - breakdowns: collapseHeapSnapshotBreakdown(breakdowns), - }; -} - -async function getMemoryUsage(pid: number) { - const path = `/proc/${pid}/status`; - const status = await fs.readFile(path, 'utf-8'); - return parseMemoryFile(status, procStatusKeys, path, true); -} - -async function getSmapsRollupMemoryUsage(pid: number) { - const path = `/proc/${pid}/smaps_rollup`; - const smapsRollup = await fs.readFile(path, 'utf-8'); - return parseMemoryFile(smapsRollup, smapsRollupKeys, path, false); -} - -function isRecord(value: unknown): value is Record { - return value != null && typeof value === 'object'; -} - -function isGcMessage(message: unknown): message is GcMessage { - return message === 'gc ok' || message === 'gc unavailable'; -} - -function isRuntimeMemoryUsageMessage(message: unknown): message is RuntimeMemoryUsageMessage { - return isRecord(message) && message.type === 'memory usage' && isRecord(message.value); -} - -function isHeapSnapshotResponseMessage(message: unknown): message is HeapSnapshotResponseMessage { - if (!isRecord(message)) return false; - if (message.type === 'heap snapshot') return true; - return message.type === 'heap snapshot error' && typeof message.message === 'string'; -} - -function waitForMessage(serverProcess: ChildProcess, predicate: (message: unknown) => message is T, description: string, timeout = IPC_TIMEOUT) { - return new Promise((resolve, reject) => { - const timer = globalThis.setTimeout(() => { - serverProcess.off('message', onMessage); - reject(new Error(`Timed out waiting for ${description}`)); - }, timeout); - - const onMessage = (message: unknown) => { - if (!predicate(message)) return; - globalThis.clearTimeout(timer); - serverProcess.off('message', onMessage); - resolve(message); - }; - - serverProcess.on('message', onMessage); - }); -} - -async function getRuntimeMemoryUsage(serverProcess: ChildProcess) { - const response = waitForMessage( - serverProcess, - isRuntimeMemoryUsageMessage, - 'memory usage', - ); - - serverProcess.send('memory usage'); - - const message = await response; - const memoryUsage = message.value; - - return { - HeapTotal: bytesToKiB(memoryUsage.heapTotal), - HeapUsed: bytesToKiB(memoryUsage.heapUsed), - External: bytesToKiB(memoryUsage.external), - ArrayBuffers: bytesToKiB(memoryUsage.arrayBuffers), - }; -} - -async function getHeapSnapshotStatistics(serverProcess: ChildProcess): Promise { - if (!HEAP_SNAPSHOT) return null; - - const snapshotPath = join(tmpdir(), `misskey-backend-heap-${process.pid}-${serverProcess.pid}-${Date.now()}.heapsnapshot`); - const response = waitForMessage( - serverProcess, - isHeapSnapshotResponseMessage, - 'heap snapshot', - HEAP_SNAPSHOT_TIMEOUT, - ); - - serverProcess.send({ - type: 'heap snapshot', - path: snapshotPath, - }); - - const message = await response; - if (message.type === 'heap snapshot error') { - throw new Error(`Failed to write heap snapshot: ${message.message}`); - } - - const writtenPath = typeof message.path === 'string' ? message.path : snapshotPath; - - try { - if (HEAP_SNAPSHOT_SAVE_PATH != null && HEAP_SNAPSHOT_SAVE_PATH !== '') { - await fs.mkdir(dirname(HEAP_SNAPSHOT_SAVE_PATH), { recursive: true }); - await fs.copyFile(writtenPath, HEAP_SNAPSHOT_SAVE_PATH); - } - - const snapshot = JSON.parse(await fs.readFile(writtenPath, 'utf-8')); - return analyzeHeapSnapshot(snapshot); - } finally { - await fs.unlink(writtenPath).catch(err => { - process.stderr.write(`Failed to delete heap snapshot ${writtenPath}: ${err.message}\n`); - }); - } -} - -async function getAllMemoryUsage(serverProcess: ChildProcess) { - const pid = serverProcess.pid!; - return { - ...await getMemoryUsage(pid), - ...await getSmapsRollupMemoryUsage(pid), - ...await getRuntimeMemoryUsage(serverProcess), - }; -} - -async function measureMemory() { - // Start the Misskey backend server using fork to enable IPC - const serverProcess = fork(join(__dirname, '../built/entry.js'), [], { - cwd: join(__dirname, '..'), - env: { - ...process.env, - NODE_ENV: 'production', - MK_DISABLE_CLUSTERING: '1', - MK_ONLY_SERVER: '1', - MK_NO_DAEMONS: '1', - }, - stdio: ['pipe', 'pipe', 'pipe', 'ipc'], - execArgv: [...process.execArgv, '--expose-gc'], - }); - - let serverReady = false; - - // Listen for the 'ok' message from the server indicating it's ready - serverProcess.on('message', (message) => { - if (message === 'ok') { - serverReady = true; - } - }); - - // Handle server output - serverProcess.stdout?.on('data', (data) => { - process.stderr.write(`[server stdout] ${data}`); - }); - - serverProcess.stderr?.on('data', (data) => { - process.stderr.write(`[server stderr] ${data}`); - }); - - // Handle server error - serverProcess.on('error', (err) => { - process.stderr.write(`[server error] ${err}\n`); - }); - - async function triggerGc() { - const ok = waitForMessage( - serverProcess, - isGcMessage, - 'GC completion', - ); - - serverProcess.send('gc'); - - const message = await ok; - if (message === 'gc unavailable') { - throw new Error('GC is unavailable. Start the process with --expose-gc to enable this feature.'); - } - - await setTimeout(1000); - } - - //function createRequest() { - // return new Promise((resolve, reject) => { - // const req = http.request({ - // host: 'localhost', - // port: 61812, - // path: '/api/meta', - // method: 'POST', - // }, (res) => { - // res.on('data', () => { }); - // res.on('end', () => { - // resolve(); - // }); - // }); - // req.on('error', (err) => { - // reject(err); - // }); - // req.end(); - // }); - //} - - // Wait for server to be ready or timeout - const startupStartTime = Date.now(); - while (!serverReady) { - if (Date.now() - startupStartTime > STARTUP_TIMEOUT) { - serverProcess.kill('SIGTERM'); - throw new Error('Server startup timeout'); - } - await setTimeout(100); - } - - const startupTime = Date.now() - startupStartTime; - process.stderr.write(`Server started in ${startupTime}ms\n`); - - // Wait for memory to settle - await setTimeout(MEMORY_SETTLE_TIME); - - //const beforeGc = await getAllMemoryUsage(serverProcess); - - await triggerGc(); - - const memoryUsageAfterGC = await getAllMemoryUsage(serverProcess); - - //// create some http requests to simulate load - //await Promise.all( - // Array.from({ length: REQUEST_COUNT }).map(() => createRequest()), - //); - - //await triggerGc(); - - //const afterRequest = await getAllMemoryUsage(serverProcess); - - const heapSnapshotAfterGc = await getHeapSnapshotStatistics(serverProcess); - - // Stop the server - serverProcess.kill('SIGTERM'); - - // Wait for process to exit - let exited = false; - await new Promise((resolve) => { - serverProcess.on('exit', () => { - exited = true; - resolve(undefined); - }); - // Force kill after 10 seconds if not exited - setTimeout(10000).then(() => { - if (!exited) { - serverProcess.kill('SIGKILL'); - } - resolve(undefined); - }); - }); - - const result = { - timestamp: new Date().toISOString(), - phases: { - //beforeGc, - afterGc: { - memoryUsage: memoryUsageAfterGC, - heapSnapshot: heapSnapshotAfterGc, - }, - //afterRequest, - }, - }; - - return result; -} - -export type MemoryReportRaw = { - timestamp: string; - sampleCount: number; - measurement: { - startupTimeoutMs: number; - memorySettleTimeMs: number; - ipcTimeoutMs: number; - requestCount: number; - heapSnapshot: { - enabled: boolean; - timeoutMs: number; - breakdownTopN: number; - }; - }; - samples: Awaited>[]; -}; - -async function main() { - const results = []; - for (let i = 0; i < SAMPLE_COUNT; i++) { - process.stderr.write(`Starting sample ${i + 1}/${SAMPLE_COUNT}\n`); - const res = await measureMemory(); - results.push(res); - } - - const result: MemoryReportRaw = { - timestamp: new Date().toISOString(), - sampleCount: SAMPLE_COUNT, - measurement: { - startupTimeoutMs: STARTUP_TIMEOUT, - memorySettleTimeMs: MEMORY_SETTLE_TIME, - ipcTimeoutMs: IPC_TIMEOUT, - requestCount: REQUEST_COUNT, - heapSnapshot: { - enabled: HEAP_SNAPSHOT, - timeoutMs: HEAP_SNAPSHOT_TIMEOUT, - breakdownTopN: HEAP_SNAPSHOT_BREAKDOWN_TOP_N, - }, - }, - samples: results, - }; - - // Output as JSON to stdout - console.log(JSON.stringify(result, null, 2)); -} - -main().catch((err) => { - console.error(JSON.stringify({ - error: err.message, - timestamp: new Date().toISOString(), - })); - process.exit(1); -}); diff --git a/packages/backend/scripts/watch.mjs b/packages/backend/scripts/watch.mjs index 9d608b233c..a0ccea3b16 100644 --- a/packages/backend/scripts/watch.mjs +++ b/packages/backend/scripts/watch.mjs @@ -21,7 +21,7 @@ import { execa } from 'execa'; }); }, 3000); - execa('tsgo', ['-w', '-p', 'tsconfig.json'], { + execa('tsc', ['-w', '-p', 'tsconfig.json'], { stdout: process.stdout, stderr: process.stderr, }); diff --git a/packages/backend/src/@types/probe-image-size.d.ts b/packages/backend/src/@types/probe-image-size.d.ts index 538836475c..f2efbc082d 100644 --- a/packages/backend/src/@types/probe-image-size.d.ts +++ b/packages/backend/src/@types/probe-image-size.d.ts @@ -26,7 +26,6 @@ declare module 'probe-image-size' { function probeImageSize(src: string | ReadStream, callback: (err: Error | null, result?: ProbeResult) => void): void; function probeImageSize(src: string | ReadStream, options: ProbeOptions, callback: (err: Error | null, result?: ProbeResult) => void): void; - namespace probeImageSize {} // Hack - - export = probeImageSize; + // eslint-disable-next-line import/no-default-export + export { probeImageSize as default }; } diff --git a/packages/backend/src/boot/entry.ts b/packages/backend/src/boot/entry.ts index 9fcb4d73ca..03e5c395c3 100644 --- a/packages/backend/src/boot/entry.ts +++ b/packages/backend/src/boot/entry.ts @@ -10,10 +10,11 @@ import cluster from 'node:cluster'; import { EventEmitter } from 'node:events'; import { writeHeapSnapshot } from 'node:v8'; -import chalk from 'chalk'; import Xev from 'xev'; import Logger from '@/logger.js'; import { envOption } from '../env.js'; +import { installProcessErrorHandlers } from './process-error-handler.js'; +import { isShutdownInProgress } from './shutdown-handler.js'; import { readyRef } from './ready.js'; import 'reflect-metadata'; @@ -27,6 +28,8 @@ const logger = new Logger('core', 'cyan'); const clusterLogger = logger.createSubLogger('cluster', 'orange'); const ev = new Xev(); +installProcessErrorHandlers({ logger, quiet: envOption.quiet }); + //#region Events // Listen new workers @@ -41,27 +44,19 @@ cluster.on('online', worker => { // Listen for dying workers cluster.on('exit', worker => { - // Replace the dead worker, - // we're not sentimental - clusterLogger.error(chalk.red(`[${worker.id}] died :(`)); + if (isShutdownInProgress()) { + clusterLogger.info(`Process exited during shutdown: [${worker.id}]`); + return; + } + + // 終了したワーカーは従来どおり再生成し、表示色は出力処理へ任せます。 + clusterLogger.error(`[${worker.id}] died :(`); cluster.fork(); }); -// Display detail of unhandled promise rejection -if (!envOption.quiet) { - process.on('unhandledRejection', console.dir); -} - -// Display detail of uncaught exception -process.on('uncaughtException', err => { - try { - logger.error(err); - console.trace(err); - } catch { } -}); - // Dying away... process.on('exit', code => { + if (isShutdownInProgress()) return; logger.info(`The process is going to exit with code ${code}`); }); @@ -69,12 +64,10 @@ process.on('exit', code => { if (!envOption.disableClustering) { if (cluster.isPrimary) { - logger.info(`Start main process... pid: ${process.pid}`); const { masterMain } = await import('./master.js'); await masterMain(); ev.mount(); } else if (cluster.isWorker) { - logger.info(`Start worker process... pid: ${process.pid}`); const { workerMain } = await import('./worker.js'); await workerMain(); } else { @@ -82,7 +75,6 @@ if (!envOption.disableClustering) { } } else { // 非clusterの場合はMasterのみが起動するため、Workerの処理は行わない(cluster.isWorker === trueの状態でこのブロックに来ることはない) - logger.info(`Start main process... pid: ${process.pid}`); const { masterMain } = await import('./master.js'); await masterMain(); ev.mount(); diff --git a/packages/backend/src/boot/master.ts b/packages/backend/src/boot/master.ts index 533ebe5bbb..3cb340e963 100644 --- a/packages/backend/src/boot/master.ts +++ b/packages/backend/src/boot/master.ts @@ -11,16 +11,30 @@ import chalkTemplate from 'chalk-template'; import Logger from '@/logger.js'; import { loadConfig } from '@/config.js'; import type { Config } from '@/config.js'; +import { configureLogging, shutdownLogging } from '@/logging/logging-runtime.js'; +import type { LogFormat } from '@/logging/types.js'; import { showMachineInfo } from '@/misc/show-machine-info.js'; import { envOption } from '@/env.js'; +import { initTelemetry, shutdownTelemetry } from '@/core/telemetry/telemetry-registry.js'; import { initExtraThreadPool, jobQueue, server } from './common.js'; +import { installShutdownSignalHandlers } from './shutdown-handler.js'; const logger = new Logger('core', 'cyan'); const bootLogger = logger.createSubLogger('boot', 'magenta'); const themeColor = chalk.hex('#86b300'); -function greet(props: { version: string }) { +/** 起動時の案内を、選択されたログ形式に合わせて出力します。 */ +function greet(props: { version: string; format: LogFormat }) { + if (!envOption.quiet && props.format === 'json') { + // JSONモードでは生のコンソール出力を避け、各案内を1件ずつ構造化ログにします。 + bootLogger.info('Welcome to Misskey!'); + bootLogger.info(`Misskey v${props.version}`, null, true); + bootLogger.info('Misskey is an open-source decentralized microblogging platform.'); + bootLogger.info('If you like Misskey, please consider donating to support dev. https://misskey-hub.net/docs/donate/'); + return; + } + if (!envOption.quiet) { //#region Misskey logo const v = `v${props.version}`; @@ -51,7 +65,9 @@ export async function masterMain() { // initialize app try { config = loadConfigBoot(); - greet({ version: config.version }); + logger.info(`Start main process... pid: ${process.pid}`); + bootLogger.createSubLogger('config').succ('Loaded'); + greet({ version: config.version, format: config.logging?.format ?? 'pretty' }); showEnvironment(); await showMachineInfo(bootLogger); showNodejsVersion(); @@ -66,26 +82,16 @@ export async function masterMain() { initExtraThreadPool(config); - if (config.sentryForBackend) { - const Sentry = await import('@sentry/node'); - const { nodeProfilingIntegration } = await import('@sentry/profiling-node'); - - Sentry.init({ - integrations: [ - ...(config.sentryForBackend.enableNodeProfiling ? [nodeProfilingIntegration()] : []), - ], - - // Performance Monitoring - tracesSampleRate: 1.0, // Capture 100% of the transactions - - // Set sampling rate for profiling - this is relative to tracesSampleRate - profilesSampleRate: 1.0, - - maxBreadcrumbs: 0, - - ...config.sentryForBackend.options, - }); + try { + await initTelemetry(config); + } catch (e) { + bootLogger.error(e instanceof Error ? e : new Error(String(e)), null, true); + process.exit(1); } + installShutdownSignalHandlers({ + shutdownTasks: [shutdownTelemetry, shutdownLogging], + onRegistered: message => bootLogger.info(message), + }); bootLogger.info( `mode: [disableClustering: ${envOption.disableClustering}, onlyServer: ${envOption.onlyServer}, onlyQueue: ${envOption.onlyQueue}]`, @@ -143,12 +149,14 @@ function showNodejsVersion(): void { nodejsLogger.info(`Version ${process.version} detected.`); } +/** 設定を読み込み、成功時に後続のログ出力形式を適用します。 */ function loadConfigBoot(): Config { const configLogger = bootLogger.createSubLogger('config'); let config; try { config = loadConfig(); + configureLogging(config.logging); } catch (exception) { if (typeof exception === 'string') { configLogger.error(exception); @@ -160,8 +168,6 @@ function loadConfigBoot(): Config { throw exception; } - configLogger.succ('Loaded'); - return config; } diff --git a/packages/backend/src/boot/process-error-handler.ts b/packages/backend/src/boot/process-error-handler.ts new file mode 100644 index 0000000000..c1224f1923 --- /dev/null +++ b/packages/backend/src/boot/process-error-handler.ts @@ -0,0 +1,59 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { LogWriteInput } from '@/logging/types.js'; + +/** プロセス例外をロガーへ渡すために必要な最小の処理対象です。 */ +export type ProcessErrorHandlerProcess = { + on(event: 'unhandledRejection', listener: (reason: unknown) => void): unknown; + on(event: 'uncaughtException', listener: (error: Error) => void): unknown; +}; + +/** 例外記録に必要なロガーの最小インターフェースです。 */ +export type ProcessErrorHandlerLogger = { + write(input: LogWriteInput): void; +}; + +/** プロセス例外ハンドラーの登録に使う依存関係です。 */ +export type ProcessErrorHandlerOptions = { + readonly process?: ProcessErrorHandlerProcess; + readonly logger: ProcessErrorHandlerLogger; + readonly quiet: boolean; +}; + +/** ログ記録の失敗が別の例外を起こさないよう、プロセス異常を安全に記録します。 */ +function writeProcessError( + logger: ProcessErrorHandlerLogger, + eventName: 'process.unhandled_rejection' | 'process.uncaught_exception', + message: string, + error: unknown, +): void { + try { + // 元の値をerrorへ渡し、JSON形式の出力処理で安全な形へ正規化します。 + logger.write({ + level: 'error', + eventName, + message, + error, + }); + } catch { + // 例外処理中のログ失敗で、さらにプロセスを不安定にしないよう握りつぶします。 + } +} + +/** 未処理のPromise拒否と未捕捉例外を構造化ログへ接続します。 */ +export function installProcessErrorHandlers(options: ProcessErrorHandlerOptions): void { + const processLike: ProcessErrorHandlerProcess = options.process ?? (process as unknown as ProcessErrorHandlerProcess); + + if (!options.quiet) { + processLike.on('unhandledRejection', reason => { + writeProcessError(options.logger, 'process.unhandled_rejection', 'Unhandled promise rejection', reason); + }); + } + + processLike.on('uncaughtException', error => { + writeProcessError(options.logger, 'process.uncaught_exception', 'Uncaught exception', error); + }); +} diff --git a/packages/backend/src/boot/shutdown-handler.ts b/packages/backend/src/boot/shutdown-handler.ts new file mode 100644 index 0000000000..a0d85d1df6 --- /dev/null +++ b/packages/backend/src/boot/shutdown-handler.ts @@ -0,0 +1,100 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +type ShutdownSignalProcess = { + once(event: 'SIGTERM' | 'SIGINT', listener: () => Promise): unknown; +}; + +const SHUTDOWN_TIMEOUT_MS = 10_000; + +export type ShutdownTask = () => Promise; + +export type ShutdownHandlerOptions = { + /** The process-like object that receives the signal handlers. */ + process?: ShutdownSignalProcess; + /** Shutdown tasks, executed in array order. */ + shutdownTasks: readonly ShutdownTask[]; + /** Process termination function. */ + exit?: (code: number) => void; + /** Optional boot logger hook used after signal handlers are registered. */ + onRegistered?: (message: string) => void; +}; + +let shuttingDown = false; + +/** + * Register the process-level shutdown signals. + * + * Boot owns signal coordination and receives shutdown tasks through callbacks + * so individual domains do not depend on each other. + * + * 注意: このプロジェクトでは app.enableShutdownHooks() が一切呼ばれていないため、 + * NestJSのOnApplicationShutdown経由のgraceful shutdown(GlobalModule.dispose()によるDB/Redis切断、 + * QueueProcessorService.stop()によるqueue drain、ServerService.dispose()によるfastify/WebSocket close)は + * SIGTERM/SIGINTを起点には発火しない。このhandlerはそれらを経由せず、登録された終了処理を実行して即exitする。 + * 将来enableShutdownHooks()を配線する場合は、この即exitとNestJS側のshutdown sequenceが競合しないよう順序を設計すること。 + */ +export function installShutdownSignalHandlers(options: ShutdownHandlerOptions): void { + // テストではprocess/exitを差し替え、本番では実processにSIGTERM/SIGINT handlerを登録する。 + const processLike = options.process ?? process; + const exit = options.exit ?? ((code: number) => process.exit(code)); + + const handleSignal = async () => { + // 同時に複数signalが来てもflushを二重実行せず、cluster refork抑止用の状態もここで立てる。 + if (shuttingDown) return; + shuttingDown = true; + + let timedOut = false; + let timeout: NodeJS.Timeout | undefined; + try { + // 処理時間上限つきのシャットダウンプロセス + await Promise.race([ + (async () => { + for (const shutdownTask of options.shutdownTasks) { + if (timedOut) return; + try { + await shutdownTask(); + } catch (error) { + // 1つの終了処理の失敗で後続タスクを妨げないよう、stderrへフォールバックする。 + try { + console.error('Shutdown task failed:', error); + } catch { + // stderrの出力自体が失敗しても、残りの終了処理とexitは継続する。 + } + } + } + })(), + new Promise(resolve => { + timeout = setTimeout(() => { + timedOut = true; + try { + console.error(`Shutdown tasks timed out after ${SHUTDOWN_TIMEOUT_MS}ms.`); + } catch { + // stderrの出力自体が失敗してもexitは継続する。 + } + resolve(); + }, SHUTDOWN_TIMEOUT_MS); + }), + ]); + } finally { + if (timeout != null) clearTimeout(timeout); + } + + // 既存挙動と同じく、終了処理後はプロセスを終了する。 + exit(0); + }; + + // onceにして、同じsignalでhandlerが再入しないようにする。 + processLike.once('SIGTERM', handleSignal); + processLike.once('SIGINT', handleSignal); + + // app.enableShutdownHooks()未配線の現状、SIGTERM/SIGINT時には登録済み終了処理のみを行う。 + options.onRegistered?.('Registered SIGTERM/SIGINT shutdown handler (this process does not perform NestJS graceful shutdown on these signals).'); +} + +export function isShutdownInProgress(): boolean { + // masterのcluster exit handlerが、意図したshutdown中のworker終了を再forkしないために参照する。 + return shuttingDown; +} diff --git a/packages/backend/src/boot/worker.ts b/packages/backend/src/boot/worker.ts index d8fb1eeaac..71857f9a52 100644 --- a/packages/backend/src/boot/worker.ts +++ b/packages/backend/src/boot/worker.ts @@ -4,38 +4,45 @@ */ import cluster from 'node:cluster'; +import Logger from '@/logger.js'; import { envOption } from '@/env.js'; import { loadConfig } from '@/config.js'; +import type { Config } from '@/config.js'; +import { configureLogging, shutdownLogging } from '@/logging/logging-runtime.js'; +import { initTelemetry, shutdownTelemetry } from '@/core/telemetry/telemetry-registry.js'; import { initExtraThreadPool, jobQueue, server } from './common.js'; +import { installShutdownSignalHandlers } from './shutdown-handler.js'; + +const logger = new Logger('core', 'cyan'); +const bootLogger = logger.createSubLogger('boot', 'magenta'); /** * Init worker process */ export async function workerMain() { - const config = loadConfig(); + let config: Config; + try { + config = loadConfig(); + configureLogging(config.logging); + logger.info(`Start worker process... pid: ${process.pid}`); + } catch (e) { + bootLogger.error(e instanceof Error ? e : new Error(String(e)), null, true); + process.exit(1); + return; + } initExtraThreadPool(config); - if (config.sentryForBackend) { - const Sentry = await import('@sentry/node'); - const { nodeProfilingIntegration } = await import('@sentry/profiling-node'); - - Sentry.init({ - integrations: [ - ...(config.sentryForBackend.enableNodeProfiling ? [nodeProfilingIntegration()] : []), - ], - - // Performance Monitoring - tracesSampleRate: 1.0, // Capture 100% of the transactions - - // Set sampling rate for profiling - this is relative to tracesSampleRate - profilesSampleRate: 1.0, - - maxBreadcrumbs: 0, - - ...config.sentryForBackend.options, - }); + try { + await initTelemetry(config); + } catch (e) { + bootLogger.error(e instanceof Error ? e : new Error(String(e)), null, true); + process.exit(1); } + installShutdownSignalHandlers({ + shutdownTasks: [shutdownTelemetry, shutdownLogging], + onRegistered: message => bootLogger.info(message), + }); if (envOption.onlyServer) { await server(); diff --git a/packages/backend/src/config.ts b/packages/backend/src/config.ts index d67fe6fc6f..66540ae1d1 100644 --- a/packages/backend/src/config.ts +++ b/packages/backend/src/config.ts @@ -10,6 +10,7 @@ import { type FastifyServerOptions } from 'fastify'; import type * as Sentry from '@sentry/node'; import type * as SentryVue from '@sentry/vue'; import type { RedisOptions } from 'ioredis'; +import type { AccessLogConfiguration, LogFormat, LogLevelSetting } from './logging/types.js'; type RedisOptionsSource = Partial & { host: string; @@ -20,6 +21,27 @@ type RedisOptionsSource = Partial & { prefix?: string; }; +type SentryBackendConfig = { + options: Partial; + enableNodeProfiling: boolean; + disabledIntegrations?: string[]; +}; + +type OtelBackendConfig = { + endpoint?: string; + headers?: Record; + sampleRate?: number; + capturePgSpans?: boolean; + capturePgStatement?: boolean; + capturePgConnectionSpans?: boolean; + captureRedisCommandSpans?: boolean; + captureRedisConnectionSpans?: boolean; + captureRedisRootSpans?: boolean; + resourceAttributes?: Record; + propagateTraceToRemote?: boolean; + jobTraceContextMode?: 'link' | 'parent'; +}; + /** * 設定ファイルの型 */ @@ -64,7 +86,8 @@ type Source = { index: string; scope?: 'local' | 'global' | string[]; }; - sentryForBackend?: { options: Partial; enableNodeProfiling: boolean; }; + sentryForBackend?: SentryBackendConfig; + otelForBackend?: OtelBackendConfig; sentryForFrontend?: { options: Partial & { dsn: string }; vueIntegration?: SentryVue.VueIntegrationOptions | null; @@ -110,6 +133,10 @@ type Source = { pidFile: string; logging?: { + format?: LogFormat; + level?: LogLevelSetting; + domains?: Record | null; + access?: AccessLogConfiguration; sql?: { disableQueryTruncation?: boolean, enableQueryParamLogging?: boolean, @@ -172,6 +199,10 @@ export type Config = { deliverJobMaxAttempts: number | undefined; inboxJobMaxAttempts: number | undefined; logging?: { + format?: LogFormat; + level?: LogLevelSetting; + domains?: Record | null; + access?: AccessLogConfiguration; sql?: { disableQueryTruncation?: boolean, enableQueryParamLogging?: boolean, @@ -201,7 +232,8 @@ export type Config = { redisForJobQueue: RedisOptions & RedisOptionsSource; redisForTimelines: RedisOptions & RedisOptionsSource; redisForReactions: RedisOptions & RedisOptionsSource; - sentryForBackend: { options: Partial; enableNodeProfiling: boolean; } | undefined; + sentryForBackend: SentryBackendConfig | undefined; + otelForBackend: OtelBackendConfig | undefined; sentryForFrontend: { options: Partial & { dsn: string }; vueIntegration?: SentryVue.VueIntegrationOptions | null; @@ -307,6 +339,7 @@ export function loadConfig(): Config { redisForTimelines: config.redisForTimelines ? convertRedisOptions(config.redisForTimelines, host) : redis, redisForReactions: config.redisForReactions ? convertRedisOptions(config.redisForReactions, host) : redis, sentryForBackend: config.sentryForBackend, + otelForBackend: config.otelForBackend, sentryForFrontend: config.sentryForFrontend, id: config.id, proxy: config.proxy, diff --git a/packages/backend/src/core/ChannelFollowingService.ts b/packages/backend/src/core/ChannelFollowingService.ts index 7dae8d10f3..54198fdc06 100644 --- a/packages/backend/src/core/ChannelFollowingService.ts +++ b/packages/backend/src/core/ChannelFollowingService.ts @@ -13,6 +13,8 @@ import { GlobalEvents, GlobalEventService } from '@/core/GlobalEventService.js'; import { bindThis } from '@/decorators.js'; import type { MiLocalUser } from '@/models/User.js'; import { RedisKVCache } from '@/misc/cache.js'; +import { IdentifiableError } from '@/misc/identifiable-error.js'; +import { isDuplicateKeyValueError } from '@/misc/is-duplicate-key-value-error.js'; @Injectable() export class ChannelFollowingService implements OnModuleInit { @@ -96,11 +98,18 @@ export class ChannelFollowingService implements OnModuleInit { requestUser: MiLocalUser, targetChannel: MiChannel, ): Promise { - await this.channelFollowingsRepository.insert({ - id: this.idService.gen(), - followerId: requestUser.id, - followeeId: targetChannel.id, - }); + try { + await this.channelFollowingsRepository.insert({ + id: this.idService.gen(), + followerId: requestUser.id, + followeeId: targetChannel.id, + }); + } catch (e) { + if (isDuplicateKeyValueError(e)) { + throw new IdentifiableError('6e335e39-0203-4418-a936-b3f2dc987845', 'already following'); + } + throw e; + } this.globalEventService.publishInternalEvent('followChannel', { userId: requestUser.id, diff --git a/packages/backend/src/core/CoreModule.ts b/packages/backend/src/core/CoreModule.ts index 8b1adccd0d..bc59095682 100644 --- a/packages/backend/src/core/CoreModule.ts +++ b/packages/backend/src/core/CoreModule.ts @@ -159,10 +159,12 @@ import { WorldRoomEntityService } from './entities/WorldRoomEntityService.js'; import { WorldRoomMultiplayService } from './WorldRoomMultiplayService.js'; import { WorldAvatarService } from './WorldAvatarService.js'; import { WorldAvatarEntityService } from './entities/WorldAvatarEntityService.js'; +import { TelemetryService } from './telemetry/TelemetryService.js'; import type { Provider } from '@nestjs/common'; //#region 文字列ベースでのinjection用(循環参照対応のため) const $LoggerService: Provider = { provide: 'LoggerService', useExisting: LoggerService }; +const $TelemetryService: Provider = { provide: 'TelemetryService', useExisting: TelemetryService }; const $AbuseReportService: Provider = { provide: 'AbuseReportService', useExisting: AbuseReportService }; const $AbuseReportNotificationService: Provider = { provide: 'AbuseReportNotificationService', useExisting: AbuseReportNotificationService }; const $AccountMoveService: Provider = { provide: 'AccountMoveService', useExisting: AccountMoveService }; @@ -473,6 +475,7 @@ const $ApQuestionService: Provider = { provide: 'ApQuestionService', useExisting ApPersonService, ApQuestionService, QueueService, + TelemetryService, //#region 文字列ベースでのinjection用(循環参照対応のため) $LoggerService, @@ -626,6 +629,7 @@ const $ApQuestionService: Provider = { provide: 'ApQuestionService', useExisting $ApNoteService, $ApPersonService, $ApQuestionService, + $TelemetryService, //#endregion ], exports: [ @@ -782,6 +786,7 @@ const $ApQuestionService: Provider = { provide: 'ApQuestionService', useExisting ApPersonService, ApQuestionService, QueueService, + TelemetryService, //#region 文字列ベースでのinjection用(循環参照対応のため) $LoggerService, @@ -933,6 +938,7 @@ const $ApQuestionService: Provider = { provide: 'ApQuestionService', useExisting $ApNoteService, $ApPersonService, $ApQuestionService, + $TelemetryService, //#endregion ], }) diff --git a/packages/backend/src/core/EmailService.ts b/packages/backend/src/core/EmailService.ts index 384704b252..5324864ad1 100644 --- a/packages/backend/src/core/EmailService.ts +++ b/packages/backend/src/core/EmailService.ts @@ -3,11 +3,9 @@ * SPDX-License-Identifier: AGPL-3.0-only */ -import { URLSearchParams } from 'node:url'; import * as nodemailer from 'nodemailer'; import juice from 'juice'; import { Inject, Injectable } from '@nestjs/common'; -import { validate as validateEmail } from 'deep-email-validator'; import { UtilityService } from '@/core/UtilityService.js'; import { DI } from '@/di-symbols.js'; import type { Config } from '@/config.js'; @@ -197,6 +195,7 @@ export class EmailService { } else if (this.meta.enableTruemailApi && this.meta.truemailInstance && this.meta.truemailAuthKey != null) { validated = await this.trueMail(this.meta.truemailInstance, emailAddress, this.meta.truemailAuthKey); } else { + const { validate: validateEmail } = await import('deep-email-validator'); validated = await validateEmail({ email: emailAddress, validateRegex: true, diff --git a/packages/backend/src/core/HashtagService.ts b/packages/backend/src/core/HashtagService.ts index beed786d1b..750fbb06ff 100644 --- a/packages/backend/src/core/HashtagService.ts +++ b/packages/backend/src/core/HashtagService.ts @@ -16,11 +16,20 @@ import { UserEntityService } from '@/core/entities/UserEntityService.js'; import { bindThis } from '@/decorators.js'; import { FeaturedService } from '@/core/FeaturedService.js'; import { UtilityService } from '@/core/UtilityService.js'; -import { isDuplicateKeyValueError } from '@/misc/is-duplicate-key-value-error.js'; import Logger from '../logger.js'; const logger = new Logger('hashtag/create'); +type AttachedOrMentioned = 'attached' | 'mentioned'; +type UpdatingHashtagColumn = { + totalUserIds: keyof MiHashtag & `${AttachedOrMentioned}UserIds`, + totalUsersCount: keyof MiHashtag & `${AttachedOrMentioned}UsersCount`, + localUserIds: keyof MiHashtag & `${AttachedOrMentioned}LocalUserIds`, + localUsersCount: keyof MiHashtag & `${AttachedOrMentioned}LocalUsersCount`, + remoteUserIds: keyof MiHashtag & `${AttachedOrMentioned}RemoteUserIds`, + remoteUsersCount: keyof MiHashtag & `${AttachedOrMentioned}RemoteUsersCount`, +}; + @Injectable() export class HashtagService { constructor( @@ -68,126 +77,93 @@ export class HashtagService { // TODO: サンプリング this.updateHashtagsRanking(tag, user.id); - { - const index = await this.hashtagsRepository.findOneBy({ name: tag }); + const column: UpdatingHashtagColumn = isUserAttached ? { + totalUserIds: 'attachedUserIds', + totalUsersCount: 'attachedUsersCount', + localUserIds: 'attachedLocalUserIds', + localUsersCount: 'attachedLocalUsersCount', + remoteUserIds: 'attachedRemoteUserIds', + remoteUsersCount: 'attachedRemoteUsersCount', + } : { + totalUserIds: 'mentionedUserIds', + totalUsersCount: 'mentionedUsersCount', + localUserIds: 'mentionedLocalUserIds', + localUsersCount: 'mentionedLocalUsersCount', + remoteUserIds: 'mentionedRemoteUserIds', + remoteUsersCount: 'mentionedRemoteUsersCount', + }; - if (index == null && inc) { - try { - if (isUserAttached) { - await this.hashtagsRepository.insert({ - id: this.idService.gen(), - name: tag, - mentionedUserIds: [], - mentionedUsersCount: 0, - mentionedLocalUserIds: [], - mentionedLocalUsersCount: 0, - mentionedRemoteUserIds: [], - mentionedRemoteUsersCount: 0, - attachedUserIds: [user.id], - attachedUsersCount: 1, - attachedLocalUserIds: this.userEntityService.isLocalUser(user) ? [user.id] : [], - attachedLocalUsersCount: this.userEntityService.isLocalUser(user) ? 1 : 0, - attachedRemoteUserIds: this.userEntityService.isRemoteUser(user) ? [user.id] : [], - attachedRemoteUsersCount: this.userEntityService.isRemoteUser(user) ? 1 : 0, - } as MiHashtag); - } else { - await this.hashtagsRepository.insert({ - id: this.idService.gen(), - name: tag, - mentionedUserIds: [user.id], - mentionedUsersCount: 1, - mentionedLocalUserIds: this.userEntityService.isLocalUser(user) ? [user.id] : [], - mentionedLocalUsersCount: this.userEntityService.isLocalUser(user) ? 1 : 0, - mentionedRemoteUserIds: this.userEntityService.isRemoteUser(user) ? [user.id] : [], - mentionedRemoteUsersCount: this.userEntityService.isRemoteUser(user) ? 1 : 0, - attachedUserIds: [], - attachedUsersCount: 0, - attachedLocalUserIds: [], - attachedLocalUsersCount: 0, - attachedRemoteUserIds: [], - attachedRemoteUsersCount: 0, - } as MiHashtag); - } - return; - } catch (err) { - if (isDuplicateKeyValueError(err)) { - logger.info(`Duplicate insertion detected. Falling back to update. #${tag}`); - } else { - throw err; - } - } - } + if (inc) { + await this.#incrementHashTag(user, tag, column); + } else { + await this.#decrementHashTag(user, tag, column); + } + } + + async #incrementHashTag( + user: { id: MiUser['id']; host: MiUser['host']; }, + tag: string, + columns: UpdatingHashtagColumn, + ) { + const isLocal = this.userEntityService.isLocalUser(user); + const { totalUserIds, totalUsersCount } = columns; + const localOrRemoteUserIds = isLocal ? columns.localUserIds : columns.remoteUserIds; + const localOrRemoteUserCount = isLocal ? columns.localUsersCount : columns.remoteUsersCount; + + const runner = this.db.createQueryRunner('master'); + try { + await runner.query( + `INSERT into "hashtag"("id", "name", "${totalUserIds}", "${totalUsersCount}", "${localOrRemoteUserIds}", + "${localOrRemoteUserCount}") + VALUES ($3, $1, ARRAY [$2], 1, ARRAY [$2], 1) + ON CONFLICT ("name") + DO UPDATE SET "${totalUserIds}" = ${appendUserIdIfNotExists(totalUserIds)}, + "${totalUsersCount}" = ${incrementCountIfNotExists(totalUserIds, totalUsersCount)}, + "${localOrRemoteUserIds}" = ${appendUserIdIfNotExists(localOrRemoteUserIds)}, + "${localOrRemoteUserCount}" = ${incrementCountIfNotExists(localOrRemoteUserIds, localOrRemoteUserCount)}`, + [tag, user.id, this.idService.gen()], + ); + } finally { + await runner.release(); } - await this.db.transaction(async transactionalEntityManager => { - const transactionalHashtagRepository = transactionalEntityManager - .getRepository(MiHashtag); + function appendUserIdIfNotExists(userIds: keyof MiHashtag & `${string}UserIds`): string { + return `CASE WHEN NOT ("hashtag"."${userIds}" @> ARRAY[$2 ::varchar]) THEN array_append("hashtag"."${userIds}", $2) ELSE "hashtag"."${userIds}" END`; + } - const index = await transactionalHashtagRepository - .createQueryBuilder() - .setLock('pessimistic_write') - .where('name = :name', { name: tag }) - .getOne(); + function incrementCountIfNotExists(userIds: keyof MiHashtag & `${string}UserIds`, userCount: keyof MiHashtag & `${string}UsersCount`): string { + return `CASE WHEN NOT ("hashtag"."${userIds}" @> ARRAY[$2 ::varchar]) THEN "hashtag"."${userCount}" + 1 ELSE "hashtag"."${userCount}" END`; + } + } - if (index == null) return; + async #decrementHashTag( + user: { id: MiUser['id']; host: MiUser['host']; }, + tag: string, + columns: UpdatingHashtagColumn, + ) { + const isLocal = this.userEntityService.isLocalUser(user); + const { totalUserIds, totalUsersCount } = columns; + const localOrRemoteUserIds = isLocal ? columns.localUserIds : columns.remoteUserIds; + const localOrRemoteUserCount = isLocal ? columns.localUsersCount : columns.remoteUsersCount; - const set = {} as any; + const runner = this.db.createQueryRunner('master'); + try { + await runner.query( + `UPDATE "hashtag" + SET "${totalUserIds}" = array_remove("${totalUserIds}", $2), + "${totalUsersCount}" = ${decrementIfExists(totalUserIds, totalUsersCount)}, + "${localOrRemoteUserIds}" = array_remove("${localOrRemoteUserIds}", $2), + "${localOrRemoteUserCount}" = ${decrementIfExists(localOrRemoteUserIds, localOrRemoteUserCount)} + WHERE "name" = $1`, + [tag, user.id], + ); + } finally { + await runner.release(); + } - if (isUserAttached) { - if (inc) { - // 自分が初めてこのタグを使ったなら - if (!index.attachedUserIds.some(id => id === user.id)) { - set.attachedUserIds = () => `array_append("attachedUserIds", '${user.id}')`; - set.attachedUsersCount = () => '"attachedUsersCount" + 1'; - } - // 自分が(ローカル内で)初めてこのタグを使ったなら - if (this.userEntityService.isLocalUser(user) && !index.attachedLocalUserIds.some(id => id === user.id)) { - set.attachedLocalUserIds = () => `array_append("attachedLocalUserIds", '${user.id}')`; - set.attachedLocalUsersCount = () => '"attachedLocalUsersCount" + 1'; - } - // 自分が(リモートで)初めてこのタグを使ったなら - if (this.userEntityService.isRemoteUser(user) && !index.attachedRemoteUserIds.some(id => id === user.id)) { - set.attachedRemoteUserIds = () => `array_append("attachedRemoteUserIds", '${user.id}')`; - set.attachedRemoteUsersCount = () => '"attachedRemoteUsersCount" + 1'; - } - } else { - set.attachedUserIds = () => `array_remove("attachedUserIds", '${user.id}')`; - set.attachedUsersCount = () => '"attachedUsersCount" - 1'; - if (this.userEntityService.isLocalUser(user)) { - set.attachedLocalUserIds = () => `array_remove("attachedLocalUserIds", '${user.id}')`; - set.attachedLocalUsersCount = () => '"attachedLocalUsersCount" - 1'; - } else { - set.attachedRemoteUserIds = () => `array_remove("attachedRemoteUserIds", '${user.id}')`; - set.attachedRemoteUsersCount = () => '"attachedRemoteUsersCount" - 1'; - } - } - } else { - // 自分が初めてこのタグを使ったなら - if (!index.mentionedUserIds.some(id => id === user.id)) { - set.mentionedUserIds = () => `array_append("mentionedUserIds", '${user.id}')`; - set.mentionedUsersCount = () => '"mentionedUsersCount" + 1'; - } - // 自分が(ローカル内で)初めてこのタグを使ったなら - if (this.userEntityService.isLocalUser(user) && !index.mentionedLocalUserIds.some(id => id === user.id)) { - set.mentionedLocalUserIds = () => `array_append("mentionedLocalUserIds", '${user.id}')`; - set.mentionedLocalUsersCount = () => '"mentionedLocalUsersCount" + 1'; - } - // 自分が(リモートで)初めてこのタグを使ったなら - if (this.userEntityService.isRemoteUser(user) && !index.mentionedRemoteUserIds.some(id => id === user.id)) { - set.mentionedRemoteUserIds = () => `array_append("mentionedRemoteUserIds", '${user.id}')`; - set.mentionedRemoteUsersCount = () => '"mentionedRemoteUsersCount" + 1'; - } - } - - if (Object.keys(set).length > 0) { - await transactionalHashtagRepository - .createQueryBuilder() - .update() - .where('id = :id', { id: index.id }) - .set(set) - .execute(); - } - }); + function decrementIfExists(userIds: keyof MiHashtag & `${string}UserIds`, userCount: keyof MiHashtag & `${string}UsersCount`): string { + return `CASE WHEN ("${userIds}" @> ARRAY[$2]) THEN "${userCount}" - 1 ELSE "${userCount}" END`; + } } @bindThis diff --git a/packages/backend/src/core/NoteCreateService.ts b/packages/backend/src/core/NoteCreateService.ts index 8ea1630d4d..05e9854184 100644 --- a/packages/backend/src/core/NoteCreateService.ts +++ b/packages/backend/src/core/NoteCreateService.ts @@ -520,6 +520,11 @@ export class NoteCreateService implements OnApplicationShutdown { // specified / direct noteはreject throw new Error('Renote target is not public or home'); } + + // ローカルのみをRenoteしたらローカルのみにする + if (data.renote.localOnly && data.channel == null) { + data.localOnly = true; + } } // Check blocking @@ -534,19 +539,35 @@ export class NoteCreateService implements OnApplicationShutdown { } } - // 返信対象がpublicではないならhomeにする - if (data.reply && data.reply.visibility !== 'public' && data.visibility === 'public') { - data.visibility = 'home'; - } + if (data.reply) { + switch (data.reply.visibility) { + case 'public': + // public noteは無条件にreply可能 + break; + case 'home': + // home noteはhome以下にreply可能 + if (data.visibility === 'public') { + data.visibility = 'home'; + } + break; + case 'followers': + // followers noteはfollowers以下にreply可能 + if (data.visibility === 'public' || data.visibility === 'home') { + data.visibility = 'followers'; + } + break; + case 'specified': + // specified / direct noteはspecifiedのみreply可能 + if (data.visibility !== 'specified') { + data.visibility = 'specified'; + } + break; + } - // ローカルのみをRenoteしたらローカルのみにする - if (data.renote && data.renote.localOnly && data.channel == null) { - data.localOnly = true; - } - - // ローカルのみにリプライしたらローカルのみにする - if (data.reply && data.reply.localOnly && data.channel == null) { - data.localOnly = true; + // ローカルのみにリプライしたらローカルのみにする + if (data.reply.localOnly && data.channel == null) { + data.localOnly = true; + } } if (data.text) { diff --git a/packages/backend/src/core/QueueModule.ts b/packages/backend/src/core/QueueModule.ts index ecd96261e0..ad25631152 100644 --- a/packages/backend/src/core/QueueModule.ts +++ b/packages/backend/src/core/QueueModule.ts @@ -9,6 +9,7 @@ import { DI } from '@/di-symbols.js'; import type { Config } from '@/config.js'; import { baseQueueOptions, QUEUE } from '@/queue/const.js'; import { allSettled } from '@/misc/promise-tracker.js'; +import { instrumentQueue } from '@/core/telemetry/queue-instrumentation.js'; import { DeliverJobData, EndedPollNotificationJobData, @@ -31,63 +32,72 @@ export type ObjectStorageQueue = Bull.Queue; export type UserWebhookDeliverQueue = Bull.Queue; export type SystemWebhookDeliverQueue = Bull.Queue; +function createQueue(queueName: string, config: Config): Bull.Queue { + const queue = new Bull.Queue(queueName, baseQueueOptions(config, queueName)); + // Queue のラップは、enqueue 時に OTel context をジョブデータへ埋め込むためのもの。 + // Sentry 単独ではジョブ間の context 伝播を使わないので、OTel 未設定時は元の Queue を返す。 + if (config.otelForBackend == null) return queue; + + return instrumentQueue(queue); +} + const $system: Provider = { provide: 'queue:system', - useFactory: (config: Config) => new Bull.Queue(QUEUE.SYSTEM, baseQueueOptions(config, QUEUE.SYSTEM)), + useFactory: (config: Config) => createQueue>(QUEUE.SYSTEM, config), inject: [DI.config], }; const $endedPollNotification: Provider = { provide: 'queue:endedPollNotification', - useFactory: (config: Config) => new Bull.Queue(QUEUE.ENDED_POLL_NOTIFICATION, baseQueueOptions(config, QUEUE.ENDED_POLL_NOTIFICATION)), + useFactory: (config: Config) => createQueue(QUEUE.ENDED_POLL_NOTIFICATION, config), inject: [DI.config], }; const $postScheduledNote: Provider = { provide: 'queue:postScheduledNote', - useFactory: (config: Config) => new Bull.Queue(QUEUE.POST_SCHEDULED_NOTE, baseQueueOptions(config, QUEUE.POST_SCHEDULED_NOTE)), + useFactory: (config: Config) => createQueue(QUEUE.POST_SCHEDULED_NOTE, config), inject: [DI.config], }; const $deliver: Provider = { provide: 'queue:deliver', - useFactory: (config: Config) => new Bull.Queue(QUEUE.DELIVER, baseQueueOptions(config, QUEUE.DELIVER)), + useFactory: (config: Config) => createQueue(QUEUE.DELIVER, config), inject: [DI.config], }; const $inbox: Provider = { provide: 'queue:inbox', - useFactory: (config: Config) => new Bull.Queue(QUEUE.INBOX, baseQueueOptions(config, QUEUE.INBOX)), + useFactory: (config: Config) => createQueue(QUEUE.INBOX, config), inject: [DI.config], }; const $db: Provider = { provide: 'queue:db', - useFactory: (config: Config) => new Bull.Queue(QUEUE.DB, baseQueueOptions(config, QUEUE.DB)), + useFactory: (config: Config) => createQueue>(QUEUE.DB, config), inject: [DI.config], }; const $relationship: Provider = { provide: 'queue:relationship', - useFactory: (config: Config) => new Bull.Queue(QUEUE.RELATIONSHIP, baseQueueOptions(config, QUEUE.RELATIONSHIP)), + useFactory: (config: Config) => createQueue(QUEUE.RELATIONSHIP, config), inject: [DI.config], }; const $objectStorage: Provider = { provide: 'queue:objectStorage', - useFactory: (config: Config) => new Bull.Queue(QUEUE.OBJECT_STORAGE, baseQueueOptions(config, QUEUE.OBJECT_STORAGE)), + useFactory: (config: Config) => createQueue>(QUEUE.OBJECT_STORAGE, config), inject: [DI.config], }; const $userWebhookDeliver: Provider = { provide: 'queue:userWebhookDeliver', - useFactory: (config: Config) => new Bull.Queue(QUEUE.USER_WEBHOOK_DELIVER, baseQueueOptions(config, QUEUE.USER_WEBHOOK_DELIVER)), + useFactory: (config: Config) => createQueue(QUEUE.USER_WEBHOOK_DELIVER, config), inject: [DI.config], }; const $systemWebhookDeliver: Provider = { provide: 'queue:systemWebhookDeliver', - useFactory: (config: Config) => new Bull.Queue(QUEUE.SYSTEM_WEBHOOK_DELIVER, baseQueueOptions(config, QUEUE.SYSTEM_WEBHOOK_DELIVER)), + useFactory: (config: Config) => createQueue(QUEUE.SYSTEM_WEBHOOK_DELIVER, config), inject: [DI.config], }; diff --git a/packages/backend/src/core/telemetry/TelemetryService.ts b/packages/backend/src/core/telemetry/TelemetryService.ts new file mode 100644 index 0000000000..8a62a521df --- /dev/null +++ b/packages/backend/src/core/telemetry/TelemetryService.ts @@ -0,0 +1,35 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { Injectable } from '@nestjs/common'; +import { bindThis } from '@/decorators.js'; +import { captureMessage, shutdownTelemetry, startSpan, startSpanWithTraceContext } from './telemetry-registry.js'; +import type { OnApplicationShutdown } from '@nestjs/common'; +import type { TelemetryCaptureMessageOptions } from './adapters/TelemetryAdapter.js'; + +@Injectable() +export class TelemetryService implements OnApplicationShutdown { + @bindThis + public captureMessage(message: string, opts: TelemetryCaptureMessageOptions): void { + captureMessage(message, opts); + } + + @bindThis + public startSpan(name: string, fn: () => T): T { + return startSpan(name, fn); + } + + @bindThis + public startSpanWithTraceContext(name: string, jobData: object, fn: () => T): T { + // jobData に enqueue 元の context があれば worker span へ復元する。 + // context の無い既存ジョブは、通常の startSpan と同じ扱いになる。 + return startSpanWithTraceContext(name, jobData, fn); + } + + @bindThis + public async onApplicationShutdown(_signal?: string): Promise { + await shutdownTelemetry(); + } +} diff --git a/packages/backend/src/core/telemetry/adapters/OpenTelemetryAdapter.ts b/packages/backend/src/core/telemetry/adapters/OpenTelemetryAdapter.ts new file mode 100644 index 0000000000..c3d5064a6b --- /dev/null +++ b/packages/backend/src/core/telemetry/adapters/OpenTelemetryAdapter.ts @@ -0,0 +1,259 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import * as os from 'node:os'; +import cluster from 'node:cluster'; +import { envOption } from '@/env.js'; +import { registerDiagLogger } from '@/core/telemetry/telemetry-diag.js'; +import { installHttpClientInstrumentation } from '@/core/telemetry/http-client-instrumentation.js'; +import { installDatabaseInstrumentation } from '@/core/telemetry/database-instrumentation.js'; +import { installRedisInstrumentation } from '@/core/telemetry/redis-instrumentation.js'; +import { executeSpan, getQueueTraceContextMode, injectActiveTraceContext, recordSpanError, startSpanWithQueueTraceContext } from '@/core/telemetry/queue-trace-context.js'; +import type { LogTraceContext } from '@/logging/types.js'; +import type { Span, SpanStatusCode, Tracer } from '@opentelemetry/api'; +import type { Resource, ResourceDetector } from '@opentelemetry/resources'; +import type { ParentBasedSampler, Sampler } from '@opentelemetry/sdk-trace-base'; +import type { OtelBackendRuntimeConfig, TelemetryAdapter, TelemetryCaptureMessageOptions } from './TelemetryAdapter.js'; +import type { QueueTraceContextCarrier, QueueTraceContextDeps } from '../queue-trace-context.js'; + +const DEFAULT_SHUTDOWN_TIMEOUT = 5000; + +type OpenTelemetryAdapterDeps = { + tracer: Pick; + provider: { + shutdown(): Promise; + }; + getActiveSpan: () => Span | undefined; + spanStatusCodeError: SpanStatusCode; + shutdownTimeout: number; + shutdownHttpClientInstrumentation?: () => void; + shutdownDatabaseInstrumentation?: () => void; + shutdownRedisInstrumentation?: () => void; + queueTraceContext?: QueueTraceContextDeps; +}; + +type CreateSamplerDeps = { + ParentBasedSampler: new (config: { root: Sampler }) => ParentBasedSampler; + TraceIdRatioBasedSampler: new (sampleRate: number) => Sampler; +}; + +type CreateResourceDeps = { + defaultResource: () => Resource; + resourceFromAttributes: (attributes: Record) => Resource; + detectResources: (config: { detectors: ResourceDetector[] }) => Resource; + envDetector: ResourceDetector; + serviceNameAttribute: string; + serviceInstanceIdAttribute: string; + serviceVersionAttribute: string; + serviceVersion: string; +}; + +export class OpenTelemetryAdapter implements TelemetryAdapter { + public constructor( + private readonly deps: OpenTelemetryAdapterDeps, + ) { + } + + public static async create(config: OtelBackendRuntimeConfig): Promise { + const [ + { context, diag, DiagLogLevel, propagation, ROOT_CONTEXT, SpanKind, SpanStatusCode, trace }, + { W3CTraceContextPropagator }, + { OTLPTraceExporter }, + { defaultResource, detectResources, envDetector, resourceFromAttributes }, + { BatchSpanProcessor, ParentBasedSampler, TraceIdRatioBasedSampler }, + { NodeTracerProvider }, + { ATTR_SERVICE_INSTANCE_ID, ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION }, + ] = await Promise.all([ + import('@opentelemetry/api'), + import('@opentelemetry/core'), + import('@opentelemetry/exporter-trace-otlp-proto'), + import('@opentelemetry/resources'), + import('@opentelemetry/sdk-trace-base'), + import('@opentelemetry/sdk-trace-node'), + import('@opentelemetry/semantic-conventions'), + ]); + + // OTel SDK内部のexport失敗は既定だと見えにくいため、Misskeyのloggerへ橋渡しする。 + registerDiagLogger(diag, DiagLogLevel.WARN); + + // endpoint/headersを未指定にしておくと、OTEL_EXPORTER_OTLP_* 環境変数の標準fallbackが効く。 + const exporter = new OTLPTraceExporter({ + ...(config.endpoint != null ? { url: config.endpoint } : {}), + ...(config.headers != null ? { headers: config.headers } : {}), + }); + const spanProcessor = new BatchSpanProcessor(exporter); + + // SDK 2.xではSpanProcessorをprovider生成時に渡す。ここでOTel単体用のproviderを作る。 + const provider = new NodeTracerProvider({ + resource: createResource(config, { + defaultResource, + resourceFromAttributes, + detectResources, + envDetector, + serviceNameAttribute: ATTR_SERVICE_NAME, + serviceInstanceIdAttribute: ATTR_SERVICE_INSTANCE_ID, + serviceVersionAttribute: ATTR_SERVICE_VERSION, + serviceVersion: config.serviceVersion, + }), + ...(config.sampleRate != null ? { sampler: createSampler(config.sampleRate, { + ParentBasedSampler, + TraceIdRatioBasedSampler, + }) } : {}), + spanProcessors: [spanProcessor], + }); + + // HTTP送信には注入しないが、将来のQueue連結でpropagation APIを使える状態にする。 + provider.register({ + propagator: new W3CTraceContextPropagator(), + }); + + // provider操作をdepsに閉じ込め、span wrapper本体をユニットテストしやすくする。 + const tracer = provider.getTracer('misskey-backend'); + + return new OpenTelemetryAdapter({ + tracer, + provider, + getActiveSpan: () => trace.getActiveSpan(), + spanStatusCodeError: SpanStatusCode.ERROR, + shutdownTimeout: DEFAULT_SHUTDOWN_TIMEOUT, + shutdownHttpClientInstrumentation: installHttpClientInstrumentation({ + tracer, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + }), + // pg のrequire hookとioredis diagnostics channelは、Nest moduleの動的importより前に有効化する。 + shutdownDatabaseInstrumentation: await installDatabaseInstrumentation(provider, { + capturePgSpans: config.capturePgSpans === true, + capturePgStatement: config.capturePgStatement === true, + capturePgConnectionSpans: config.capturePgConnectionSpans === true, + }), + shutdownRedisInstrumentation: installRedisInstrumentation(tracer, SpanKind.CLIENT, SpanStatusCode.ERROR, { + captureConnectionSpans: config.captureRedisConnectionSpans === true, + captureCommandSpans: config.captureRedisCommandSpans === true, + requireParentSpan: config.captureRedisRootSpans !== true, + }), + queueTraceContext: { + tracer, + propagation, + trace, + getActiveContext: () => context.active(), + rootContext: ROOT_CONTEXT, + mode: getQueueTraceContextMode(config.jobTraceContextMode), + spanStatusCodeError: SpanStatusCode.ERROR, + }, + }); + } + + public captureMessage(message: string, _opts: TelemetryCaptureMessageOptions): void { + // captureMessageは例外通知APIなので、OTelでは対象spanにエラー状態を付ける。 + // アクティブspanが無い場合(例: BullMQのjob処理が既に完了しspanが閉じた後の'failed'イベント)でも + // 通知を握り潰さないよう、報告専用の短命spanを作ってそこに記録する。 + const span = this.deps.getActiveSpan(); + if (span != null) { + recordSpanError(span, new Error(message), this.deps.spanStatusCodeError); + return; + } + + this.deps.tracer.startActiveSpan('captureMessage', reportSpan => { + recordSpanError(reportSpan, new Error(message), this.deps.spanStatusCodeError); + reportSpan.end(); + }); + } + + /** activeなSpanの識別子を、Logging基盤で扱える形式へ変換します。 */ + public getActiveTraceContext(): LogTraceContext | undefined { + const activeSpan = this.deps.getActiveSpan(); + if (activeSpan == null) return undefined; + + const { traceId, spanId, traceFlags } = activeSpan.spanContext(); + return { traceId, spanId, traceFlags }; + } + + public startSpan(name: string, fn: () => T): T { + // 既存のTelemetryAdapter契約に合わせ、同期/非同期どちらでも同じspan lifetimeを保証する。 + return this.deps.tracer.startActiveSpan(name, span => executeSpan(span, fn, this.deps.spanStatusCodeError)); + } + + public injectTraceContext(carrier: QueueTraceContextCarrier): void { + const queueTraceContext = this.deps.queueTraceContext; + // Queue context 用の依存は任意なので、無い場合はジョブデータを変更しない。 + if (queueTraceContext == null) return; + injectActiveTraceContext(queueTraceContext, carrier); + } + + public startSpanWithTraceContext(name: string, jobData: object, fn: () => T): T { + const queueTraceContext = this.deps.queueTraceContext; + // Queue context 用の依存が無い場合は、従来の span 作成経路と同じ動作を保つ。 + if (queueTraceContext == null) return this.startSpan(name, fn); + + return startSpanWithQueueTraceContext(queueTraceContext, name, jobData, fn, () => this.startSpan(name, fn)); + } + + public async shutdown(): Promise { + this.deps.shutdownHttpClientInstrumentation?.(); + this.deps.shutdownDatabaseInstrumentation?.(); + this.deps.shutdownRedisInstrumentation?.(); + // BatchSpanProcessorのflushが詰まってもプロセス終了を妨げないよう、上限時間を設ける。 + // タイムアウト側のtimerは、flushが先に終わった場合にイベントループを無駄に引き留めないようclearする。 + let timer: NodeJS.Timeout | undefined; + await Promise.race([ + this.deps.provider.shutdown(), + new Promise(resolve => { + timer = setTimeout(resolve, this.deps.shutdownTimeout); + }), + ]).finally(() => { + if (timer != null) clearTimeout(timer); + }); + } +} + +export function createResource(config: OtelBackendRuntimeConfig, deps: CreateResourceDeps): Resource { + // resourceを明示指定するとSDKのdefaultResource()は自動付与されなくなる(マージではなく上書き)ため、 + // telemetry.sdk.*等の標準属性を失わないよう明示的にmergeする。 + const misskeyDefaultResource = deps.resourceFromAttributes({ + [deps.serviceNameAttribute]: 'misskey-backend', + [deps.serviceInstanceIdAttribute]: `${os.hostname()}:${process.pid}`, + [deps.serviceVersionAttribute]: deps.serviceVersion, + 'misskey.process.role': getMisskeyProcessRole(), + }); + + // OTel標準の OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES を尊重する。 + // mergeは右辺が優先されるため、config.resourceAttributesを最優先にする。 + return deps.defaultResource() + .merge(misskeyDefaultResource) + .merge(deps.detectResources({ detectors: [deps.envDetector] })) + .merge(deps.resourceFromAttributes(config.resourceAttributes ?? {})); +} + +export function createSampler(sampleRate: number, deps: CreateSamplerDeps): ParentBasedSampler { + // 設定ミスを無言でAlwaysOn/AlwaysOffに倒さず、起動時に明確に失敗させる。 + // (YAMLでクォートされた数値文字列などnumber型の保証が無い値が来てもここで弾く) + if (typeof sampleRate !== 'number' || !Number.isFinite(sampleRate) || sampleRate < 0 || sampleRate > 1) { + throw new Error('otelForBackend.sampleRate must be a number between 0.0 and 1.0.'); + } + + return new deps.ParentBasedSampler({ + root: new deps.TraceIdRatioBasedSampler(sampleRate), + }); +} + +export function getMisskeyProcessRole(): string { + // Trace backend上でserver/queue/workerを見分けられるよう、Misskey固有の役割をresourceに載せる。 + if (envOption.disableClustering) { + if (envOption.onlyServer) return 'primary-server'; + if (envOption.onlyQueue) return 'primary-queue'; + return 'primary-server+queue'; + } + + if (cluster.isPrimary) { + if (envOption.onlyServer) return 'fork-only'; + if (envOption.onlyQueue) return 'primary-queue'; + return 'primary-server'; + } + + // worker.tsのworkerMainに合わせる: onlyServerならserver()、それ以外はjobQueue()を実行する。 + if (envOption.onlyServer) return 'worker-server'; + return 'worker-queue'; +} diff --git a/packages/backend/src/core/telemetry/adapters/SentryTelemetryAdapter.ts b/packages/backend/src/core/telemetry/adapters/SentryTelemetryAdapter.ts new file mode 100644 index 0000000000..fd65f414f6 --- /dev/null +++ b/packages/backend/src/core/telemetry/adapters/SentryTelemetryAdapter.ts @@ -0,0 +1,209 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import Logger from '@/logger.js'; +import { registerDiagLogger } from '@/core/telemetry/telemetry-diag.js'; +import { getQueueTraceContextMode, injectActiveTraceContext, startSpanWithQueueTraceContext } from '@/core/telemetry/queue-trace-context.js'; +import type { LogTraceContext } from '@/logging/types.js'; +import type * as SentryNode from '@sentry/node'; +import type { NodeOptions } from '@sentry/node'; +import type { OtelBackendRuntimeConfig, SentryBackendConfig, TelemetryAdapter, TelemetryCaptureMessageOptions } from './TelemetryAdapter.js'; +import type { QueueTraceContextCarrier, QueueTraceContextDeps } from '../queue-trace-context.js'; + +// OpenTelemetryAdapterのDEFAULT_SHUTDOWN_TIMEOUTと揃え、Sentryのtransportが詰まってもプロセス終了を妨げないようにする。 +const DEFAULT_SHUTDOWN_TIMEOUT = 5000; +const logger = new Logger('telemetry', 'green'); + +type SentryIntegrationsOption = NonNullable; +// eslint-disable-next-line @typescript-eslint/no-explicit-any +type SentryIntegrationFactory = Extract any[]>; +type SentryIntegration = Parameters[0][number]; +type SentryNodeOptions = NodeOptions; + +type BuildSentryIntegrationsOptions = { + disabledIntegrations?: string[]; + enableNodeProfiling: boolean; + nodeProfilingIntegration?: () => SentryIntegration; + warn?: (message: string) => void; +}; + +export function buildSentryIntegrations(options: BuildSentryIntegrationsOptions): SentryIntegrationFactory { + return (defaults) => { + const disabledIntegrations = new Set(options.disabledIntegrations ?? []); + const defaultIntegrationNames = new Set(defaults.map((integration) => integration.name)); + const unknownIntegrations = [...disabledIntegrations].filter((name) => !defaultIntegrationNames.has(name)); + + if (unknownIntegrations.length > 0) { + (options.warn ?? console.warn)(`Unknown Sentry integration configured in sentryForBackend.disabledIntegrations: ${unknownIntegrations.join(', ')}`); + } + + return [ + ...defaults.filter((integration) => !disabledIntegrations.has(integration.name)), + ...(options.enableNodeProfiling && options.nodeProfilingIntegration != null ? [options.nodeProfilingIntegration()] : []), + ]; + }; +} + +export function buildSentryNodeOptions( + config: SentryBackendConfig, + nodeProfilingIntegration?: () => SentryIntegration, +): SentryNodeOptions { + return { + // Do not send Sentry trace headers to remote ActivityPub/Webhook/etc. hosts by default. + // Admins can opt in for trusted internal services via sentryForBackend.options. + tracePropagationTargets: [], + + // Performance Monitoring + tracesSampleRate: 1.0, // Capture 100% of the transactions + + // Set sampling rate for profiling - this is relative to tracesSampleRate + profilesSampleRate: 1.0, + + maxBreadcrumbs: 0, + + ...config.options, + + integrations: buildSentryIntegrations({ + disabledIntegrations: config.disabledIntegrations, + enableNodeProfiling: config.enableNodeProfiling, + nodeProfilingIntegration, + }), + }; +} + +type BuildSentryOtlpInitOptions = { + sentryConfig: SentryBackendConfig; + otelConfig: OtelBackendRuntimeConfig; + otlpProcessor: unknown; + nodeProfilingIntegration?: () => SentryIntegration; + warn?: (message: string) => void; +}; + +export function buildSentryOtlpInitOptions(options: BuildSentryOtlpInitOptions): SentryNodeOptions { + // OTel併存時も、remoteへtrace headerを漏らさないデフォルトはSentry単体時と揃える。 + // propagateTraceToRemote: true か、options.tracePropagationTargets の明示指定がある場合のみ既定を上書きする。 + const { tracePropagationTargets, ...sentryOptions } = options.sentryConfig.options; + const propagateTraceToRemote = options.otelConfig.propagateTraceToRemote === true || tracePropagationTargets != null; + const warn = options.warn ?? ((message: string) => logger.warn(message)); + + if (options.otelConfig.sampleRate != null) { + warn('otelForBackend.sampleRate is ignored when sentryForBackend is also configured; configure sentryForBackend.options.tracesSampleRate or tracesSampler instead.'); + } + + if (options.otelConfig.resourceAttributes != null) { + warn('otelForBackend.resourceAttributes is ignored when sentryForBackend is also configured; configure OTEL_RESOURCE_ATTRIBUTES instead.'); + } + + return { + ...buildSentryNodeOptions({ + ...options.sentryConfig, + options: { + ...sentryOptions, + ...(propagateTraceToRemote ? { tracePropagationTargets } : {}), + }, + }, options.nodeProfilingIntegration), + + // Sentryの単一TracerProviderにOTLP processorを追加し、親欠損や二重providerを避ける。 + openTelemetrySpanProcessors: [ + ...(options.sentryConfig.options.openTelemetrySpanProcessors ?? []), + options.otlpProcessor as NonNullable[number], + ], + }; +} + +export class SentryTelemetryAdapter implements TelemetryAdapter { + private constructor( + private readonly Sentry: typeof SentryNode, + private readonly queueTraceContext?: QueueTraceContextDeps, + ) { + } + + public static async create(config: SentryBackendConfig): Promise { + const Sentry = await import('@sentry/node'); + const { nodeProfilingIntegration } = await import('@sentry/profiling-node'); + + Sentry.init(buildSentryNodeOptions(config, nodeProfilingIntegration)); + + return new SentryTelemetryAdapter(Sentry); + } + + public static async createWithOtlpExport( + sentryConfig: SentryBackendConfig, + otelConfig: OtelBackendRuntimeConfig, + ): Promise { + const Sentry = await import('@sentry/node'); + const { nodeProfilingIntegration } = await import('@sentry/profiling-node'); + const { context, diag, DiagLogLevel, propagation, ROOT_CONTEXT, SpanStatusCode, trace } = await import('@opentelemetry/api'); + const { BatchSpanProcessor } = await import('@opentelemetry/sdk-trace-base'); + const { OTLPTraceExporter } = await import('@opentelemetry/exporter-trace-otlp-proto'); + + registerDiagLogger(diag, DiagLogLevel.WARN); + + // OTLP送信だけを担うprocessorを作り、provider生成はSentry.init側に任せる。 + const otlpProcessor = new BatchSpanProcessor(new OTLPTraceExporter({ + ...(otelConfig.endpoint != null ? { url: otelConfig.endpoint } : {}), + ...(otelConfig.headers != null ? { headers: otelConfig.headers } : {}), + })); + + // SentryとOTLPを同一providerに集約することで、どちらの宛先にも同じspan実体を流す。 + Sentry.init(buildSentryOtlpInitOptions({ + sentryConfig, + otelConfig, + otlpProcessor, + nodeProfilingIntegration, + })); + + // Sentry が初期化した同じ OTel provider から tracer/context API を受け取り、 + // Queue を跨ぐ context 伝播も Sentry と OTLP の両方へ同一 span として出力する。 + return new SentryTelemetryAdapter(Sentry, { + tracer: trace.getTracer('misskey-backend'), + propagation, + trace, + getActiveContext: () => context.active(), + rootContext: ROOT_CONTEXT, + mode: getQueueTraceContextMode(otelConfig.jobTraceContextMode), + spanStatusCodeError: SpanStatusCode.ERROR, + }); + } + + public captureMessage(message: string, opts: TelemetryCaptureMessageOptions): void { + this.Sentry.captureMessage(message, { + level: opts.level, + ...(opts.userId != null ? { user: { id: opts.userId } } : {}), + extra: opts.extra, + }); + } + + /** activeなSpanの識別子を、Logging基盤で扱える形式へ変換します。 */ + public getActiveTraceContext(): LogTraceContext | undefined { + const activeSpan = this.Sentry.getActiveSpan(); + if (activeSpan == null) return undefined; + + const { traceId, spanId, traceFlags } = activeSpan.spanContext(); + return { traceId, spanId, traceFlags }; + } + + public startSpan(name: string, fn: () => T): T { + return this.Sentry.startSpan({ name }, fn); + } + + public injectTraceContext(carrier: QueueTraceContextCarrier): void { + // Sentry 単体構成では queueTraceContext を持たず、従来どおりジョブデータを変更しない。 + if (this.queueTraceContext == null) return; + injectActiveTraceContext(this.queueTraceContext, carrier); + } + + public startSpanWithTraceContext(name: string, jobData: object, fn: () => T): T { + // Sentry 単体構成では Sentry 既存の span 作成経路を使う。 + if (this.queueTraceContext == null) return this.startSpan(name, fn); + + return startSpanWithQueueTraceContext(this.queueTraceContext, name, jobData, fn, () => this.startSpan(name, fn)); + } + + public async shutdown(): Promise { + // timeout未指定だとtransportのflushが詰まった際にプロセス終了を妨げるため、上限時間を設ける。 + await this.Sentry.close(DEFAULT_SHUTDOWN_TIMEOUT); + } +} diff --git a/packages/backend/src/core/telemetry/adapters/TelemetryAdapter.ts b/packages/backend/src/core/telemetry/adapters/TelemetryAdapter.ts new file mode 100644 index 0000000000..29cbb61f4c --- /dev/null +++ b/packages/backend/src/core/telemetry/adapters/TelemetryAdapter.ts @@ -0,0 +1,65 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { Config } from '@/config.js'; +import type { LogTraceContext } from '@/logging/types.js'; +import type { QueueTraceContextCarrier } from '../queue-trace-context.js'; + +export type SentryBackendConfig = NonNullable; +export type OtelBackendConfig = NonNullable; +export type OtelBackendRuntimeConfig = OtelBackendConfig & { + serviceVersion: string; +}; + +export interface TelemetryCaptureMessageOptions { + /** 現在はエラー通知用途だけに絞る。追加する場合は各adapterでの扱いを揃えること。 */ + level: 'error'; + + /** Sentryではuser.idへ渡す。OTel adapterは現在span属性へ付与していないため、必要ならadapter側で拡張する。 */ + userId?: string; + + /** queue名やendpoint名など、通知先で調査に使う補助情報。 */ + extra?: Record; +} + +/** + * Sentry・OpenTelemetryなど、エラートラッキング/APMサービスごとの実装差異を隠蔽するための抽象。 + * 新しいサービスを追加する場合はこのインターフェースを実装するアダプタをこのディレクトリに追加し、 + * telemetry-registry.tsのinitTelemetry内で登録する。 + */ +export interface TelemetryAdapter { + /** + * 実行中の処理で起きたエラー相当の事象を記録する。 + * Sentryはmessage通知、OTelはactive spanまたは短命spanへの例外記録として扱う。 + */ + captureMessage(message: string, opts: TelemetryCaptureMessageOptions): void; + + /** 現在のactive Spanからログへ付加するTrace Contextを取得する。 */ + getActiveTraceContext?(): LogTraceContext | undefined; + + /** + * API endpointやqueue jobなど、呼び出し側の処理単位をspanで包む。 + * fnの戻り値・例外はそのまま呼び出し側へ返し、Promiseの場合はsettleまでspanを閉じない。 + */ + startSpan(name: string, fn: () => T): T; + + /** + * BullMQ のジョブデータへ保存する carrier に、active trace context を注入する。 + * OTel を使わない adapter は実装しない。 + */ + injectTraceContext?(carrier: QueueTraceContextCarrier): void; + + /** + * ジョブに保存された enqueue 元の context を、worker span の Link または parent として復元する。 + * context を持たないジョブの互換性は adapter 側で保つ。 + */ + startSpanWithTraceContext?(name: string, jobData: object, fn: () => T): T; + + /** + * プロセス終了時にtelemetry backendへ残りのデータをflushする。 + * 実装側ではtransport停止に引きずられないよう、待機時間に上限を設ける。 + */ + shutdown(): Promise; +} diff --git a/packages/backend/src/core/telemetry/database-instrumentation.ts b/packages/backend/src/core/telemetry/database-instrumentation.ts new file mode 100644 index 0000000000..206fa4c916 --- /dev/null +++ b/packages/backend/src/core/telemetry/database-instrumentation.ts @@ -0,0 +1,86 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { Span, TracerProvider } from '@opentelemetry/api'; + +type Instrumentation = { + setTracerProvider(provider: TracerProvider): void; + enable(): void; + disable(): void; +}; + +type PgInstrumentationConfig = { + enhancedDatabaseReporting: boolean; + requireParentSpan: boolean; + ignoreConnectSpans: boolean; + requestHook?(span: Span): void; +}; + +type InstrumentationConstructor = new (config: PgInstrumentationConfig) => Instrumentation; + +type InstrumentationDeps = { + PgInstrumentation: InstrumentationConstructor; +}; + +type DatabaseInstrumentationOptions = { + capturePgStatement: boolean; + capturePgConnectionSpans: boolean; +}; + +type InstallDatabaseInstrumentationOptions = DatabaseInstrumentationOptions & { + capturePgSpans: boolean; +}; + +/** + * pg はアプリケーションが import する前に有効化しないと、require hook 型の + * 自動計装がモジュールを patch できない。そのため、 + * telemetry provider の登録直後にこの関数を呼び出す。 + */ +export async function installDatabaseInstrumentation(provider: TracerProvider, options: InstallDatabaseInstrumentationOptions): Promise<() => void> { + if (!options.capturePgSpans) return () => {}; + + const { PgInstrumentation } = await import('@opentelemetry/instrumentation-pg'); + + return installInstrumentation(provider, { PgInstrumentation }, options); +} + +export function installInstrumentation(provider: TracerProvider, deps: InstrumentationDeps, options: DatabaseInstrumentationOptions = { + capturePgStatement: false, + capturePgConnectionSpans: false, +}): () => void { + const instrumentations = [ + new deps.PgInstrumentation({ + // SQLパラメータには投稿内容・認証情報などが含まれ得るため、常に記録しない。 + enhancedDatabaseReporting: false, + requireParentSpan: true, + ignoreConnectSpans: !options.capturePgConnectionSpans, + // instrumentation-pgはSQL本文を無加工で属性へ追加する。明示opt-in時だけ残す。 + ...(options.capturePgStatement ? {} : { + requestHook: (span: Span) => { + span.setAttribute('db.statement', '[REDACTED]'); + span.setAttribute('db.query.text', '[REDACTED]'); + }, + }), + }), + ]; + + try { + for (const instrumentation of instrumentations) { + instrumentation.setTracerProvider(provider); + instrumentation.enable(); + } + } catch (error) { + for (const instrumentation of instrumentations) { + instrumentation.disable(); + } + throw error; + } + + return () => { + for (const instrumentation of instrumentations) { + instrumentation.disable(); + } + }; +} diff --git a/packages/backend/src/core/telemetry/http-client-instrumentation.ts b/packages/backend/src/core/telemetry/http-client-instrumentation.ts new file mode 100644 index 0000000000..a2ae4b5692 --- /dev/null +++ b/packages/backend/src/core/telemetry/http-client-instrumentation.ts @@ -0,0 +1,122 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { channel } from 'node:diagnostics_channel'; +import type { ClientRequest, IncomingMessage } from 'node:http'; +import type { Span, SpanOptions, SpanStatusCode, Tracer } from '@opentelemetry/api'; + +const HTTP_CLIENT_REQUEST_CREATED = 'http.client.request.created'; +const HTTP_CLIENT_RESPONSE_FINISH = 'http.client.response.finish'; +const HTTP_CLIENT_REQUEST_ERROR = 'http.client.request.error'; + +type HttpClientSpan = Pick; + +type HttpClientInstrumentationDeps = { + tracer: Pick; + spanKindClient: SpanOptions['kind']; + spanStatusCodeError: SpanStatusCode; + subscribe: (name: string, listener: (message: unknown) => void) => () => void; +}; + +type RequestCreatedMessage = { request: ClientRequest }; +type ResponseFinishMessage = { request: ClientRequest; response: IncomingMessage }; +type RequestErrorMessage = { request: ClientRequest; error: Error }; + +/** + * require フックを使わず、Node.js 組み込み HTTP クライアントの diagnostics channel を計装する。 + * telemetry 初期化前に読み込まれたモジュールも対象になる。 + */ +export function createHttpClientInstrumentation(deps: HttpClientInstrumentationDeps): () => void { + const spans = new WeakMap(); + + const unsubscribeCreated = deps.subscribe(HTTP_CLIENT_REQUEST_CREATED, (message: unknown) => { + const { request } = message as RequestCreatedMessage; + const { url, host, port } = getRequestDetails(request); + const method = request.method ?? 'GET'; + const span = deps.tracer.startSpan(method, { + kind: deps.spanKindClient, + attributes: { + 'http.request.method': method, + 'url.full': url, + 'server.address': host, + 'server.port': port, + }, + }); + spans.set(request, span); + }); + + const unsubscribeResponseFinish = deps.subscribe(HTTP_CLIENT_RESPONSE_FINISH, (message: unknown) => { + const { request, response } = message as ResponseFinishMessage; + const span = spans.get(request); + if (span == null) return; + + const statusCode = response.statusCode; + if (statusCode != null) { + span.setAttribute('http.response.status_code', statusCode); + } + if (response.httpVersion != null) { + span.setAttribute('network.protocol.version', response.httpVersion); + } + if (statusCode != null && statusCode >= 400) { + span.setAttribute('error.type', String(statusCode)); + span.setStatus({ code: deps.spanStatusCodeError }); + } + span.end(); + spans.delete(request); + }); + + const unsubscribeRequestError = deps.subscribe(HTTP_CLIENT_REQUEST_ERROR, (message: unknown) => { + const { request, error } = message as RequestErrorMessage; + const span = spans.get(request); + if (span == null) return; + + span.recordException(error); + span.setAttribute('error.type', getErrorType(error)); + span.setStatus({ code: deps.spanStatusCodeError }); + span.end(); + spans.delete(request); + }); + + return () => { + unsubscribeCreated(); + unsubscribeResponseFinish(); + unsubscribeRequestError(); + }; +} + +export function installHttpClientInstrumentation(deps: Omit): () => void { + return createHttpClientInstrumentation({ + ...deps, + subscribe: (name, listener) => { + const diagnosticChannel = channel(name); + diagnosticChannel.subscribe(listener); + return () => diagnosticChannel.unsubscribe(listener); + }, + }); +} + +function getRequestDetails(request: ClientRequest): { url: string; host: string; port: number } { + const protocol = request.protocol ?? 'http:'; + const host = request.getHeader('host')?.toString() ?? request.host ?? 'localhost'; + const url = new URL(request.path || '/', `${protocol}//${host}`); + // URL 属性には認証情報やクエリ文字列を含めない。 + url.username = ''; + url.password = ''; + url.search = ''; + url.hash = ''; + + return { + url: url.toString(), + host: url.hostname, + // URL.port は既定ポートでは空文字列になるため、スキームから補う。 + port: url.port === '' ? (url.protocol === 'https:' ? 443 : 80) : Number(url.port), + }; +} + +function getErrorType(error: Error): string { + // Node.js の system error code は安定した低カーディナリティの識別子になる。 + const code = (error as NodeJS.ErrnoException).code; + return code ?? error.name; +} diff --git a/packages/backend/src/core/telemetry/queue-instrumentation.ts b/packages/backend/src/core/telemetry/queue-instrumentation.ts new file mode 100644 index 0000000000..90eb710efa --- /dev/null +++ b/packages/backend/src/core/telemetry/queue-instrumentation.ts @@ -0,0 +1,33 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { injectTraceContext } from './telemetry-registry.js'; +import { injectQueueTraceContext } from './queue-trace-context.js'; +import type * as Bull from 'bullmq'; + +/** + * Queue の add/addBulk をラップし、全ての BullMQ enqueue 経路を一箇所で捕捉する。 + * QueueService を通さず直接 add/addBulk する呼び出し元もあるため、それぞれで注入すると漏れやすい。 + */ +export function instrumentQueue(queue: Bull.Queue): Bull.Queue { + // BullMQ のメソッドは Queue インスタンスを this として使うため、差し替え前に bind して保持する。 + const add = queue.add.bind(queue); + queue.add = ((name, data, opts) => { + // BullMQ が data を Redis 用にシリアライズする前に、enqueue 元の context を内部フィールドへ追加する。 + injectQueueTraceContext(data, injectTraceContext); + return add(name, data, opts); + }) as typeof queue.add; + + // addBulk は複数ジョブを一度にシリアライズするので、各 data へ同じ context を注入する。 + const addBulk = queue.addBulk.bind(queue); + queue.addBulk = ((jobs) => { + for (const job of jobs) { + injectQueueTraceContext(job.data, injectTraceContext); + } + return addBulk(jobs); + }) as typeof queue.addBulk; + + return queue; +} diff --git a/packages/backend/src/core/telemetry/queue-trace-context.ts b/packages/backend/src/core/telemetry/queue-trace-context.ts new file mode 100644 index 0000000000..d9f7f7462d --- /dev/null +++ b/packages/backend/src/core/telemetry/queue-trace-context.ts @@ -0,0 +1,182 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { Context, PropagationAPI, Span, SpanContext, SpanOptions, SpanStatusCode, Tracer } from '@opentelemetry/api'; + +/* + * enqueue (push) から worker による取得・処理 (pop) までをテレメトリ上で関連付け、 + * Queue を挟む非同期処理を一連の流れとして追跡できるようにする。 + * そのため、trace context を次の流れで伝播する: + * + * 1. producer 側で active context を W3C Trace Context 形式の carrier へ注入する。 + * 2. carrier をジョブデータの内部フィールドに保存し、BullMQ/Redis 経由で worker へ渡す。 + * 3. worker 側で carrier を context へ戻し、設定に応じて Link または parent に使う。 + * + * OTel API への依存を引数にしているのは、OTel 単体構成と Sentry + OTLP 構成で + * 同じ伝播処理を使うため。 + */ +/** Redis 上のジョブデータにだけ保存する、OpenTelemetry propagator 用の carrier。 */ +export type QueueTraceContextCarrier = Record; +export type QueueTraceContextMode = 'link' | 'parent'; + +// 通常のジョブプロセッサが参照しない Misskey 内部フィールドとして、ユーザー定義の data と区別する。 +const QUEUE_TRACE_CONTEXT_KEY = '__misskeyTraceContext'; + +export type QueueTraceContextDeps = { + tracer: Pick; + propagation: Pick; + trace: { + getSpanContext(context: Context): SpanContext | undefined; + }; + getActiveContext: () => Context; + rootContext: Context; + mode: QueueTraceContextMode; + spanStatusCodeError: SpanStatusCode; +}; + +export type QueueSpanContext = { + options: SpanOptions; + parentContext: Context; +}; + +type QueueSpanContextDeps = Pick; + +/** + * enqueue 元の active context を、ジョブ本来のデータを壊さない内部フィールドとして保持する。 + * propagator が何も注入しなかった場合は、Redis に不要な空オブジェクトを残さない。 + */ +export function injectQueueTraceContext(data: unknown, inject: (carrier: QueueTraceContextCarrier) => void): void { + // BullMQ の型定義外から渡される値も考慮し、書き込めない値は無視する。 + if (data == null || typeof data !== 'object') return; + + const carrier: QueueTraceContextCarrier = {}; + inject(carrier); + + // active span が無い場合、propagator は何も注入しない。空の内部フィールドは Redis に保存しない。 + if (Object.keys(carrier).length === 0) return; + Object.assign(data, { [QUEUE_TRACE_CONTEXT_KEY]: carrier }); +} + +/** 現在実行中の span を propagator の標準形式で carrier へ書き出す。 */ +export function injectActiveTraceContext(deps: QueueTraceContextDeps, carrier: QueueTraceContextCarrier): void { + deps.propagation.inject(deps.getActiveContext(), carrier); +} + +/** + * ジョブに保存された carrier から、worker span の開始に必要な context と options を組み立てる。 + * + * - parent: enqueue span の子として同じ trace を継続する。 + * - link: worker span を別の root trace にし、enqueue span への関連だけを Link に残す。 + * + * link モードで trace を分けることで、worker 側の sampling 判定を enqueue 側から独立させられる。 + */ +export function getQueueSpanContext(data: unknown, deps: QueueSpanContextDeps): QueueSpanContext | undefined { + const carrier = getQueueTraceContextCarrier(data); + if (carrier == null) return undefined; + + // Redis から復元した carrier は別プロセス由来なので、現在の context ではなく ROOT_CONTEXT から展開する。 + const extractedContext = deps.propagation.extract(deps.rootContext, carrier); + if (deps.mode === 'parent') { + return { + options: {}, + parentContext: extractedContext, + }; + } + + // Link が受け取るのは Context ではなく SpanContext なので、extract 後に取り出す。 + const spanContext = deps.trace.getSpanContext(extractedContext); + return { + options: { + root: true, + ...(spanContext != null ? { links: [{ context: spanContext }] } : {}), + }, + parentContext: deps.rootContext, + }; +} + +/** + * context を持つジョブは Link/parent の規則で span を開始する。 + * デプロイ前に enqueue されたジョブなど、context を持たない場合は既存の adapter 固有実装へ委ねる。 + */ +export function startSpanWithQueueTraceContext( + deps: QueueTraceContextDeps, + name: string, + jobData: object, + fn: () => T, + fallback: () => T, +): T { + const spanContext = getQueueSpanContext(jobData, deps); + if (spanContext == null) return fallback(); + + return deps.tracer.startActiveSpan(name, spanContext.options, spanContext.parentContext, span => executeSpan(span, fn, deps.spanStatusCodeError)); +} + +/** + * 既存の TelemetryAdapter 契約に合わせ、同期値と Promise のどちらを返す処理も span で包む。 + * 成功・失敗のどちらでも処理完了まで span を開き、必ず一度だけ閉じる。 + */ +export function executeSpan(span: Span, fn: () => T, spanStatusCodeError: SpanStatusCode): T { + try { + const result = fn(); + if (isPromiseLike(result)) { + // fn() の戻り値は T のまま保ちつつ、Promise の settle 時に span を閉じる。 + return result.then( + value => { + span.end(); + return value; + }, + error => { + recordSpanError(span, error, spanStatusCodeError); + span.end(); + throw error; + }, + ) as T; + } + + span.end(); + return result; + } catch (error) { + // fn() が同期的に throw した場合も、Promise の reject と同じ形で記録して呼び出し元へ戻す。 + recordSpanError(span, error, spanStatusCodeError); + span.end(); + throw error; + } +} + +/** Error 以外の throw 値も OTel exporter が扱える例外に正規化し、span をエラー状態にする。 */ +export function recordSpanError(span: Span, error: unknown, spanStatusCodeError: SpanStatusCode): void { + const exception = error instanceof Error ? error : new Error(String(error)); + span.recordException(exception); + span.setStatus({ + code: spanStatusCodeError, + message: exception.message, + }); +} + +/** + * 未設定時は、enqueue と worker の sampling を分離できる link を使う。 + * 設定ミスを無言でフォールバックさせないため、未知の値は起動時にエラーにする。 + */ +export function getQueueTraceContextMode(mode: unknown): QueueTraceContextMode { + if (mode == null || mode === 'link') return 'link'; + if (mode === 'parent') return 'parent'; + throw new Error('otelForBackend.jobTraceContextMode must be either \'link\' or \'parent\'.'); +} + +function getQueueTraceContextCarrier(data: unknown): QueueTraceContextCarrier | undefined { + if (data == null || typeof data !== 'object') return undefined; + const carrier = (data as Record)[QUEUE_TRACE_CONTEXT_KEY]; + if (carrier == null || typeof carrier !== 'object' || Array.isArray(carrier)) return undefined; + + // Redis 上の旧いジョブや壊れたデータで worker を落とさないよう、propagator に渡せる string map だけを受け入れる。 + const entries = Object.entries(carrier); + if (entries.length === 0 || entries.some(([, value]) => typeof value !== 'string')) return undefined; + return Object.fromEntries(entries) as QueueTraceContextCarrier; +} + +function isPromiseLike(value: T): value is T & PromiseLike> { + // native Promise に限らず thenable も span の完了を待てるよう、instanceof ではなく then の有無で判定する。 + return value != null && typeof (value as { then?: unknown }).then === 'function'; +} diff --git a/packages/backend/src/core/telemetry/redis-instrumentation.ts b/packages/backend/src/core/telemetry/redis-instrumentation.ts new file mode 100644 index 0000000000..b3e8d80b39 --- /dev/null +++ b/packages/backend/src/core/telemetry/redis-instrumentation.ts @@ -0,0 +1,178 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { tracingChannel } from 'node:diagnostics_channel'; +import { context, trace } from '@opentelemetry/api'; +import type { Span, SpanKind, SpanStatusCode, Tracer } from '@opentelemetry/api'; + +type IORedisCommandContext = { + command: string; + args: string[]; + database: number; + serverAddress: string; + serverPort: number | undefined; +}; + +type IORedisConnectContext = { + serverAddress: string; + serverPort: number | undefined; +}; + +type TracingChannelSubscribers = { + start(message: T): void; + end(message: T & { error?: unknown }): void; + asyncStart(message: T & { error?: unknown }): void; + asyncEnd(message: T & { error?: unknown }): void; + error(message: T & { error: unknown }): void; +}; + +type TracingChannel = { + subscribe(subscribers: TracingChannelSubscribers): void; + unsubscribe(subscribers: TracingChannelSubscribers): void; +}; + +type RedisInstrumentationDeps = { + tracingChannel(name: string): TracingChannel; + tracer: Pick; + getActiveSpan(): Span | undefined; + spanKindClient: SpanKind; + spanStatusCodeError: SpanStatusCode; +}; + +type RedisInstrumentationOptions = { + captureCommandSpans?: boolean; + /** requireParentSpan is the official ioredis instrumentation's default. */ + requireParentSpan?: boolean; + captureConnectionSpans?: boolean; +}; + +/** + * ioredis 5.11以降が公開する native diagnostics channel を購読する。 + * バンドル後もioredis自身が発行するイベントを使うため、require hook に + * 依存せず、ESM import 経路でもコマンド span を取得できる。 + */ +export function installRedisInstrumentation( + tracer: Pick, + spanKindClient: SpanKind, + spanStatusCodeError: SpanStatusCode, + options: RedisInstrumentationOptions = {}, +): () => void { + return createRedisInstrumentation({ + tracingChannel, + tracer, + getActiveSpan: () => trace.getSpan(context.active()), + spanKindClient, + spanStatusCodeError, + }, options); +} + +export function createRedisInstrumentation(deps: RedisInstrumentationDeps, options: RedisInstrumentationOptions = {}): () => void { + const requireParentSpan = options.requireParentSpan ?? true; + const cleanup: Array<() => void> = []; + if (options.captureCommandSpans === true) { + const commandChannel = deps.tracingChannel('ioredis:command'); + const commandSubscribers = createTracingChannelSubscribers(commandChannel, deps, requireParentSpan, message => ({ + name: message.command, + attributes: { + 'db.system.name': 'redis', + 'db.namespace': message.database.toString(10), + 'db.operation.name': message.command, + 'server.address': message.serverAddress, + ...(message.serverPort != null ? { 'server.port': message.serverPort } : {}), + }, + })); + cleanup.push(() => commandChannel.unsubscribe(commandSubscribers)); + } + if (options.captureConnectionSpans === true) { + const connectChannel = deps.tracingChannel('ioredis:connect'); + // Connection spans are explicitly opt-in and should include startup and reconnect attempts. + const connectSubscribers = createTracingChannelSubscribers(connectChannel, deps, false, message => ({ + name: 'connect', + attributes: { + 'db.system.name': 'redis', + 'db.operation.name': 'connect', + 'server.address': message.serverAddress, + ...(message.serverPort != null ? { 'server.port': message.serverPort } : {}), + }, + })); + cleanup.push(() => connectChannel.unsubscribe(connectSubscribers)); + } + + return () => { + for (const unsubscribe of cleanup) { + unsubscribe(); + } + }; +} + +function createTracingChannelSubscribers( + channel: TracingChannel, + deps: RedisInstrumentationDeps, + requireParentSpan: boolean, + getSpanOptions: (message: T) => { name: string; attributes: Record }, +): TracingChannelSubscribers { + const spans = new WeakMap(); + + const recordError = (state: { span: Span; recordedError: boolean }, value: unknown): void => { + if (state.recordedError) return; + const error = toError(value); + state.span.recordException(error); + state.span.setStatus({ code: deps.spanStatusCodeError, message: error.message }); + state.span.setAttribute('error.type', getErrorType(error)); + const statusCode = getRedisErrorStatusCode(error.message); + if (statusCode != null) state.span.setAttribute('db.response.status_code', statusCode); + state.recordedError = true; + }; + + const finish = (message: T & { error?: unknown }): void => { + const state = spans.get(message); + if (state == null) return; + + if (message.error != null) { + recordError(state, message.error); + } + state.span.end(); + spans.delete(message); + }; + + const subscribers: TracingChannelSubscribers = { + start: (message) => { + if (requireParentSpan && deps.getActiveSpan() == null) return; + + const options = getSpanOptions(message); + const span = deps.tracer.startSpan(options.name, { + kind: deps.spanKindClient, + attributes: options.attributes, + }); + spans.set(message, { span, recordedError: false }); + }, + // Promiseを返すコマンドでは完了前にもendイベントが来るため、同期例外だけここで閉じる。 + end: (message) => { + if (message.error != null) finish(message); + }, + asyncStart: () => {}, + asyncEnd: finish, + error: (message) => { + const state = spans.get(message); + if (state == null) return; + recordError(state, message.error); + }, + }; + + channel.subscribe(subscribers); + return subscribers; +} + +function toError(value: unknown): Error { + return value instanceof Error ? value : new Error(String(value)); +} + +function getRedisErrorStatusCode(message: string): string | undefined { + return message.match(/^([A-Z][A-Z0-9_]*)\b/)?.[1]; +} + +function getErrorType(error: Error): string { + return (error as NodeJS.ErrnoException).code ?? error.name; +} diff --git a/packages/backend/src/core/telemetry/telemetry-diag.ts b/packages/backend/src/core/telemetry/telemetry-diag.ts new file mode 100644 index 0000000000..9d72d0a6bf --- /dev/null +++ b/packages/backend/src/core/telemetry/telemetry-diag.ts @@ -0,0 +1,40 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import Logger from '@/logger.js'; +import type { DiagAPI, DiagLogger, DiagLogLevel } from '@opentelemetry/api'; + +export function registerDiagLogger( + diagApi: DiagAPI, + diagLogLevelWarn: DiagLogLevel, +): void { + // diagはプロセスグローバルなので、通常運用で必要なWARN以上だけをMisskeyのログに流す。 + const logger = new Logger('otel', 'green'); + const diagLogger: DiagLogger = { + error: (message, ...args) => logger.error(formatDiagMessage(message, args)), + warn: (message, ...args) => logger.warn(formatDiagMessage(message, args)), + info: (message, ...args) => logger.info(formatDiagMessage(message, args)), + debug: (message, ...args) => logger.debug(formatDiagMessage(message, args)), + verbose: (message, ...args) => logger.debug(formatDiagMessage(message, args)), + }; + + diagApi.setLogger(diagLogger, { + logLevel: diagLogLevelWarn, + suppressOverrideMessage: true, + }); +} + +function formatDiagMessage(message: string, args: unknown[]): string { + if (args.length === 0) return message; + return `${message} ${args.map(arg => { + if (arg instanceof Error) return arg.stack ?? arg.message; + if (typeof arg === 'string') return arg; + try { + return JSON.stringify(arg); + } catch { + return String(arg); + } + }).join(' ')}`; +} diff --git a/packages/backend/src/core/telemetry/telemetry-registry.ts b/packages/backend/src/core/telemetry/telemetry-registry.ts new file mode 100644 index 0000000000..94e26e21d8 --- /dev/null +++ b/packages/backend/src/core/telemetry/telemetry-registry.ts @@ -0,0 +1,82 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { Config } from '@/config.js'; +import { setLogTraceContextProvider } from '@/logging/logging-runtime.js'; +import { OpenTelemetryAdapter } from './adapters/OpenTelemetryAdapter.js'; +import { SentryTelemetryAdapter } from './adapters/SentryTelemetryAdapter.js'; +import type { OtelBackendRuntimeConfig, TelemetryAdapter, TelemetryCaptureMessageOptions } from './adapters/TelemetryAdapter.js'; +import type { QueueTraceContextCarrier } from './queue-trace-context.js'; + +/** + * NestのDIコンテナが構築される前(boot処理内)で初期化する必要があるため、 + * DIを介さないモジュールレベルの状態として有効なアダプタを保持する。 + * TelemetryServiceはこの状態への薄いラッパーとして振る舞う。 + */ +const adapters: TelemetryAdapter[] = []; + +export async function initTelemetry(config: Config): Promise { + const otelForBackend: OtelBackendRuntimeConfig | undefined = config.otelForBackend == null ? undefined : { + ...config.otelForBackend, + serviceVersion: config.version, + }; + + // SentryとOTelを同時に使う場合はproviderを分けず、Sentry側へOTLP processorを追加する。 + let adapter: TelemetryAdapter | undefined; + if (config.sentryForBackend && otelForBackend) { + adapter = await SentryTelemetryAdapter.createWithOtlpExport(config.sentryForBackend, otelForBackend); + } else if (config.sentryForBackend) { + // Sentry単体時は既存のSentry adapterだけを登録する。 + adapter = await SentryTelemetryAdapter.create(config.sentryForBackend); + } else if (otelForBackend) { + // OTel単体時だけMisskey自身でNodeTracerProviderを立てる。 + adapter = await OpenTelemetryAdapter.create(otelForBackend); + } + + if (adapter != null) { + adapters.push(adapter); + // Telemetryの初期化後に登録し、初期化前のBootstrapログは従来どおり出力する。 + setLogTraceContextProvider(() => adapter.getActiveTraceContext?.()); + } +} + +export function captureMessage(message: string, opts: TelemetryCaptureMessageOptions): void { + // 有効なadapterすべてへ通知し、宛先ごとの差異はadapter内に閉じ込める。 + for (const adapter of adapters) { + adapter.captureMessage(message, opts); + } +} + +export function startSpan(name: string, fn: () => T): T { + // 有効なadapterが無い/1つだけの場合(実運用上の大半のケース)は、 + // 毎リクエスト/ジョブでreduceRightのclosureを組み立てる無駄を避ける。 + if (adapters.length === 0) return fn(); + if (adapters.length === 1) return adapters[0].startSpan(name, fn); + + // 将来複数adapterを登録する場合でも同じ処理を入れ子にラップし、呼び出し側のAPIは1回のstartSpanに保つ。 + const wrapped = adapters.reduceRight<() => T>( + (inner, adapter) => () => adapter.startSpan(name, inner), + fn, + ); + return wrapped(); +} + +export function injectTraceContext(carrier: QueueTraceContextCarrier): void { + // Queue の carrier は共有データなので、通知と異なり全 adapter にブロードキャストしない。 + // OTel provider は現在 1 つだけなので、同じ header を上書きしないよう最初の対応 adapter だけを使う。 + adapters.find(adapter => adapter.injectTraceContext != null)?.injectTraceContext?.(carrier); +} + +export function startSpanWithTraceContext(name: string, jobData: object, fn: () => T): T { + // Queue context を解釈できる adapter に span 作成を任せる。 + // 対応 adapter が無い構成では通常の startSpan へフォールバックする。 + const adapter = adapters.find(adapter => adapter.startSpanWithTraceContext != null); + return adapter?.startSpanWithTraceContext?.(name, jobData, fn) ?? startSpan(name, fn); +} + +export async function shutdownTelemetry(): Promise { + // 終了時は登録済みadapterを並列にflush/shutdownする。 + await Promise.allSettled(adapters.map(adapter => adapter.shutdown())); +} diff --git a/packages/backend/src/core/telemetry/telemetry-shutdown.ts b/packages/backend/src/core/telemetry/telemetry-shutdown.ts new file mode 100644 index 0000000000..1ee1b42649 --- /dev/null +++ b/packages/backend/src/core/telemetry/telemetry-shutdown.ts @@ -0,0 +1,11 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +/** + * Telemetry-specific shutdown is implemented by telemetry-registry.ts. + * Signal coordination belongs to boot/shutdown-handler.ts so telemetry and + * logging remain independent domains. + */ +export { shutdownTelemetry } from './telemetry-registry.js'; diff --git a/packages/backend/src/env.ts b/packages/backend/src/env.ts index ba44cfa2e6..c3b1cd015f 100644 --- a/packages/backend/src/env.ts +++ b/packages/backend/src/env.ts @@ -11,6 +11,7 @@ const envOption = { verbose: false, withLogTime: false, quiet: false, + enableCrossOriginIsolation: false, }; for (const key of Object.keys(envOption) as (keyof typeof envOption)[]) { diff --git a/packages/backend/src/logger.ts b/packages/backend/src/logger.ts index ce76f8d05e..4144c1cf04 100644 --- a/packages/backend/src/logger.ts +++ b/packages/backend/src/logger.ts @@ -3,109 +3,149 @@ * SPDX-License-Identifier: AGPL-3.0-only */ -import cluster from 'node:cluster'; -import chalk from 'chalk'; -import { default as convertColor } from 'color-convert'; -import { format as dateFormat } from 'date-fns'; import { bindThis } from '@/decorators.js'; -import { envOption } from './env.js'; +import { logManager } from './logging/logging-runtime.js'; +import type { LogEntryInput, LogLevel, LoggerContext, LogWriteInput } from './logging/types.js'; import type { Keyword } from 'color-convert'; -type Context = { - name: string; - color?: Keyword; -}; - -type Level = 'error' | 'success' | 'warning' | 'debug' | 'info'; +// 旧APIのdataは表示用の任意値を受け取り、Errorや配列も既存呼び出しで使用されています。 +type LegacyData = Record | null; +/** + * ロガー名の階層と従来の公開APIを提供する薄い窓口です。 + * 出力条件の判断や整形はLogManagerとLogBackendへ委譲します。 + */ // eslint-disable-next-line import/no-default-export export default class Logger { - private context: Context; - private parentLogger: Logger | null = null; + private context: readonly LoggerContext[]; + /** 指定した名前を起点とするLoggerを作成します。 */ constructor(context: string, color?: Keyword) { - this.context = { + this.context = [{ name: context, - color: color, - }; + color, + }]; } + /** + * 現在のロガーを親として、下位の名前を持つLoggerを作成します。 + */ @bindThis public createSubLogger(context: string, color?: Keyword): Logger { const logger = new Logger(context, color); - logger.parentLogger = this; + logger.context = [...this.context, ...logger.context]; return logger; } + /** + * 従来APIの引数を共通形式へ変換し、LogManagerへ渡します。 + */ @bindThis - private log(level: Level, message: string, data?: Record | null, important = false, subContexts: Context[] = []): void { - if (envOption.quiet) return; - - if (this.parentLogger) { - this.parentLogger.log(level, message, data, important, [this.context].concat(subContexts)); - return; - } - - const time = dateFormat(new Date(), 'HH:mm:ss'); - const worker = cluster.isPrimary ? '*' : cluster.worker!.id; - const l = - level === 'error' ? important ? chalk.bgRed.white('ERR ') : chalk.red('ERR ') : - level === 'warning' ? chalk.yellow('WARN') : - level === 'success' ? important ? chalk.bgGreen.white('DONE') : chalk.green('DONE') : - level === 'debug' ? chalk.gray('VERB') : - level === 'info' ? chalk.blue('INFO') : - null; - const contexts = [this.context].concat(subContexts).map(d => d.color ? chalk.rgb(...convertColor.keyword.rgb(d.color))(d.name) : chalk.white(d.name)); - const m = - level === 'error' ? chalk.red(message) : - level === 'warning' ? chalk.yellow(message) : - level === 'success' ? chalk.green(message) : - level === 'debug' ? chalk.gray(message) : - level === 'info' ? message : - null; - - let log = `${l} ${worker}\t[${contexts.join(' ')}]\t${m}`; - if (envOption.withLogTime) log = chalk.gray(time) + ' ' + log; - - const args: unknown[] = [important ? chalk.bold(log) : log]; - if (data != null) { - args.push(data); - } - console.log(...args); + private log(level: LogLevel, message: string, data?: unknown, important = false, legacyLevel?: 'success', error?: unknown): void { + logManager.write({ + level, + message, + context: this.context, + ...(typeof error !== 'undefined' ? { error } : {}), + compatibility: { + legacyLevel, + important, + data, + }, + }); } + /** level別メソッドの構造化入力にlevelとLoggerのcontextを付けて渡します。 */ @bindThis - public error(x: string | Error, data?: Record | null, important = false): void { // 実行を継続できない状況で使う + private logStructured(level: LogLevel, input: LogEntryInput): void { + this.write({ + ...input, + level, + }); + } + + /** 構造化ログをLoggerのcontext付きでLogManagerへ渡します。 */ + @bindThis + public write(input: LogWriteInput): void { + logManager.write({ + ...input, + context: this.context, + }); + } + + /** 処理を継続できない状況を記録します。 */ + public error(input: LogEntryInput): void; + public error(error: Error, data?: LegacyData, important?: boolean): void; + public error(message: string, data?: LegacyData, important?: boolean): void; + public error(errorOrMessage: string | Error, data?: LegacyData, important?: boolean): void; + @bindThis + public error(x: LogEntryInput | string | Error, data?: LegacyData, important = false): void { if (x instanceof Error) { + // エラー本体も第2引数へ残し、従来どおりスタックなどを確認できるようにします。 data = data ?? {}; data.e = x; - this.log('error', x.toString(), data, important); - } else if (typeof x === 'object') { - this.log('error', `${(x as any).message ?? (x as any).name ?? x}`, data, important); + this.log('error', x.toString(), data, important, undefined, x); + } else if (typeof x === 'string') { + this.log('error', x, data, important); } else { - this.log('error', `${x}`, data, important); + this.logStructured('error', x); } } + /** 処理は継続できるものの、改善が必要な状況を記録します。 */ + public warn(input: LogEntryInput): void; + public warn(message: string): void; + public warn(message: string, data?: LegacyData, important?: boolean): void; @bindThis - public warn(message: string, data?: Record | null, important = false): void { // 実行を継続できるが改善すべき状況で使う - this.log('warning', message, data, important); - } - - @bindThis - public succ(message: string, data?: Record | null, important = false): void { // 何かに成功した状況で使う - this.log('success', message, data, important); - } - - @bindThis - public debug(message: string, data?: Record | null, important = false): void { // デバッグ用に使う(開発者に必要だが利用者に不要な情報) - if (process.env.NODE_ENV !== 'production' || envOption.verbose) { - this.log('debug', message, data, important); + public warn(inputOrMessage: LogEntryInput | string, data?: LegacyData, important = false): void { + if (typeof inputOrMessage === 'string') { + this.log('warn', inputOrMessage, data, important); + } else { + this.logStructured('warn', inputOrMessage); } } + /** 処理が成功したことを、従来のDONE表示で記録します。 */ @bindThis - public info(message: string, data?: Record | null, important = false): void { // それ以外 - this.log('info', message, data, important); + public succ(message: string, data?: Record | null, important = false): void { + this.log('info', message, data, important, 'success'); + } + + /** 開発者向けの調査情報を記録します。 */ + public debug(input: LogEntryInput): void; + public debug(message: string): void; + public debug(message: string, data?: LegacyData, important?: boolean): void; + @bindThis + public debug(inputOrMessage: LogEntryInput | string, data?: LegacyData, important = false): void { + if (typeof inputOrMessage === 'string') { + this.log('debug', inputOrMessage, data, important); + } else { + this.logStructured('debug', inputOrMessage); + } + } + + /** 通常の動作状況を記録します。 */ + public info(input: LogEntryInput): void; + public info(message: string): void; + public info(message: string, data?: LegacyData, important?: boolean): void; + @bindThis + public info(inputOrMessage: LogEntryInput | string, data?: LegacyData, important = false): void { + if (typeof inputOrMessage === 'string') { + this.log('info', inputOrMessage, data, important); + } else { + this.logStructured('info', inputOrMessage); + } + } + + /** 致命的な状況を構造化ログとして記録します。 */ + public fatal(input: LogEntryInput): void; + public fatal(message: string): void; + @bindThis + public fatal(inputOrMessage: LogEntryInput | string): void { + if (typeof inputOrMessage === 'string') { + this.logStructured('fatal', { message: inputOrMessage }); + } else { + this.logStructured('fatal', inputOrMessage); + } } } diff --git a/packages/backend/src/logging/BootstrapConsoleBackend.ts b/packages/backend/src/logging/BootstrapConsoleBackend.ts new file mode 100644 index 0000000000..42016acef6 --- /dev/null +++ b/packages/backend/src/logging/BootstrapConsoleBackend.ts @@ -0,0 +1,54 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { LogBackend } from './LogBackend.js'; +import type { AccessLogRecord, LogRecord } from './types.js'; + +/** 設定読込前の最小出力処理が外部から受け取る依存関係です。 */ +export type BootstrapConsoleBackendDependencies = { + readonly output: (...args: unknown[]) => void; +}; + +const defaultDependencies: BootstrapConsoleBackendDependencies = { + output: (...args) => console.log(...args), +}; + +/** + * 設定読込前でも利用できる、依存の少ないコンソール出力です。 + * 設定ファイルの読み込み失敗を報告するため、Pretty backendやTelemetryには依存しません。 + */ +export class BootstrapConsoleBackend implements LogBackend { + private readonly dependencies: BootstrapConsoleBackendDependencies; + + constructor(dependencies: Partial = {}) { + this.dependencies = { + ...defaultDependencies, + ...dependencies, + }; + } + + public write(record: LogRecord): void { + const worker = record.isPrimary ? '*' : record.workerId ?? '?'; + const line = `${record.timestamp} ${record.level.toUpperCase()} ${worker}\t[${record.loggerName}]\t${record.message}`; + const args: unknown[] = [line]; + + if (record.compatibility?.data != null) { + args.push(record.compatibility.data); + } else if (record.eventName != null || record.attributes != null || record.error != null) { + args.push({ + ...(record.eventName != null ? { eventName: record.eventName } : {}), + ...(record.attributes != null ? { attributes: record.attributes } : {}), + ...(record.error != null ? { error: record.error } : {}), + }); + } + + this.dependencies.output(...args); + } + + /** 設定前のAccess logは本文を含めず、最小限の情報だけを出力します。 */ + public writeAccess(record: AccessLogRecord): void { + this.dependencies.output(`${record.timestamp} ACCESS ${record.method} ${record.route ?? '-'} ${record.statusCode}`); + } +} diff --git a/packages/backend/src/logging/JsonConsoleBackend.ts b/packages/backend/src/logging/JsonConsoleBackend.ts new file mode 100644 index 0000000000..fe6f8f0e2b --- /dev/null +++ b/packages/backend/src/logging/JsonConsoleBackend.ts @@ -0,0 +1,126 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { LogBackend } from './LogBackend.js'; +import type { AccessLogRecord, LogRecord } from './types.js'; + +/** JSON形式のログを1行で出力する処理が外部から受け取る依存関係です。 */ +export type JsonConsoleBackendDependencies = { + readonly output: (line: string) => void; +}; + +/** JSON形式で出力する項目を、運用上安定した形として定義します。 */ +type JsonLogRecord = { + readonly timestamp: string; + readonly level: LogRecord['level']; + readonly message: string; + readonly loggerName: string; + readonly eventName?: string; + readonly attributes?: LogRecord['attributes']; + readonly error?: LogRecord['error']; + readonly processId: number; + readonly isPrimary: boolean; + readonly workerId: number | null; + readonly trace_id?: string; + readonly span_id?: string; + readonly trace_flags?: number; +}; + +/** Access logのJSON出力項目です。通常ログと混同しない形を保ちます。 */ +type JsonAccessLogRecord = { + readonly type: 'access'; + readonly timestamp: string; + readonly method: string; + readonly route: string | null; + readonly statusCode: number; + readonly durationMs: number; + readonly responseSizeBytes: number | null; + readonly errorType?: string; + readonly requestBody?: AccessLogRecord['requestBody']; + readonly responseBody?: AccessLogRecord['responseBody']; + readonly processId: number; + readonly isPrimary: boolean; + readonly workerId: number | null; + readonly trace_id?: string; + readonly span_id?: string; + readonly trace_flags?: number; +}; + +const defaultDependencies: JsonConsoleBackendDependencies = { + output: line => console.log(line), +}; + +/** LogRecordからJSONへ出す項目だけを選び、内部情報を誤って含めないようにします。 */ +function createJsonLogRecord(record: LogRecord): JsonLogRecord { + // 色や旧APIの生データは表示専用の情報なので、機械向け形式へは持ち込みません。 + return { + timestamp: record.timestamp, + level: record.level, + message: record.message, + loggerName: record.loggerName, + // 任意項目は値がある場合だけ含め、空の項目を増やさないようにします。 + ...(record.eventName != null ? { eventName: record.eventName } : {}), + ...(record.attributes != null ? { attributes: record.attributes } : {}), + ...(record.error != null ? { error: record.error } : {}), + // 実行主体の情報は常に出し、ログを横断して検索できる形を保ちます。 + processId: record.processId, + isPrimary: record.isPrimary, + workerId: record.workerId, + // active Spanの情報は値が存在するログだけ、標準のsnake_case名で出力します。 + ...(record.traceId != null ? { trace_id: record.traceId } : {}), + ...(record.spanId != null ? { span_id: record.spanId } : {}), + ...(record.traceFlags != null ? { trace_flags: record.traceFlags } : {}), + }; +} + +/** Access logから公開する項目だけを選び、1行JSONの形へ整えます。 */ +function createJsonAccessLogRecord(record: AccessLogRecord): JsonAccessLogRecord { + return { + type: 'access', + timestamp: record.timestamp, + method: record.method, + route: record.route, + statusCode: record.statusCode, + durationMs: record.durationMs, + responseSizeBytes: record.responseSizeBytes, + ...(record.errorType != null ? { errorType: record.errorType } : {}), + ...(record.requestBody !== undefined ? { requestBody: record.requestBody } : {}), + ...(record.responseBody !== undefined ? { responseBody: record.responseBody } : {}), + processId: record.processId, + isPrimary: record.isPrimary, + workerId: record.workerId, + ...(record.traceId != null ? { trace_id: record.traceId } : {}), + ...(record.spanId != null ? { span_id: record.spanId } : {}), + ...(record.traceFlags != null ? { trace_flags: record.traceFlags } : {}), + }; +} + +/** + * LogRecordを1行のJSONへ変換し、ログ収集基盤が扱える標準出力へ渡します。 + * LoggerやLogManagerから出力形式を切り離し、Pretty形式と同じ記録を共有します。 + */ +export class JsonConsoleBackend implements LogBackend { + private readonly dependencies: JsonConsoleBackendDependencies; + + /** 出力処理を受け取り、テストや起動環境ごとに出力先を差し替えます。 */ + constructor(dependencies: Partial = {}) { + this.dependencies = { + ...defaultDependencies, + ...dependencies, + }; + } + + /** 1件のログをJSON文字列へ変換し、改行を含まない1回の出力として渡します。 */ + public write(record: LogRecord): void { + // JSON.stringifyが改行などをエスケープするため、1ログ1行の契約を保てます。 + this.dependencies.output(JSON.stringify(createJsonLogRecord(record))); + } + + /** Access logを1行JSONとして出力します。本文は設定で有効化された場合のみ、秘匿処理済みで含まれます。 */ + public writeAccess(record: AccessLogRecord): void { + // JSON.stringifyが改行をエスケープするため、1件を1物理行に保ちます。 + this.dependencies.output(JSON.stringify(createJsonAccessLogRecord(record))); + } +} diff --git a/packages/backend/src/logging/LogBackend.ts b/packages/backend/src/logging/LogBackend.ts new file mode 100644 index 0000000000..0b79effe7f --- /dev/null +++ b/packages/backend/src/logging/LogBackend.ts @@ -0,0 +1,24 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { AccessLogRecord, LogRecord } from './types.js'; + +/** + * 整形済みのログを実際の出力先へ渡すための共通窓口です。 + * Loggerを特定の出力形式へ依存させず、後から出力先を追加できるようにします。 + */ +export interface LogBackend { + /** ログを一件出力します。 */ + write(record: LogRecord): void; + + /** Access logを一件出力します。 */ + writeAccess?(record: AccessLogRecord): void; + + /** 保留中の出力がある場合に、すべて書き出します。 */ + flush?(): void | Promise; + + /** 出力先が持つ資源を解放します。 */ + close?(): void | Promise; +} diff --git a/packages/backend/src/logging/LogManager.ts b/packages/backend/src/logging/LogManager.ts new file mode 100644 index 0000000000..ef78f97bfe --- /dev/null +++ b/packages/backend/src/logging/LogManager.ts @@ -0,0 +1,414 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import cluster from 'node:cluster'; +import process from 'node:process'; +import { envOption } from '@/env.js'; +import { + findLegacyLogError, + normalizeLogAttributes, + normalizeLogValue, + serializeLogError, + type LogNormalizationProfile, +} from './LogNormalizer.js'; +import type { LogBackend } from './LogBackend.js'; +import type { + AccessLogConfiguration, + AccessLogRecord, + AccessLogRecordInput, + AccessLogStatusClass, + LogLevel, + LogLevelSetting, + LogRecord, + LogRecordInput, + LogTraceContext, + LogTraceContextProvider, +} from './types.js'; + +/** ログを出力したプロセスを識別するための情報です。 */ +export type LogProcessInfo = { + readonly processId: number; + readonly isPrimary: boolean; + readonly workerId: number | null; +}; + +/** + * 実行環境から取得する値をまとめた依存関係です。 + * テストでは固定値へ差し替え、時刻やプロセス状態に左右されないようにします。 + */ +export type LogManagerDependencies = { + readonly now: () => Date; + readonly getProcessInfo: () => LogProcessInfo; + readonly isQuiet: () => boolean; + readonly isVerbose: () => boolean; + readonly getNodeEnv: () => string | undefined; +}; + +/** ログ管理の初期化時に指定できる正規化設定です。 */ +export type LogManagerOptions = { + readonly normalizationProfile?: LogNormalizationProfile; +}; + +/** 起動時に適用するログ出力設定です。 */ +export type LogManagerConfiguration = { + readonly level?: LogLevelSetting; + readonly domains?: Readonly> | null; + readonly access?: AccessLogConfiguration; +}; + +/** 正規化済みのAccess log設定です。 */ +export type ResolvedAccessLogConfiguration = { + readonly statusClasses: readonly AccessLogStatusClass[]; + readonly bodies: { + readonly request: boolean; + readonly response: boolean; + readonly maxBytes: number; + }; +}; + +const logLevelOrder: Readonly> = { + debug: 0, + info: 1, + warn: 2, + error: 3, + fatal: 4, +}; + +const validLogLevels = new Set(['debug', 'info', 'warn', 'error', 'fatal', 'off']); +const validAccessStatusClasses = new Set(['2xx', '3xx', '4xx', '5xx']); +const defaultAccessBodyMaxBytes = 16 * 1024; +const maxAccessBodyBytes = 128 * 1024; + +function validateLogLevel(value: unknown, path: string): LogLevelSetting | undefined { + if (typeof value === 'undefined') return undefined; + if (typeof value !== 'string' || !validLogLevels.has(value as LogLevelSetting)) { + throw new Error(`${path} must be one of debug, info, warn, error, fatal, or off`); + } + return value as LogLevelSetting; +} + +function validateDomainName(domain: string): void { + if (domain.length === 0 || domain.trim() !== domain || domain.split('.').some(segment => segment.length === 0)) { + throw new Error(`logging.domains contains an invalid domain name: ${JSON.stringify(domain)}`); + } +} + +/** Access logを明示的に有効化していない場合の設定を作成します。 */ +function createDisabledAccessLogConfiguration(): ResolvedAccessLogConfiguration { + return { + statusClasses: [], + bodies: { + request: false, + response: false, + maxBytes: defaultAccessBodyMaxBytes, + }, + }; +} + +/** Access log設定を検証し、本番環境では本文だけを無効化します。 */ +function resolveAccessConfiguration(configuration: unknown, nodeEnv: string | undefined): { + readonly access: ResolvedAccessLogConfiguration; + readonly warnings: readonly string[]; +} { + if (configuration == null) return { access: createDisabledAccessLogConfiguration(), warnings: [] }; + if (typeof configuration !== 'object' || Array.isArray(configuration)) { + throw new Error('logging.access must be an object'); + } + + const raw = configuration as { + statusClasses?: unknown; + bodies?: unknown; + }; + let statusClasses: AccessLogStatusClass[] = []; + if (typeof raw.statusClasses !== 'undefined') { + if (!Array.isArray(raw.statusClasses)) { + throw new Error('logging.access.statusClasses must be an array'); + } + statusClasses = [...new Set(raw.statusClasses.map((statusClass, index) => { + if (typeof statusClass !== 'string' || !validAccessStatusClasses.has(statusClass as AccessLogStatusClass)) { + throw new Error(`logging.access.statusClasses[${index}] must be one of 2xx, 3xx, 4xx, or 5xx`); + } + return statusClass as AccessLogStatusClass; + }))]; + } + + let request = false; + let response = false; + let maxBytes = defaultAccessBodyMaxBytes; + if (typeof raw.bodies !== 'undefined') { + if (typeof raw.bodies !== 'object' || raw.bodies === null || Array.isArray(raw.bodies)) { + throw new Error('logging.access.bodies must be an object'); + } + const bodies = raw.bodies as { request?: unknown; response?: unknown; maxBytes?: unknown }; + if (typeof bodies.request !== 'undefined' && typeof bodies.request !== 'boolean') { + throw new Error('logging.access.bodies.request must be a boolean'); + } + if (typeof bodies.response !== 'undefined' && typeof bodies.response !== 'boolean') { + throw new Error('logging.access.bodies.response must be a boolean'); + } + const configuredMaxBytes = bodies.maxBytes; + if (typeof configuredMaxBytes !== 'undefined' && (typeof configuredMaxBytes !== 'number' || !Number.isSafeInteger(configuredMaxBytes) || configuredMaxBytes <= 0 || configuredMaxBytes > maxAccessBodyBytes)) { + throw new Error(`logging.access.bodies.maxBytes must be a positive integer no greater than ${maxAccessBodyBytes}`); + } + request = bodies.request ?? false; + response = bodies.response ?? false; + maxBytes = typeof configuredMaxBytes === 'number' ? configuredMaxBytes : defaultAccessBodyMaxBytes; + } + + if (nodeEnv === 'production' && (request || response)) { + return { + access: { + statusClasses, + bodies: { request: false, response: false, maxBytes }, + }, + warnings: ['logging.access.bodies is disabled in production mode'], + }; + } + + return { + access: { statusClasses, bodies: { request, response, maxBytes } }, + warnings: [], + }; +} + +/** 通常ログとAccess logの設定をまとめて検証し、起動時警告も返します。 */ +function resolveConfiguration(configuration: LogManagerConfiguration | undefined, nodeEnv: string | undefined): { + readonly level: LogLevelSetting | undefined; + readonly domains: readonly (readonly [string, LogLevelSetting])[]; + readonly access: ResolvedAccessLogConfiguration; + readonly warnings: readonly string[]; +} { + if (configuration == null) return { level: undefined, domains: [], access: createDisabledAccessLogConfiguration(), warnings: [] }; + + const level = validateLogLevel(configuration.level, 'logging.level'); + const access = resolveAccessConfiguration(configuration.access, nodeEnv); + if (configuration.domains == null) return { level, domains: [], ...access }; + if (typeof configuration.domains !== 'object' || configuration.domains === null || Array.isArray(configuration.domains)) { + throw new Error('logging.domains must be an object'); + } + + const domains = Object.entries(configuration.domains).map(([domain, value]) => { + validateDomainName(domain); + const level = validateLogLevel(value, `logging.domains.${domain}`); + if (typeof level === 'undefined') { + throw new Error(`logging.domains.${domain} must be configured`); + } + return [domain, level] as const; + }).sort((left, right) => right[0].length - left[0].length); + + return { level, domains, ...access }; +} + +const defaultDependencies: LogManagerDependencies = { + now: () => new Date(), + getProcessInfo: () => ({ + processId: process.pid, + isPrimary: cluster.isPrimary, + workerId: cluster.isPrimary ? null : (cluster.worker?.id ?? null), + }), + isQuiet: () => envOption.quiet, + isVerbose: () => envOption.verbose, + getNodeEnv: () => process.env.NODE_ENV, +}; + +/** + * ログの出力可否を判断し、すべての出力先で共通となる情報を付加します。 + * Loggerと出力先の間に置くことで、設定や共通情報の扱いを一か所へ集約します。 + */ +export class LogManager { + private backend: LogBackend; + private readonly dependencies: LogManagerDependencies; + private normalizationProfile: LogNormalizationProfile; + private traceContextProvider: LogTraceContextProvider | undefined; + private configuredLevel: LogLevelSetting | undefined; + private configuredDomains: readonly (readonly [string, LogLevelSetting])[]; + private accessConfiguration: ResolvedAccessLogConfiguration; + private shutdownPromise: Promise | undefined; + + /** + * 出力先と実行環境から値を取得する処理を受け取ります。 + * 実行環境の取得処理は、必要な項目だけテスト用に差し替えられます。 + */ + constructor( + backend: LogBackend, + dependencies: Partial = {}, + options: LogManagerOptions = {}, + ) { + this.backend = backend; + this.dependencies = { + ...defaultDependencies, + ...dependencies, + }; + this.normalizationProfile = options.normalizationProfile ?? 'standard'; + this.traceContextProvider = undefined; + this.configuredLevel = undefined; + this.configuredDomains = []; + this.accessConfiguration = createDisabledAccessLogConfiguration(); + } + + /** + * 以後のログを書き込む出力先を切り替えます。 + * 作成済みのLoggerにも切り替えを反映するため、LogManager側で保持します。 + */ + public setBackend(backend: LogBackend): void { + this.backend = backend; + } + + /** 起動時の既定levelとdomain別levelを適用します。 */ + public configure(configuration?: LogManagerConfiguration): readonly string[] { + const resolved = resolveConfiguration(configuration, this.dependencies.getNodeEnv()); + this.configuredLevel = resolved.level; + this.configuredDomains = resolved.domains; + this.accessConfiguration = resolved.access; + return resolved.warnings; + } + + /** Fastifyフックが参照する正規化済みのAccess log設定を返します。 */ + public getAccessLogConfiguration(): ResolvedAccessLogConfiguration { + return this.accessConfiguration; + } + + /** 正規化方式を切り替え、既に作成済みのLoggerにも反映します。 */ + public setNormalizationProfile(profile: LogNormalizationProfile): void { + this.normalizationProfile = profile; + } + + /** ログ出力時にactiveなTrace Contextを取得する処理を登録します。 */ + public setTraceContextProvider(provider?: LogTraceContextProvider): void { + this.traceContextProvider = provider; + } + + /** 現在の処理に紐付くTrace Contextを取得します。 */ + public getActiveTraceContext(): LogTraceContext | undefined { + return this.traceContextProvider?.(); + } + + /** backendに残っているログをflushしてから終了処理を行います。 */ + public shutdown(): Promise { + if (this.shutdownPromise != null) return this.shutdownPromise; + + this.shutdownPromise = (async () => { + try { + await this.backend.flush?.(); + } finally { + await this.backend.close?.(); + } + })(); + + return this.shutdownPromise; + } + + private getDefaultLevel(): LogLevel { + if (this.dependencies.isVerbose()) return 'debug'; + return this.dependencies.getNodeEnv() === 'production' ? 'info' : 'debug'; + } + + private getThreshold(loggerName: string): LogLevelSetting { + let threshold: LogLevelSetting | undefined; + for (const [domain, level] of this.configuredDomains) { + if (loggerName === domain || loggerName.startsWith(`${domain}.`)) { + threshold = level; + break; + } + } + + threshold ??= this.configuredLevel ?? this.getDefaultLevel(); + + // verboseは障害調査用の緊急モードとして、明示されたoff以外をdebugまで下げる。 + // offは意図的な無効化なので、verboseでも再有効化しない。 + return threshold === 'off' || !this.dependencies.isVerbose() ? threshold : 'debug'; + } + + private shouldWrite(input: LogRecordInput, loggerName: string): boolean { + const threshold = this.getThreshold(loggerName); + if (threshold === 'off') return false; + return logLevelOrder[input.level] >= logLevelOrder[threshold]; + } + + /** + * 出力条件を確認し、共通情報を付加して出力先へ渡します。 + */ + public write(input: LogRecordInput): void { + // `quiet`は他の条件より優先し、ログに付随する情報の取得も行いません。 + if (this.dependencies.isQuiet()) return; + + const loggerName = input.context.map(segment => segment.name).join('.'); + if (!this.shouldWrite(input, loggerName)) return; + + const processInfo = this.dependencies.getProcessInfo(); + // 呼び出し側の配列を共有せず、親から末端までの順序を固定します。 + const context = [...input.context]; + // 出力を実際に行う直前にだけ正規化し、捨てられるdebugログのコストを抑えます。 + const { attributes, error: inputError, ...inputWithoutStructuredValues } = input; + const normalizedAttributes = typeof attributes !== 'undefined' + ? normalizeLogAttributes(attributes, { profile: this.normalizationProfile }) + : undefined; + const error = inputError ?? findLegacyLogError(input.compatibility?.data); + const normalizedError = typeof error !== 'undefined' + ? serializeLogError(error, { profile: this.normalizationProfile }) + : undefined; + // 実際に出力するログだけ、TelemetryからactiveなTrace Contextを取得します。 + const traceContext = this.traceContextProvider?.(); + const record = { + ...inputWithoutStructuredValues, + context, + timestamp: this.dependencies.now().toISOString(), + loggerName, + processId: processInfo.processId, + isPrimary: processInfo.isPrimary, + workerId: processInfo.workerId, + ...(traceContext ?? {}), + ...(normalizedAttributes ? { attributes: normalizedAttributes } : {}), + ...(normalizedError ? { error: normalizedError } : {}), + } as LogRecord; + + this.backend.write(record); + } + + /** status classの設定を確認し、Access logの出力対象か判断します。 */ + public shouldWriteAccess(statusCode: number): boolean { + if (this.dependencies.isQuiet()) return false; + const statusClass = `${Math.floor(statusCode / 100)}xx` as AccessLogStatusClass; + return validAccessStatusClasses.has(statusClass) && this.accessConfiguration.statusClasses.includes(statusClass); + } + + /** Access logのフック自体を登録してよい状態か、quietを含めて判定します。 */ + public isAccessLogEnabled(): boolean { + return !this.dependencies.isQuiet() && this.accessConfiguration.statusClasses.length > 0; + } + + /** HTTP応答へ共通情報と本文の安全な正規化を加えてAccess logを渡します。 */ + public writeAccess(input: AccessLogRecordInput): void { + if (!this.shouldWriteAccess(input.statusCode)) return; + + const processInfo = this.dependencies.getProcessInfo(); + const { requestBody, responseBody, traceContext, ...inputWithoutOptionalValues } = input; + const bodyOptions = { + profile: this.normalizationProfile, + limits: { maxBytes: this.accessConfiguration.bodies.maxBytes }, + } as const; + const normalizedRequestBody = this.accessConfiguration.bodies.request && typeof requestBody !== 'undefined' + ? normalizeLogValue(requestBody, bodyOptions) + : undefined; + const normalizedResponseBody = this.accessConfiguration.bodies.response && typeof responseBody !== 'undefined' + ? normalizeLogValue(responseBody, bodyOptions) + : undefined; + // HTTPフックが開始時に保持したContextだけを使い、応答時に別のSpanを紐付けません。 + const resolvedTraceContext = traceContext; + const record: AccessLogRecord = { + type: 'access', + ...inputWithoutOptionalValues, + timestamp: this.dependencies.now().toISOString(), + processId: processInfo.processId, + isPrimary: processInfo.isPrimary, + workerId: processInfo.workerId, + ...(typeof normalizedRequestBody !== 'undefined' ? { requestBody: normalizedRequestBody } : {}), + ...(typeof normalizedResponseBody !== 'undefined' ? { responseBody: normalizedResponseBody } : {}), + ...(resolvedTraceContext ?? {}), + }; + + this.backend.writeAccess?.(record); + } +} diff --git a/packages/backend/src/logging/LogNormalizer.ts b/packages/backend/src/logging/LogNormalizer.ts new file mode 100644 index 0000000000..f020c3defc --- /dev/null +++ b/packages/backend/src/logging/LogNormalizer.ts @@ -0,0 +1,395 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { Buffer } from 'node:buffer'; +import type { LogAttributeValue, LogAttributes, SerializedError } from './types.js'; + +/** 正規化の粒度を表します。詳細指定でも秘匿処理は常に有効です。 */ +export type LogNormalizationProfile = 'standard' | 'detailed'; + +/** 正規化で使う上限値です。 */ +export type LogNormalizationLimits = { + readonly maxDepth: number; + readonly maxEntries: number; + readonly maxStringBytes: number; + readonly maxBytes: number; +}; + +/** 属性のキーを秘匿すべきか判定する関数です。 */ +export type LogRedactor = (path: readonly string[], key: string) => boolean; + +/** 正規化処理へ渡す設定です。 */ +export type LogNormalizationOptions = { + readonly profile?: LogNormalizationProfile; + readonly limits?: Partial; + readonly redactor?: LogRedactor; +}; + +/** 通常運用でログが肥大化しないようにした上限です。 */ +export const STANDARD_LOG_NORMALIZATION_LIMITS: LogNormalizationLimits = { + maxDepth: 6, + maxEntries: 100, + maxStringBytes: 8 * 1024, + maxBytes: 64 * 1024, +}; + +/** 障害調査時により多くの情報を残す上限です。 */ +export const DETAILED_LOG_NORMALIZATION_LIMITS: LogNormalizationLimits = { + maxDepth: 10, + maxEntries: 1000, + maxStringBytes: 64 * 1024, + maxBytes: 256 * 1024, +}; + +const REDACTED = '[REDACTED]'; +const CIRCULAR = '[Circular]'; +const TRUNCATED = '[Truncated]'; +const UNSUPPORTED = '[Unsupported]'; +const TRUNCATED_KEY = '[Truncated]'; + +const sensitiveKeyParts = [ + 'password', + 'passwd', + 'passphrase', + 'token', + 'secret', + 'authorization', + 'cookie', + 'apikey', + 'privatekey', + 'credential', + 'captcha', + 'hcaptcharesponse', + 'grecaptcharesponse', + 'turnstileresponse', + 'mcaptcharesponse', + 'testcaptcharesponse', +]; + +/** キー名を比較用に揃え、区切り文字による表記揺れを吸収します。 */ +function normalizeKey(key: string): string { + return key.toLowerCase().replace(/[-_.\s]/g, ''); +} + +/** 既定の秘匿対象を判定します。Misskey APIの`i`も認証情報として扱います。 */ +export function defaultLogRedactor(_path: readonly string[], key: string): boolean { + const normalized = normalizeKey(key); + return normalized === 'i' || sensitiveKeyParts.some(part => normalized.includes(part)); +} + +/** 選択した方式と個別指定を合わせて、実際の上限値を決めます。 */ +export function resolveLogNormalizationLimits(options: LogNormalizationOptions = {}): LogNormalizationLimits { + const base = options.profile === 'detailed' + ? DETAILED_LOG_NORMALIZATION_LIMITS + : STANDARD_LOG_NORMALIZATION_LIMITS; + const limits = { + ...base, + ...options.limits, + }; + return { + maxDepth: Math.max(0, limits.maxDepth), + maxEntries: Math.max(0, limits.maxEntries), + maxStringBytes: Math.max(1, limits.maxStringBytes), + maxBytes: Math.max(2, limits.maxBytes), + }; +} + +/** 値が通常のオブジェクトとして読めるか判定します。 */ +function isObject(value: unknown): value is Record { + return typeof value === 'object' && value !== null; +} + +/** 外部入力のキーを安全に格納するため、prototypeを持たない属性領域を作成します。 */ +function createAttributeMap(): Record { + return Object.create(null) as Record; +} + +/** 値を文字列化し、文字列化処理自体の例外もログ処理へ漏らさないようにします。 */ +function stringifySafely(value: unknown): string { + try { + return String(value); + } catch { + return UNSUPPORTED; + } +} + +/** UTF-8のバイト数を測ります。ログの上限を文字数ではなく出力サイズで揃えるために使います。 */ +function byteLength(value: string): number { + return Buffer.byteLength(value, 'utf8'); +} + +/** 文字列をUTF-8の上限内へ切り詰めます。 */ +function normalizeString(value: string, maxBytes: number): string { + if (byteLength(value) <= maxBytes) return value; + const suffix = `…${TRUNCATED}`; + if (byteLength(suffix) > maxBytes) { + const end = findMaxPrefixLength(value, '', maxBytes); + return value.slice(0, end); + } + const end = findMaxPrefixLength(value, suffix, maxBytes); + return value.slice(0, end) + suffix; +} + +/** 指定した後置文字列を含めて上限に収まる接頭辞の長さを二分探索します。 */ +function findMaxPrefixLength(value: string, suffix: string, maxBytes: number): number { + let lower = 0; + let upper = value.length; + while (lower < upper) { + const middle = Math.ceil((lower + upper) / 2); + if (byteLength(value.slice(0, middle) + suffix) <= maxBytes) { + lower = middle; + } else { + upper = middle - 1; + } + } + // UTF-16のサロゲート対を途中で切らないよう、必要なら1文字戻します。 + if (lower > 0 && lower < value.length) { + const code = value.charCodeAt(lower - 1); + if (code >= 0xd800 && code <= 0xdbff) lower--; + } + return lower; +} + +/** 特殊な値に対しても、エラー判定で例外を発生させないようにします。 */ +function isErrorValue(value: unknown): value is Error { + try { + return value instanceof Error; + } catch { + return false; + } +} + +/** 値の読み出しを安全に行い、壊れたログ属性が本処理を中断しないようにします。 */ +function readProperty(value: Record, key: string): unknown { + try { + return value[key]; + } catch { + return `${UNSUPPORTED}: property access failed`; + } +} + +/** 属性をJSONへ出力できる値へ変換します。 */ +function normalizeValue( + value: unknown, + path: readonly string[], + depth: number, + seen: WeakSet, + limits: LogNormalizationLimits, + redactor: LogRedactor, +): LogAttributeValue { + if (value === null) return null; + if (typeof value === 'string') return normalizeString(value, limits.maxStringBytes); + if (typeof value === 'boolean') return value; + if (typeof value === 'number') return Number.isFinite(value) ? value : stringifySafely(value); + if (typeof value === 'bigint') return value.toString(10); + if (typeof value === 'undefined') return `${UNSUPPORTED}: undefined`; + if (typeof value === 'function' || typeof value === 'symbol') return `${UNSUPPORTED}: ${typeof value}`; + let isArray = false; + try { + if (value instanceof Date) { + return normalizeString(value.toISOString(), limits.maxStringBytes); + } + isArray = Array.isArray(value); + if (!isArray) { + const prototype = Object.getPrototypeOf(value); + if (prototype !== Object.prototype && prototype !== null) return `${UNSUPPORTED}: object`; + } + } catch { + return `${UNSUPPORTED}: object access failed`; + } + if (seen.has(value)) return CIRCULAR; + if (depth >= limits.maxDepth) return TRUNCATED; + seen.add(value); + + try { + if (isArray) { + const arrayValue = value as readonly unknown[]; + const result: LogAttributeValue[] = []; + const entries = Math.min(arrayValue.length, limits.maxEntries); + for (let i = 0; i < entries; i++) { + result.push(normalizeValue(arrayValue[i], [...path, String(i)], depth + 1, seen, limits, redactor)); + } + if (arrayValue.length > entries) result.push(TRUNCATED); + return result; + } + + // 攻撃者が指定した`__proto__`を通常の属性として保持するため、null prototypeを使います。 + const result = createAttributeMap(); + const keys = Object.keys(value).sort(); + const entries = Math.min(keys.length, limits.maxEntries); + for (let i = 0; i < entries; i++) { + const key = keys[i]; + // 秘匿対象は値を読み出す前に置き換え、読み出し処理や巨大な値にも触れません。 + result[key] = redactor(path, key) + ? REDACTED + : normalizeValue(readProperty(value as Record, key), [...path, key], depth + 1, seen, limits, redactor); + } + if (keys.length > entries) result[TRUNCATED_KEY] = TRUNCATED; + return result; + } catch { + return `${UNSUPPORTED}: object access failed`; + } finally { + seen.delete(value); + } +} + +/** JSON文字列化した値のバイト数を測ります。正規化後の最終上限に使います。 */ +function serializedByteLength(value: LogAttributeValue): number { + return byteLength(JSON.stringify(value)); +} + +/** 正規化済みの値を上限内へ再帰的に縮めます。 */ +function trimToByteLimit(value: LogAttributeValue, maxBytes: number): LogAttributeValue { + if (serializedByteLength(value) <= maxBytes) return value; + if (typeof value === 'string') return normalizeString(value, maxBytes); + if (Array.isArray(value)) { + const result: LogAttributeValue[] = []; + for (const item of value) { + const trimmedItem = trimToByteLimit(item, maxBytes); + const candidate = [...result, trimmedItem]; + if (serializedByteLength(candidate) > maxBytes) break; + result.push(trimmedItem); + } + if (result.length < value.length && serializedByteLength([...result, TRUNCATED]) <= maxBytes) result.push(TRUNCATED); + return result; + } + if (isObject(value)) { + // 上限調整中も`__proto__`を安全に属性として扱えるようにします。 + const result = createAttributeMap(); + for (const key of Object.keys(value).sort()) { + const candidate = Object.assign(createAttributeMap(), result, { [key]: trimToByteLimit(value[key], maxBytes) }); + if (serializedByteLength(candidate) > maxBytes) break; + result[key] = candidate[key]; + } + if (Object.keys(result).length < Object.keys(value).length) { + const candidate = Object.assign(createAttributeMap(), result, { [TRUNCATED_KEY]: TRUNCATED }); + if (serializedByteLength(candidate) <= maxBytes) return candidate; + } + return result; + } + return TRUNCATED; +} + +/** エラーらしい原因情報かを安全に判定します。特殊な値もログ処理を壊しません。 */ +function isErrorLike(value: unknown): value is Record { + if (!isObject(value)) return false; + try { + return ['name', 'message', 'stack', 'cause'].some(key => key in value); + } catch { + return false; + } +} + +/** 属性をJSONへ出力できるオブジェクトへ正規化します。 */ +export function normalizeLogAttributes(value: unknown, options: LogNormalizationOptions = {}): LogAttributes { + const limits = resolveLogNormalizationLimits(options); + const redactor = options.redactor ?? defaultLogRedactor; + const normalized = normalizeValue(value, [], 0, new WeakSet(), limits, redactor); + const root = isObject(normalized) && !Array.isArray(normalized) ? normalized : { value: normalized }; + return trimToByteLimit(root as LogAttributes, limits.maxBytes) as LogAttributes; +} + +/** 本文など単独の値を、既存ログと同じ秘匿・切り詰め規則で正規化します。 */ +export function normalizeLogValue(value: unknown, options: LogNormalizationOptions = {}): LogAttributeValue { + const limits = resolveLogNormalizationLimits(options); + const redactor = options.redactor ?? defaultLogRedactor; + const normalized = normalizeValue(value, [], 0, new WeakSet(), limits, redactor); + return trimToByteLimit(normalized, limits.maxBytes); +} + +/** 旧APIのdata領域から、エラー本体を見つけて構造化した形へ渡します。 */ +export function findLegacyLogError(value: unknown): unknown { + if (isErrorValue(value)) return value; + if (!isObject(value)) return undefined; + for (const key of ['e', 'err', 'error', 'stack']) { + const candidate = readProperty(value, key); + if (isErrorValue(candidate)) return candidate; + } + return undefined; +} + +/** エラーの原因情報を含めて、一定の形へ正規化します。 */ +function serializeErrorValue( + value: unknown, + depth: number, + seen: WeakSet, + limits: LogNormalizationLimits, + redactor: LogRedactor, +): SerializedError { + if (!isObject(value)) { + return { + type: value === null ? 'null' : typeof value, + message: normalizeString(stringifySafely(value), limits.maxStringBytes), + }; + } + if (seen.has(value)) return { type: 'CircularError', message: CIRCULAR }; + seen.add(value); + + try { + const name = readProperty(value, 'name'); + const message = readProperty(value, 'message'); + const stack = readProperty(value, 'stack'); + const cause = readProperty(value, 'cause'); + const constructor = readProperty(value, 'constructor'); + const constructorName = typeof constructor === 'function' || isObject(constructor) + ? readProperty(constructor as Record, 'name') + : undefined; + const type = normalizeString( + typeof name === 'string' && name.length > 0 + ? name + : typeof constructorName === 'string' && constructorName.length > 0 ? constructorName : 'Error', + limits.maxStringBytes, + ); + const result: { type: string; message: string; stack?: string; cause?: SerializedError | LogAttributeValue } = { + type, + message: normalizeString(typeof message === 'string' ? message : stringifySafely(value), limits.maxStringBytes), + }; + if (typeof stack === 'string') result.stack = normalizeString(stack, limits.maxStringBytes); + if (typeof cause !== 'undefined') { + result.cause = depth < limits.maxDepth + ? isErrorValue(cause) || isErrorLike(cause) + ? serializeErrorValue(cause, depth + 1, seen, limits, redactor) + : normalizeValue(cause, ['cause'], depth + 1, seen, limits, redactor) + : TRUNCATED; + } + return result; + } finally { + seen.delete(value); + } +} + +/** エラーの必須項目を残したまま、全体の出力サイズを上限内へ縮めます。 */ +function trimSerializedError(value: SerializedError, maxBytes: number): SerializedError { + const requiredLimit = Math.max(0, Math.floor((maxBytes - 24) / 2)); + const result: { type: string; message: string; stack?: string; cause?: SerializedError | LogAttributeValue } = { + type: normalizeString(value.type, requiredLimit), + message: normalizeString(value.message, requiredLimit), + }; + if (serializedByteLength(result as unknown as LogAttributeValue) > maxBytes) { + return { type: '', message: '' }; + } + if (value.stack != null) { + const candidate = { ...result, stack: value.stack }; + if (serializedByteLength(candidate as unknown as LogAttributeValue) <= maxBytes) result.stack = value.stack; + } + if (value.cause != null) { + const cause = typeof value.cause === 'object' + ? trimToByteLimit(value.cause as LogAttributeValue, maxBytes) + : value.cause; + const candidate = { ...result, cause }; + if (serializedByteLength(candidate as unknown as LogAttributeValue) <= maxBytes) result.cause = cause as SerializedError | LogAttributeValue; + } + return result; +} + +/** 任意のエラー入力を、ログへ安全に埋め込める形へ変換します。 */ +export function serializeLogError(value: unknown, options: LogNormalizationOptions = {}): SerializedError | undefined { + if (typeof value === 'undefined' || value === null) return undefined; + const limits = resolveLogNormalizationLimits(options); + // 必須項目を含む最小のJSON形すら収まらない場合は、上限を超えないよう出力しません。 + if (limits.maxBytes < 24) return undefined; + const serialized = serializeErrorValue(value, 0, new WeakSet(), limits, options.redactor ?? defaultLogRedactor); + return trimSerializedError(serialized, limits.maxBytes); +} diff --git a/packages/backend/src/logging/PrettyConsoleBackend.ts b/packages/backend/src/logging/PrettyConsoleBackend.ts new file mode 100644 index 0000000000..779a6192a1 --- /dev/null +++ b/packages/backend/src/logging/PrettyConsoleBackend.ts @@ -0,0 +1,125 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import chalk from 'chalk'; +import { default as convertColor } from 'color-convert'; +import { format as dateFormat } from 'date-fns'; +import { envOption } from '@/env.js'; +import type { LogBackend } from './LogBackend.js'; +import type { AccessLogRecord, LogAttributeValue, LogRecord } from './types.js'; + +/** 見やすい形式の出力処理が外部から受け取る依存関係です。 */ +export type PrettyConsoleBackendDependencies = { + readonly output: (...args: unknown[]) => void; + readonly withLogTime: () => boolean; +}; + +const defaultDependencies: PrettyConsoleBackendDependencies = { + output: (...args) => console.log(...args), + withLogTime: () => envOption.withLogTime, +}; + +/** Pretty形式でだけnull prototypeを通常のオブジェクトへ変換し、正規化済みの値自体は変更しません。 */ +function toPrettyLogValue(value: LogAttributeValue): LogAttributeValue { + if (Array.isArray(value)) return value.map(item => toPrettyLogValue(item)); + if (value !== null && typeof value === 'object') { + return Object.fromEntries(Object.entries(value).map(([key, child]) => [key, toPrettyLogValue(child)])); + } + return value; +} + +/** + * 人が読みやすい従来形式へ整形し、コンソールへ出力します。 + * 色、ラベル、時刻などの見た目だけを担当し、出力可否はLogManagerへ任せます。 + */ +export class PrettyConsoleBackend implements LogBackend { + private readonly dependencies: PrettyConsoleBackendDependencies; + + /** + * 出力処理と時刻表示の判定処理を受け取ります。 + * 省略時は従来どおり、標準のコンソールと環境設定を使用します。 + */ + constructor(dependencies: Partial = {}) { + this.dependencies = { + ...defaultDependencies, + ...dependencies, + }; + } + + /** + * 共通のログを従来形式へ整形して一件出力します。 + */ + public write(record: LogRecord): void { + const legacyLevel = record.compatibility?.legacyLevel; + // `fatal`は重大なエラーとして扱い、従来の`important`と同じ強調表示にします。 + const important = record.level === 'fatal' || (record.compatibility?.important ?? false); + const presentationLevel = record.level === 'fatal' ? 'error' : (legacyLevel ?? record.level); + const label = + presentationLevel === 'error' ? important ? chalk.bgRed.white('ERR ') : chalk.red('ERR ') : + presentationLevel === 'warn' ? chalk.yellow('WARN') : + presentationLevel === 'success' ? important ? chalk.bgGreen.white('DONE') : chalk.green('DONE') : + presentationLevel === 'debug' ? chalk.gray('VERB') : + presentationLevel === 'info' ? chalk.blue('INFO') : + null; + const contexts = record.context.map(context => context.color + ? chalk.rgb(...convertColor.keyword.rgb(context.color))(context.name) + : chalk.white(context.name)); + const message = + presentationLevel === 'error' ? chalk.red(record.message) : + presentationLevel === 'warn' ? chalk.yellow(record.message) : + presentationLevel === 'success' ? chalk.green(record.message) : + presentationLevel === 'debug' ? chalk.gray(record.message) : + presentationLevel === 'info' ? record.message : + null; + // 主プロセスは従来どおり「*」、子プロセスはワーカー番号で識別します。 + const worker = record.isPrimary ? '*' : record.workerId; + + let log = `${label} ${worker}\t[${contexts.join(' ')}]\t${message}`; + if (this.dependencies.withLogTime()) { + log = chalk.gray(dateFormat(new Date(record.timestamp), 'HH:mm:ss')) + ' ' + log; + } + + // `data`は文字列へ埋め込まず、第2引数として渡す従来の挙動を維持します。 + const args: unknown[] = [important ? chalk.bold(log) : log]; + if (record.compatibility?.data != null) { + // 旧形式の値はそのまま第2引数へ渡し、既存の表示と調査方法を保ちます。 + args.push(record.compatibility.data); + } else if (record.eventName != null || record.attributes != null || record.error != null) { + // 構造化ログは、専用の出力先がなくても調査情報を確認できるようにします。 + args.push({ + ...(record.eventName != null ? { eventName: record.eventName } : {}), + ...(record.attributes != null ? { attributes: record.attributes } : {}), + ...(record.error != null ? { error: record.error } : {}), + }); + } + this.dependencies.output(...args); + } + + /** Access logを人が読みやすい1行へ整形し、本文は追加情報として表示します。 */ + public writeAccess(record: AccessLogRecord): void { + const statusLevel = record.statusCode >= 500 ? 'error' : record.statusCode >= 400 ? 'warn' : 'info'; + const label = + statusLevel === 'error' ? chalk.red('ERR ') : + statusLevel === 'warn' ? chalk.yellow('WARN') : + chalk.blue('INFO'); + const worker = record.isPrimary ? '*' : record.workerId; + const route = record.route ?? '-'; + const size = record.responseSizeBytes == null ? '-' : `${record.responseSizeBytes}B`; + const message = `${record.method} ${route} ${record.statusCode} ${record.durationMs.toFixed(2)}ms ${size}`; + let log = `${label} ${worker}\t[access]\t${message}`; + if (this.dependencies.withLogTime()) { + log = chalk.gray(dateFormat(new Date(record.timestamp), 'HH:mm:ss')) + ' ' + log; + } + + const details = { + ...(record.errorType != null ? { errorType: record.errorType } : {}), + ...(record.requestBody !== undefined ? { requestBody: toPrettyLogValue(record.requestBody) } : {}), + ...(record.responseBody !== undefined ? { responseBody: toPrettyLogValue(record.responseBody) } : {}), + }; + const args: unknown[] = [log]; + if (Object.keys(details).length > 0) args.push(details); + this.dependencies.output(...args); + } +} diff --git a/packages/backend/src/logging/logging-runtime.ts b/packages/backend/src/logging/logging-runtime.ts new file mode 100644 index 0000000000..223f6e379e --- /dev/null +++ b/packages/backend/src/logging/logging-runtime.ts @@ -0,0 +1,51 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { LogManager } from './LogManager.js'; +import { BootstrapConsoleBackend } from './BootstrapConsoleBackend.js'; +import { JsonConsoleBackend } from './JsonConsoleBackend.js'; +import { PrettyConsoleBackend } from './PrettyConsoleBackend.js'; +import type { LogManagerConfiguration } from './LogManager.js'; +import type { LogTraceContextProvider, LogFormat } from './types.js'; +import type { LogBackend } from './LogBackend.js'; + +/** + * プロセス内のすべてのLoggerが共有するLogManagerです。 + * Logger作成後も同じLogManagerを参照するため、出力先の切り替えを一括で反映できます。 + */ +export const logManager = new LogManager(new BootstrapConsoleBackend()); + +/** ログ形式を検証し、指定された形式に対応する出力処理を作成します。 */ +function createLoggingBackend(format: unknown): LogBackend { + if (format == null || format === 'pretty') return new PrettyConsoleBackend(); + if (format === 'json') return new JsonConsoleBackend(); + throw new Error('logging.format must be either pretty or json'); +} + +/** 起動時のログ設定を適用し、選択した出力処理へ切り替えます。 */ +export function configureLogging(configuration?: LogManagerConfiguration & { readonly format?: LogFormat }): void { + // 出力処理を先に検証し、設定値が不正な場合は現在の出力処理を壊さないようにします。 + const backend = createLoggingBackend(configuration?.format); + const warnings = logManager.configure(configuration); + logManager.setBackend(backend); + // 設定を無視した理由を、選択済みの形式で起動時に一度だけ知らせます。 + for (const message of warnings) { + logManager.write({ + level: 'warn', + message, + context: [{ name: 'logging' }], + }); + } +} + +/** Telemetry初期化後に、ログへTrace Contextを付加する取得処理を登録します。 */ +export function setLogTraceContextProvider(provider?: LogTraceContextProvider): void { + logManager.setTraceContextProvider(provider); +} + +/** プロセス終了前に現在のログ出力処理を保留分まで書き出して閉じます。 */ +export function shutdownLogging(): Promise { + return logManager.shutdown(); +} diff --git a/packages/backend/src/logging/types.ts b/packages/backend/src/logging/types.ts new file mode 100644 index 0000000000..ddd10f410f --- /dev/null +++ b/packages/backend/src/logging/types.ts @@ -0,0 +1,146 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { Keyword } from 'color-convert'; + +/** ログの重要度を表します。 */ +export type LogLevel = 'debug' | 'info' | 'warn' | 'error' | 'fatal'; + +/** 設定で指定できるログの閾値です。`off`はログイベントのlevelには使用しません。 */ +export type LogLevelSetting = LogLevel | 'off'; + +/** コンソールへ出すログ形式です。未指定時は見やすい形式を使用します。 */ +export type LogFormat = 'pretty' | 'json'; + +/** Access logで対象にできるHTTP status classです。 */ +export type AccessLogStatusClass = '2xx' | '3xx' | '4xx' | '5xx'; + +/** Access logの本文採取設定です。 */ +export type AccessLogBodyConfiguration = { + readonly request?: boolean; + readonly response?: boolean; + readonly maxBytes?: number; +}; + +/** Access logの出力設定です。 */ +export type AccessLogConfiguration = { + readonly statusClasses?: readonly AccessLogStatusClass[]; + readonly bodies?: AccessLogBodyConfiguration; +}; + +/** 正規化後にログ属性として扱えるJSONの値です。 */ +export type LogAttributeValue = + | string + | number + | boolean + | null + | readonly LogAttributeValue[] + | { readonly [key: string]: LogAttributeValue }; + +/** 正規化後のログ属性です。 */ +export type LogAttributes = Readonly>; + +/** activeなSpanとログを関連付けるためのTrace Contextです。 */ +export type LogTraceContext = { + readonly traceId: string; + readonly spanId: string; + readonly traceFlags: number; +}; + +/** LogManagerが出力直前に呼び出すTrace Context取得処理です。 */ +export type LogTraceContextProvider = () => LogTraceContext | undefined; + +/** level別のLoggerメソッドが受け取る構造化ログの共通入力です。 */ +export type LogEntryInput = { + readonly message: string; + readonly eventName?: string; + readonly attributes?: Readonly>; + readonly error?: unknown; +}; + +/** 動的なlevelを含めてLogManagerへ渡す構造化ログの入力です。 */ +export type LogWriteInput = LogEntryInput & { + readonly level: LogLevel; +}; + +/** + * ロガー名を構成する一要素です。 + * 色は見やすい形式での表示だけに使い、ログの意味には影響させません。 + */ +export type LoggerContext = { + readonly name: string; + readonly color?: Keyword; +}; + +/** + * 従来のコンソール表示を維持するための情報です。 + * 構造化ログの項目と混同しないよう、互換用の領域へ分離しています。 + * `data`は従来表示を保つため正規化せず、秘匿が必要な値は構造化属性へ移します。 + */ +export type LogCompatibility = { + readonly legacyLevel?: 'success'; + readonly important?: boolean; + readonly data?: unknown; +}; + +/** + * 呼び出し側からLogManagerへ渡す、時刻などを付加する前のログです。 + */ +export type LogRecordInput = LogWriteInput & { + readonly context: readonly LoggerContext[]; + readonly compatibility?: LogCompatibility; +}; + +/** エラーをJSONへ出力するために正規化した形です。 */ +export type SerializedError = { + readonly type: string; + readonly message: string; + readonly stack?: string; + readonly cause?: SerializedError | LogAttributeValue; +}; + +/** + * 出力先へ渡すログです。 + * `compatibility.data`は見やすい形式だけが使う従来値で、構造化した出力先は属性とエラーを利用します。 + */ +export type LogRecord = Omit & { + readonly timestamp: string; + readonly loggerName: string; + readonly processId: number; + readonly isPrimary: boolean; + readonly workerId: number | null; + readonly traceId?: string; + readonly spanId?: string; + readonly traceFlags?: number; + readonly attributes?: LogAttributes; + readonly error?: SerializedError; +}; + +/** Access logとして記録するHTTP応答の入力です。 */ +export type AccessLogRecordInput = { + readonly method: string; + readonly route: string | null; + readonly statusCode: number; + readonly durationMs: number; + readonly responseSizeBytes: number | null; + readonly errorType?: string; + readonly requestBody?: unknown; + readonly responseBody?: unknown; + readonly traceContext?: LogTraceContext; +}; + +/** 出力先へ渡すAccess logです。本文は正規化後の値だけを含みます。 */ +export type AccessLogRecord = Omit & { + readonly type: 'access'; + readonly timestamp: string; + readonly processId: number; + readonly isPrimary: boolean; + readonly workerId: number | null; + readonly requestBody?: LogAttributeValue; + readonly responseBody?: LogAttributeValue; + readonly traceId?: string; + readonly spanId?: string; + readonly traceFlags?: number; +}; diff --git a/packages/backend/src/misc/cache.ts b/packages/backend/src/misc/cache.ts index f1eae582a8..936b049f00 100644 --- a/packages/backend/src/misc/cache.ts +++ b/packages/backend/src/misc/cache.ts @@ -204,14 +204,13 @@ export class RedisSingleCache { } } -// TODO: メモリ節約のためあまり参照されないキーを定期的に削除できるようにする？ - export class MemoryKVCache { private readonly cache = new Map(); private readonly gcIntervalHandle = setInterval(() => this.gc(), 1000 * 60 * 3); // 3m constructor( private readonly lifetime: number, + private readonly limit: number = Infinity, ) {} @bindThis @@ -220,6 +219,25 @@ export class MemoryKVCache { * @deprecated これを直接呼び出すべきではない。InternalEventなどで変更を全てのプロセス/マシンに通知するべき */ public set(key: string, value: T): void { + if (this.limit <= 0) { + throw new Error('Limit must be greater than 0'); + } + + if (this.limit !== Infinity) { + this.gc(); + + // 挿入順を更新して LRU を保つため、同一キーは一度削除する + this.cache.delete(key); + + while (this.cache.size >= this.limit) { + const oldestKey = this.cache.keys().next().value; + if (oldestKey == null) { + throw new Error('Cache is empty but size exceeds the limit'); + } + this.cache.delete(oldestKey); + } + } + this.cache.set(key, { date: Date.now(), value, @@ -234,6 +252,11 @@ export class MemoryKVCache { this.cache.delete(key); return undefined; } + if (this.limit !== Infinity) { + // access 順を更新して LRU を保つ + this.cache.delete(key); + this.cache.set(key, cached); + } return cached.value; } diff --git a/packages/backend/src/models/Hashtag.ts b/packages/backend/src/models/Hashtag.ts index 3add06d0c3..4301348957 100644 --- a/packages/backend/src/models/Hashtag.ts +++ b/packages/backend/src/models/Hashtag.ts @@ -20,6 +20,7 @@ export class MiHashtag { @Column({ ...id(), + default: [], array: true, }) public mentionedUserIds: MiUser['id'][]; @@ -32,6 +33,7 @@ export class MiHashtag { @Column({ ...id(), + default: [], array: true, }) public mentionedLocalUserIds: MiUser['id'][]; @@ -44,6 +46,7 @@ export class MiHashtag { @Column({ ...id(), + default: [], array: true, }) public mentionedRemoteUserIds: MiUser['id'][]; @@ -56,6 +59,7 @@ export class MiHashtag { @Column({ ...id(), + default: [], array: true, }) public attachedUserIds: MiUser['id'][]; @@ -68,6 +72,7 @@ export class MiHashtag { @Column({ ...id(), + default: [], array: true, }) public attachedLocalUserIds: MiUser['id'][]; @@ -80,6 +85,7 @@ export class MiHashtag { @Column({ ...id(), + default: [], array: true, }) public attachedRemoteUserIds: MiUser['id'][]; diff --git a/packages/backend/src/models/Meta.ts b/packages/backend/src/models/Meta.ts index d4214c0e5a..307b3f7a5d 100644 --- a/packages/backend/src/models/Meta.ts +++ b/packages/backend/src/models/Meta.ts @@ -672,6 +672,11 @@ export class MiMeta { }) public urlPreviewUserAgent: string | null; + @Column('varchar', { + length: 3072, array: true, default: '{}', + }) + public urlPreviewSensitiveList: string[]; + @Column('varchar', { length: 128, default: 'none', diff --git a/packages/backend/src/queue/QueueProcessorService.ts b/packages/backend/src/queue/QueueProcessorService.ts index 2b3b3fc0ad..631fc64e6e 100644 --- a/packages/backend/src/queue/QueueProcessorService.ts +++ b/packages/backend/src/queue/QueueProcessorService.ts @@ -9,7 +9,9 @@ import type { Config } from '@/config.js'; import { DI } from '@/di-symbols.js'; import type Logger from '@/logger.js'; import { bindThis } from '@/decorators.js'; +import { TelemetryService } from '@/core/telemetry/TelemetryService.js'; import { CheckModeratorsActivityProcessorService } from '@/queue/processors/CheckModeratorsActivityProcessorService.js'; +import { runQueueJobWithTraceContext } from './queue-job-runner.js'; import { UserWebhookDeliverProcessorService } from './processors/UserWebhookDeliverProcessorService.js'; import { SystemWebhookDeliverProcessorService } from './processors/SystemWebhookDeliverProcessorService.js'; import { EndedPollNotificationProcessorService } from './processors/EndedPollNotificationProcessorService.js'; @@ -92,6 +94,7 @@ export class QueueProcessorService implements OnApplicationShutdown { private config: Config, private queueLoggerService: QueueLoggerService, + private telemetryService: TelemetryService, private userWebhookDeliverProcessorService: UserWebhookDeliverProcessorService, private systemWebhookDeliverProcessorService: SystemWebhookDeliverProcessorService, private endedPollNotificationProcessorService: EndedPollNotificationProcessorService, @@ -156,13 +159,8 @@ export class QueueProcessorService implements OnApplicationShutdown { }; } - let Sentry: typeof import('@sentry/node') | undefined; - if (this.config.sentryForBackend) { - import('@sentry/node').then((mod) => { - Sentry = mod; - }); - } - + // 以下の各 Worker は job.data に保存された enqueue 元の trace context を復元し、 + // ジョブの実処理全体を Link または parent の worker span で囲む。 //#region system { const processer = (job: Bull.Job) => { @@ -179,32 +177,30 @@ export class QueueProcessorService implements OnApplicationShutdown { default: throw new Error(`unrecognized job type ${job.name} for system`); } }; + const logger = this.logger.createSubLogger('system'); this.systemQueueWorker = new Bull.Worker(QUEUE.SYSTEM, (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: System: ' + job.name }, () => processer(job)); - } else { - return processer(job); - } + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: System: ' + job.name, + job.data, + () => processer(job) as Promise, + err => { + logger.error(`failed(${err.name}: ${err.message}) id=${job.id}`, { job: renderJob(job), e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: System: ${job.name}: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.SYSTEM), autorun: false, }); - const logger = this.logger.createSubLogger('system'); - this.systemQueueWorker .on('active', (job) => logger.debug(`active id=${job.id}`)) .on('completed', (job, result) => logger.debug(`completed(${result}) id=${job.id}`)) - .on('failed', (job, err: Error) => { - logger.error(`failed(${err.name}: ${err.message}) id=${job?.id ?? '?'}`, { job: renderJob(job), e: renderError(err) }); - if (Sentry != null) { - Sentry.captureMessage(`Queue: System: ${job?.name ?? '?'}: ${err.name}: ${err.message}`, { - level: 'error', - extra: { job, err }, - }); - } - }) .on('error', (err: Error) => logger.error(`error ${err.name}: ${err.message}`, { e: renderError(err) })) .on('stalled', (jobId) => logger.warn(`stalled id=${jobId}`)); } @@ -236,32 +232,30 @@ export class QueueProcessorService implements OnApplicationShutdown { default: throw new Error(`unrecognized job type ${job.name} for db`); } }; + const logger = this.logger.createSubLogger('db'); this.dbQueueWorker = new Bull.Worker(QUEUE.DB, (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: DB: ' + job.name }, () => processer(job)); - } else { - return processer(job); - } + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: DB: ' + job.name, + job.data, + () => processer(job), + err => { + logger.error(`failed(${err.name}: ${err.message}) id=${job.id}`, { job: renderJob(job), e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: DB: ${job.name}: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.DB), autorun: false, }); - const logger = this.logger.createSubLogger('db'); - this.dbQueueWorker .on('active', (job) => logger.debug(`active id=${job.id}`)) .on('completed', (job, result) => logger.debug(`completed(${result}) id=${job.id}`)) - .on('failed', (job, err) => { - logger.error(`failed(${err.name}: ${err.message}) id=${job?.id ?? '?'}`, { job: renderJob(job), e: renderError(err) }); - if (Sentry != null) { - Sentry.captureMessage(`Queue: DB: ${job?.name ?? '?'}: ${err.name}: ${err.message}`, { - level: 'error', - extra: { job, err }, - }); - } - }) .on('error', (err: Error) => logger.error(`error ${err.name}: ${err.message}`, { e: renderError(err) })) .on('stalled', (jobId) => logger.warn(`stalled id=${jobId}`)); } @@ -269,12 +263,22 @@ export class QueueProcessorService implements OnApplicationShutdown { //#region deliver { + const logger = this.logger.createSubLogger('deliver'); + this.deliverQueueWorker = new Bull.Worker(QUEUE.DELIVER, (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: Deliver' }, () => this.deliverProcessorService.process(job)); - } else { - return this.deliverProcessorService.process(job); - } + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: Deliver', + job.data, + () => this.deliverProcessorService.process(job), + err => { + logger.error(`failed(${err.name}: ${err.message}) ${getJobInfo(job)} to=${job.data.to}`, { e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: Deliver: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.DELIVER), autorun: false, @@ -288,20 +292,9 @@ export class QueueProcessorService implements OnApplicationShutdown { }, }); - const logger = this.logger.createSubLogger('deliver'); - this.deliverQueueWorker .on('active', (job) => logger.debug(`active ${getJobInfo(job, true)} to=${job.data.to}`)) .on('completed', (job, result) => logger.debug(`completed(${result}) ${getJobInfo(job, true)} to=${job.data.to}`)) - .on('failed', (job, err) => { - logger.error(`failed(${err.name}: ${err.message}) ${getJobInfo(job)} to=${job ? job.data.to : '-'}`); - if (Sentry != null) { - Sentry.captureMessage(`Queue: Deliver: ${err.name}: ${err.message}`, { - level: 'error', - extra: { job, err }, - }); - } - }) .on('error', (err: Error) => logger.error(`error ${err.name}: ${err.message}`, { e: renderError(err) })) .on('stalled', (jobId) => logger.warn(`stalled id=${jobId}`)); } @@ -309,12 +302,23 @@ export class QueueProcessorService implements OnApplicationShutdown { //#region inbox { + const logger = this.logger.createSubLogger('inbox'); + this.inboxQueueWorker = new Bull.Worker(QUEUE.INBOX, (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: Inbox' }, () => this.inboxProcessorService.process(job)); - } else { - return this.inboxProcessorService.process(job); - } + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: Inbox', + job.data, + () => this.inboxProcessorService.process(job), + err => { + const activityId = job.data.activity ? job.data.activity.id : 'none'; + logger.error(`failed(${err.name}: ${err.message}) ${getJobInfo(job)} activity=${activityId}`, { job: renderJob(job), e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: Inbox: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.INBOX), autorun: false, @@ -328,20 +332,9 @@ export class QueueProcessorService implements OnApplicationShutdown { }, }); - const logger = this.logger.createSubLogger('inbox'); - this.inboxQueueWorker .on('active', (job) => logger.debug(`active ${getJobInfo(job, true)}`)) .on('completed', (job, result) => logger.debug(`completed(${result}) ${getJobInfo(job, true)}`)) - .on('failed', (job, err) => { - logger.error(`failed(${err.name}: ${err.message}) ${getJobInfo(job)} activity=${job ? (job.data.activity ? job.data.activity.id : 'none') : '-'}`, { job: renderJob(job), e: renderError(err) }); - if (Sentry != null) { - Sentry.captureMessage(`Queue: Inbox: ${err.name}: ${err.message}`, { - level: 'error', - extra: { job, err }, - }); - } - }) .on('error', (err: Error) => logger.error(`error ${err.name}: ${err.message}`, { e: renderError(err) })) .on('stalled', (jobId) => logger.warn(`stalled id=${jobId}`)); } @@ -349,12 +342,22 @@ export class QueueProcessorService implements OnApplicationShutdown { //#region user-webhook deliver { + const logger = this.logger.createSubLogger('user-webhook'); + this.userWebhookDeliverQueueWorker = new Bull.Worker(QUEUE.USER_WEBHOOK_DELIVER, (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: UserWebhookDeliver' }, () => this.userWebhookDeliverProcessorService.process(job)); - } else { - return this.userWebhookDeliverProcessorService.process(job); - } + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: UserWebhookDeliver', + job.data, + () => this.userWebhookDeliverProcessorService.process(job), + err => { + logger.error(`failed(${err.name}: ${err.message}) ${getJobInfo(job)} to=${job.data.to}`, { e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: UserWebhookDeliver: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.USER_WEBHOOK_DELIVER), autorun: false, @@ -368,20 +371,9 @@ export class QueueProcessorService implements OnApplicationShutdown { }, }); - const logger = this.logger.createSubLogger('user-webhook'); - this.userWebhookDeliverQueueWorker .on('active', (job) => logger.debug(`active ${getJobInfo(job, true)} to=${job.data.to}`)) .on('completed', (job, result) => logger.debug(`completed(${result}) ${getJobInfo(job, true)} to=${job.data.to}`)) - .on('failed', (job, err) => { - logger.error(`failed(${err.name}: ${err.message}) ${getJobInfo(job)} to=${job ? job.data.to : '-'}`); - if (Sentry != null) { - Sentry.captureMessage(`Queue: UserWebhookDeliver: ${err.name}: ${err.message}`, { - level: 'error', - extra: { job, err }, - }); - } - }) .on('error', (err: Error) => logger.error(`error ${err.name}: ${err.message}`, { e: renderError(err) })) .on('stalled', (jobId) => logger.warn(`stalled id=${jobId}`)); } @@ -389,12 +381,22 @@ export class QueueProcessorService implements OnApplicationShutdown { //#region system-webhook deliver { + const logger = this.logger.createSubLogger('system-webhook'); + this.systemWebhookDeliverQueueWorker = new Bull.Worker(QUEUE.SYSTEM_WEBHOOK_DELIVER, (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: SystemWebhookDeliver' }, () => this.systemWebhookDeliverProcessorService.process(job)); - } else { - return this.systemWebhookDeliverProcessorService.process(job); - } + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: SystemWebhookDeliver', + job.data, + () => this.systemWebhookDeliverProcessorService.process(job), + err => { + logger.error(`failed(${err.name}: ${err.message}) ${getJobInfo(job)} to=${job.data.to}`, { e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: SystemWebhookDeliver: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.SYSTEM_WEBHOOK_DELIVER), autorun: false, @@ -408,20 +410,9 @@ export class QueueProcessorService implements OnApplicationShutdown { }, }); - const logger = this.logger.createSubLogger('system-webhook'); - this.systemWebhookDeliverQueueWorker .on('active', (job) => logger.debug(`active ${getJobInfo(job, true)} to=${job.data.to}`)) .on('completed', (job, result) => logger.debug(`completed(${result}) ${getJobInfo(job, true)} to=${job.data.to}`)) - .on('failed', (job, err) => { - logger.error(`failed(${err.name}: ${err.message}) ${getJobInfo(job)} to=${job ? job.data.to : '-'}`); - if (Sentry != null) { - Sentry.captureMessage(`Queue: SystemWebhookDeliver: ${err.name}: ${err.message}`, { - level: 'error', - extra: { job, err }, - }); - } - }) .on('error', (err: Error) => logger.error(`error ${err.name}: ${err.message}`, { e: renderError(err) })) .on('stalled', (jobId) => logger.warn(`stalled id=${jobId}`)); } @@ -438,13 +429,22 @@ export class QueueProcessorService implements OnApplicationShutdown { default: throw new Error(`unrecognized job type ${job.name} for relationship`); } }; + const logger = this.logger.createSubLogger('relationship'); this.relationshipQueueWorker = new Bull.Worker(QUEUE.RELATIONSHIP, (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: Relationship: ' + job.name }, () => processer(job)); - } else { - return processer(job); - } + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: Relationship: ' + job.name, + job.data, + () => processer(job), + err => { + logger.error(`failed(${err.name}: ${err.message}) id=${job.id}`, { job: renderJob(job), e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: Relationship: ${job.name}: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.RELATIONSHIP), autorun: false, @@ -455,20 +455,9 @@ export class QueueProcessorService implements OnApplicationShutdown { }, }); - const logger = this.logger.createSubLogger('relationship'); - this.relationshipQueueWorker .on('active', (job) => logger.debug(`active id=${job.id}`)) .on('completed', (job, result) => logger.debug(`completed(${result}) id=${job.id}`)) - .on('failed', (job, err) => { - logger.error(`failed(${err.name}: ${err.message}) id=${job?.id ?? '?'}`, { job: renderJob(job), e: renderError(err) }); - if (Sentry != null) { - Sentry.captureMessage(`Queue: Relationship: ${job?.name ?? '?'}: ${err.name}: ${err.message}`, { - level: 'error', - extra: { job, err }, - }); - } - }) .on('error', (err: Error) => logger.error(`error ${err.name}: ${err.message}`, { e: renderError(err) })) .on('stalled', (jobId) => logger.warn(`stalled id=${jobId}`)); } @@ -483,33 +472,31 @@ export class QueueProcessorService implements OnApplicationShutdown { default: throw new Error(`unrecognized job type ${job.name} for objectStorage`); } }; + const logger = this.logger.createSubLogger('objectStorage'); this.objectStorageQueueWorker = new Bull.Worker(QUEUE.OBJECT_STORAGE, (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: ObjectStorage: ' + job.name }, () => processer(job)); - } else { - return processer(job); - } + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: ObjectStorage: ' + job.name, + job.data, + () => processer(job) as Promise, + err => { + logger.error(`failed(${err.name}: ${err.message}) id=${job.id}`, { job: renderJob(job), e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: ObjectStorage: ${job.name}: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.OBJECT_STORAGE), autorun: false, concurrency: 16, }); - const logger = this.logger.createSubLogger('objectStorage'); - this.objectStorageQueueWorker .on('active', (job) => logger.debug(`active id=${job.id}`)) .on('completed', (job, result) => logger.debug(`completed(${result}) id=${job.id}`)) - .on('failed', (job, err) => { - logger.error(`failed(${err.name}: ${err.message}) id=${job?.id ?? '?'}`, { job: renderJob(job), e: renderError(err) }); - if (Sentry != null) { - Sentry.captureMessage(`Queue: ObjectStorage: ${job?.name ?? '?'}: ${err.name}: ${err.message}`, { - level: 'error', - extra: { job, err }, - }); - } - }) .on('error', (err: Error) => logger.error(`error ${err.name}: ${err.message}`, { e: renderError(err) })) .on('stalled', (jobId) => logger.warn(`stalled id=${jobId}`)); } @@ -517,12 +504,22 @@ export class QueueProcessorService implements OnApplicationShutdown { //#region ended poll notification { + const logger = this.logger.createSubLogger('ended-poll-notification'); + this.endedPollNotificationQueueWorker = new Bull.Worker(QUEUE.ENDED_POLL_NOTIFICATION, (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: EndedPollNotification' }, () => this.endedPollNotificationProcessorService.process(job)); - } else { - return this.endedPollNotificationProcessorService.process(job); - } + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: EndedPollNotification', + job.data, + () => this.endedPollNotificationProcessorService.process(job), + err => { + logger.error(`failed(${err.name}: ${err.message}) id=${job.id}`, { job: renderJob(job), e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: EndedPollNotification: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.ENDED_POLL_NOTIFICATION), autorun: false, @@ -532,12 +529,22 @@ export class QueueProcessorService implements OnApplicationShutdown { //#region post scheduled note { - this.postScheduledNoteQueueWorker = new Bull.Worker(QUEUE.POST_SCHEDULED_NOTE, async (job) => { - if (Sentry != null) { - return Sentry.startSpan({ name: 'Queue: PostScheduledNote' }, () => this.postScheduledNoteProcessorService.process(job)); - } else { - return this.postScheduledNoteProcessorService.process(job); - } + const logger = this.logger.createSubLogger('post-scheduled-note'); + + this.postScheduledNoteQueueWorker = new Bull.Worker(QUEUE.POST_SCHEDULED_NOTE, (job) => { + return runQueueJobWithTraceContext( + this.telemetryService, + 'Queue: PostScheduledNote', + job.data, + () => this.postScheduledNoteProcessorService.process(job), + err => { + logger.error(`failed(${err.name}: ${err.message}) id=${job.id}`, { job: renderJob(job), e: renderError(err) }); + this.telemetryService.captureMessage(`Queue: PostScheduledNote: ${err.name}: ${err.message}`, { + level: 'error', + extra: { job, err }, + }); + }, + ); }, { ...baseWorkerOptions(this.config, QUEUE.POST_SCHEDULED_NOTE), autorun: false, diff --git a/packages/backend/src/queue/queue-job-runner.ts b/packages/backend/src/queue/queue-job-runner.ts new file mode 100644 index 0000000000..aa1d29c512 --- /dev/null +++ b/packages/backend/src/queue/queue-job-runner.ts @@ -0,0 +1,32 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { TelemetryService } from '@/core/telemetry/TelemetryService.js'; + +type QueueTelemetryService = Pick; + +/** QueueのprocessorをTrace Context付きで実行し、失敗処理をSpan内で行います。 */ +export function runQueueJobWithTraceContext( + telemetryService: QueueTelemetryService, + spanName: string, + jobData: object, + processJob: () => T | Promise, + onError: (error: Error) => void, +): Promise { + return telemetryService.startSpanWithTraceContext(spanName, jobData, async (): Promise => { + try { + return await processJob(); + } catch (error) { + // 失敗イベントを待たず、processor Spanがactiveな間にログと通知を行います。 + const normalizedError = error instanceof Error ? error : new Error(String(error)); + try { + onError(normalizedError); + } catch { + // 失敗ログの処理が例外を投げても、Queueへは元のエラーを返します。 + } + throw error; + } + }); +} diff --git a/packages/backend/src/server/ServerService.ts b/packages/backend/src/server/ServerService.ts index 23ead0feba..522d6e2b71 100644 --- a/packages/backend/src/server/ServerService.ts +++ b/packages/backend/src/server/ServerService.ts @@ -21,6 +21,7 @@ import { genIdenticon } from '@/misc/gen-identicon.js'; import { UserEntityService } from '@/core/entities/UserEntityService.js'; import { LoggerService } from '@/core/LoggerService.js'; import { bindThis } from '@/decorators.js'; +import { envOption } from '@/env.js'; import { ActivityPubServerService } from './ActivityPubServerService.js'; import { NodeinfoServerService } from './NodeinfoServerService.js'; import { ApiServerService } from './api/ApiServerService.js'; @@ -31,6 +32,8 @@ import { HealthServerService } from './HealthServerService.js'; import { ClientServerService } from './web/ClientServerService.js'; import { OpenApiServerService } from './api/openapi/OpenApiServerService.js'; import { OAuth2ProviderService } from './oauth/OAuth2ProviderService.js'; +import { registerHttpServerInstrumentation } from './http-server-instrumentation.js'; +import { registerHttpAccessLog } from './http-access-log.js'; const _dirname = fileURLToPath(new URL('.', import.meta.url)); @@ -79,6 +82,8 @@ export class ServerService implements OnApplicationShutdown { logger: false, }); this.#fastify = fastify; + await registerHttpServerInstrumentation(fastify, this.config); + registerHttpAccessLog(fastify); // HSTS // 6months (15552000sec) @@ -89,6 +94,15 @@ export class ServerService implements OnApplicationShutdown { }); } + // for test + if (envOption.enableCrossOriginIsolation) { + fastify.addHook('onRequest', (request, reply, done) => { + reply.header('Cross-Origin-Opener-Policy', 'same-origin'); + reply.header('Cross-Origin-Embedder-Policy', 'credentialless'); + done(); + }); + } + // Register raw-body parser for ActivityPub HTTP signature validation. await fastify.register(fastifyRawBody, { global: false, diff --git a/packages/backend/src/server/api/ApiCallService.ts b/packages/backend/src/server/api/ApiCallService.ts index 0ccb3df631..43cba86e3d 100644 --- a/packages/backend/src/server/api/ApiCallService.ts +++ b/packages/backend/src/server/api/ApiCallService.ts @@ -16,6 +16,7 @@ import type { MiMeta, UserIpsRepository } from '@/models/_.js'; import { createTemp } from '@/misc/create-temp.js'; import { bindThis } from '@/decorators.js'; import { RoleService } from '@/core/RoleService.js'; +import { TelemetryService } from '@/core/telemetry/TelemetryService.js'; import type { Config } from '@/config.js'; import { ApiError } from './error.js'; import { RateLimiterService } from './RateLimiterService.js'; @@ -36,7 +37,6 @@ export class ApiCallService implements OnApplicationShutdown { private logger: Logger; private userIpHistories: Map>; private userIpHistoriesClearIntervalId: NodeJS.Timeout; - private Sentry: typeof import('@sentry/node') | null = null; constructor( @Inject(DI.meta) @@ -52,6 +52,7 @@ export class ApiCallService implements OnApplicationShutdown { private rateLimiterService: RateLimiterService, private roleService: RoleService, private apiLoggerService: ApiLoggerService, + private telemetryService: TelemetryService, ) { this.logger = this.apiLoggerService.logger; this.userIpHistories = new Map>(); @@ -59,12 +60,6 @@ export class ApiCallService implements OnApplicationShutdown { this.userIpHistoriesClearIntervalId = setInterval(() => { this.userIpHistories.clear(); }, 1000 * 60 * 60); - - if (this.config.sentryForBackend) { - import('@sentry/node').then((Sentry) => { - this.Sentry = Sentry; - }); - } } #sendApiError(reply: FastifyReply, err: ApiError): void { @@ -115,35 +110,31 @@ export class ApiCallService implements OnApplicationShutdown { throw err; } else { const errId = randomUUID(); - this.logger.error(`Internal error occurred in ${ep.name}: ${err.message}`, { - ep: ep.name, - ps: data, - e: { - message: err.message, - code: err.name, - stack: err.stack, - id: errId, + this.logger.write({ + level: 'error', + eventName: 'api.endpoint.failed', + message: `Internal error occurred in ${ep.name}: ${err.message}`, + attributes: { + 'api.endpoint': ep.name, + 'error.id': errId, + 'api.params': data, }, + error: err, }); - if (this.Sentry != null) { - this.Sentry.captureMessage(`Internal error occurred in ${ep.name}: ${err.message}`, { - level: 'error', - user: { - id: userId, + this.telemetryService.captureMessage(`Internal error occurred in ${ep.name}: ${err.message}`, { + level: 'error', + userId, + extra: { + ep: ep.name, + e: { + message: err.message, + code: err.name, + stack: err.stack, + id: errId, }, - extra: { - ep: ep.name, - ps: data, - e: { - message: err.message, - code: err.name, - stack: err.stack, - id: errId, - }, - }, - }); - } + }, + }); throw new ApiError(null, { e: { @@ -160,7 +151,7 @@ export class ApiCallService implements OnApplicationShutdown { endpoint: IEndpoint & { exec: any }, request: FastifyRequest<{ Body: Record | undefined, Querystring: Record }>, reply: FastifyReply, - ): void { + ): Promise { const body = request.method === 'GET' ? request.query : request.body; @@ -171,10 +162,11 @@ export class ApiCallService implements OnApplicationShutdown { : body?.['i']; if (token != null && typeof token !== 'string') { reply.code(400); - return; + return Promise.resolve(); } - this.authenticateService.authenticate(token).then(([user, app]) => { - this.call(endpoint, user, app, body, null, request).then((res) => { + + return this.telemetryService.startSpan('API: ' + endpoint.name, () => this.authenticateService.authenticate(token).then(([user, app]) => { + const call = this.call(endpoint, user, app, body, null, request).then((res) => { if (request.method === 'GET' && endpoint.meta.cacheSec && !token && !user) { reply.header('Cache-Control', `public, max-age=${endpoint.meta.cacheSec}`); } @@ -186,9 +178,11 @@ export class ApiCallService implements OnApplicationShutdown { if (user) { this.logIp(request, user); } + + return call; }).catch(err => { this.#sendAuthenticationError(reply, err); - }); + })); } @bindThis @@ -205,48 +199,54 @@ export class ApiCallService implements OnApplicationShutdown { reply.send(); return; } - const [path, cleanup] = await createTemp(); - await stream.pipeline(multipartData.file, fs.createWriteStream(path)); - // ファイルサイズが制限を超えていた場合 - // なお truncated はストリームを読み切ってからでないと機能しないため、stream.pipeline より後にある必要がある - if (multipartData.file.truncated) { - cleanup(); - reply.code(413); - reply.send(); - return; - } + try { + await stream.pipeline(multipartData.file, fs.createWriteStream(path)); - const fields = {} as Record; - for (const [k, v] of Object.entries(multipartData.fields)) { - fields[k] = typeof v === 'object' && 'value' in v ? v.value : undefined; - } - - // https://datatracker.ietf.org/doc/html/rfc6750.html#section-2.1 (case sensitive) - const token = request.headers.authorization?.startsWith('Bearer ') - ? request.headers.authorization.slice(7) - : fields['i']; - if (token != null && typeof token !== 'string') { - reply.code(400); - return; - } - this.authenticateService.authenticate(token).then(([user, app]) => { - this.call(endpoint, user, app, fields, { - name: multipartData.filename, - path: path, - }, request).then((res) => { - this.send(reply, res); - }).catch((err: ApiError) => { - this.#sendApiError(reply, err); - }); - - if (user) { - this.logIp(request, user); + // ファイルサイズが制限を超えていた場合 + // なお truncated はストリームを読み切ってからでないと機能しないため、stream.pipeline より後にある必要がある + if (multipartData.file.truncated) { + reply.code(413); + reply.send(); + return; } - }).catch(err => { - this.#sendAuthenticationError(reply, err); - }); + + const fields = {} as Record; + for (const [k, v] of Object.entries(multipartData.fields)) { + fields[k] = typeof v === 'object' && 'value' in v ? v.value : undefined; + } + + // https://datatracker.ietf.org/doc/html/rfc6750.html#section-2.1 (case sensitive) + const token = request.headers.authorization?.startsWith('Bearer ') + ? request.headers.authorization.slice(7) + : fields['i']; + if (token != null && typeof token !== 'string') { + reply.code(400); + return; + } + + return await this.telemetryService.startSpan('API: ' + endpoint.name, () => this.authenticateService.authenticate(token).then(([user, app]) => { + const call = this.call(endpoint, user, app, fields, { + name: multipartData.filename, + path: path, + }, request).then((res) => { + this.send(reply, res); + }).catch((err: ApiError) => { + this.#sendApiError(reply, err); + }); + + if (user) { + this.logIp(request, user); + } + + return call; + }).catch(err => { + this.#sendAuthenticationError(reply, err); + })); + } finally { + cleanup(); + } } @bindThis @@ -440,16 +440,10 @@ export class ApiCallService implements OnApplicationShutdown { } } - // API invoking - if (this.Sentry != null) { - return await this.Sentry.startSpan({ - name: 'API: ' + ep.name, - }, () => ep.exec(data, user, token, file, request.ip, request.headers) - .catch((err: Error) => this.#onExecError(ep, data, err, user?.id))); - } else { - return await ep.exec(data, user, token, file, request.ip, request.headers) - .catch((err: Error) => this.#onExecError(ep, data, err, user?.id)); - } + // The API span starts in handleRequest/handleMultipartRequest so it also covers + // authentication, rate limiting, and parameter validation. + return await ep.exec(data, user, token, file, request.ip, request.headers) + .catch((err: Error) => this.#onExecError(ep, data, err, user?.id)); } @bindThis diff --git a/packages/backend/src/server/api/endpoints/admin/accounts/create.ts b/packages/backend/src/server/api/endpoints/admin/accounts/create.ts index 6606202118..7aee23adda 100644 --- a/packages/backend/src/server/api/endpoints/admin/accounts/create.ts +++ b/packages/backend/src/server/api/endpoints/admin/accounts/create.ts @@ -7,6 +7,7 @@ import { Inject, Injectable } from '@nestjs/common'; import { Endpoint } from '@/server/api/endpoint-base.js'; import type { MiMeta, UsersRepository } from '@/models/_.js'; import { SignupService } from '@/core/SignupService.js'; +import { RoleService } from '@/core/RoleService.js'; import { UserEntityService } from '@/core/entities/UserEntityService.js'; import { localUsernameSchema, passwordSchema } from '@/models/User.js'; import { DI } from '@/di-symbols.js'; @@ -77,6 +78,7 @@ export default class extends Endpoint { // eslint- private userEntityService: UserEntityService, private signupService: SignupService, + private roleService: RoleService, ) { super(meta, paramDef, async (ps, _me, token) => { const me = _me ? await this.usersRepository.findOneByOrFail({ id: _me.id }) : null; @@ -93,7 +95,7 @@ export default class extends Endpoint { // eslint- // 初期パスワードが設定されていないのに初期パスワードが入力された場合 throw new ApiError(meta.errors.wrongInitialPassword); } - } else if ((this.serverSettings.rootUserId != null && (this.serverSettings.rootUserId !== me?.id)) || token !== null) { + } else if (token !== null || !(await this.roleService.isAdministrator(me))) { // 初回セットアップではなく、管理者でない場合 or 外部トークンを使用している場合 throw new ApiError(meta.errors.accessDenied); } diff --git a/packages/backend/src/server/api/endpoints/admin/meta.ts b/packages/backend/src/server/api/endpoints/admin/meta.ts index 956f991d32..849a05c334 100644 --- a/packages/backend/src/server/api/endpoints/admin/meta.ts +++ b/packages/backend/src/server/api/endpoints/admin/meta.ts @@ -544,6 +544,14 @@ export const meta = { type: 'string', optional: false, nullable: true, }, + urlPreviewSensitiveList: { + type: 'array', + optional: false, nullable: false, + items: { + type: 'string', + optional: false, nullable: false, + }, + }, federation: { type: 'string', enum: ['all', 'specified', 'none'], @@ -760,6 +768,7 @@ export default class extends Endpoint { // eslint- urlPreviewRequireContentLength: instance.urlPreviewRequireContentLength, urlPreviewUserAgent: instance.urlPreviewUserAgent, urlPreviewSummaryProxyUrl: instance.urlPreviewSummaryProxyUrl, + urlPreviewSensitiveList: instance.urlPreviewSensitiveList, federation: instance.federation, federationHosts: instance.federationHosts, deliverSuspendedSoftware: instance.deliverSuspendedSoftware, diff --git a/packages/backend/src/server/api/endpoints/admin/reset-password.ts b/packages/backend/src/server/api/endpoints/admin/reset-password.ts index f19ffa173b..61aed00077 100644 --- a/packages/backend/src/server/api/endpoints/admin/reset-password.ts +++ b/packages/backend/src/server/api/endpoints/admin/reset-password.ts @@ -10,6 +10,7 @@ import { ApiError } from '@/server/api/error.js'; import type { UsersRepository, UserProfilesRepository, MiMeta } from '@/models/_.js'; import { DI } from '@/di-symbols.js'; import { secureRndstr } from '@/misc/secure-rndstr.js'; +import { RoleService } from '@/core/RoleService.js'; import { ModerationLogService } from '@/core/ModerationLogService.js'; export const meta = { @@ -25,10 +26,10 @@ export const meta = { code: 'NO_SUCH_USER', id: 'ccafc7fe-5074-4edd-9dc0-8ef9ef6a701d', }, - cannotResetPasswordOfRootUser: { - message: 'Cannot reset password of the root user.', - code: 'CANNOT_RESET_PASSWORD_OF_ROOT_USER', - id: 'f28fc207-42ca-44c7-a577-44b4f0ec5999', + accessDenied: { + message: 'Access denied.', + code: 'ACCESS_DENIED', + id: 'cda8f8ce-89a6-4f92-8055-33bbe0c1464d', }, }, @@ -66,6 +67,7 @@ export default class extends Endpoint { // eslint- @Inject(DI.userProfilesRepository) private userProfilesRepository: UserProfilesRepository, + private roleService: RoleService, private moderationLogService: ModerationLogService, ) { super(meta, paramDef, async (ps, me) => { @@ -75,8 +77,8 @@ export default class extends Endpoint { // eslint- throw new ApiError(meta.errors.noSuchUser); } - if (this.serverSettings.rootUserId === user.id) { - throw new ApiError(meta.errors.cannotResetPasswordOfRootUser); + if (await this.roleService.isAdministrator(user) && me.id !== user.id) { + throw new ApiError(meta.errors.accessDenied); } const passwd = secureRndstr(8); diff --git a/packages/backend/src/server/api/endpoints/admin/unset-mfa.ts b/packages/backend/src/server/api/endpoints/admin/unset-mfa.ts index 3d964825d3..75b3e3e2bf 100644 --- a/packages/backend/src/server/api/endpoints/admin/unset-mfa.ts +++ b/packages/backend/src/server/api/endpoints/admin/unset-mfa.ts @@ -11,6 +11,7 @@ import { MiUserProfile } from '@/models/UserProfile.js'; import { MiUserSecurityKey } from '@/models/UserSecurityKey.js'; import type { UsersRepository } from '@/models/_.js'; import { DI } from '@/di-symbols.js'; +import { RoleService } from '@/core/RoleService.js'; import { ModerationLogService } from '@/core/ModerationLogService.js'; export const meta = { @@ -26,6 +27,11 @@ export const meta = { code: 'NO_SUCH_USER', id: 'ccafc7fe-5074-4edd-9dc0-8ef9ef6a701d', }, + accessDenied: { + message: 'Access denied.', + code: 'ACCESS_DENIED', + id: 'cda8f8ce-89a6-4f92-8055-33bbe0c1464d', + }, }, } as const; @@ -46,6 +52,7 @@ export default class extends Endpoint { // eslint- @Inject(DI.usersRepository) private usersRepository: UsersRepository, + private roleService: RoleService, private moderationLogService: ModerationLogService, ) { super(meta, paramDef, async (ps, me) => { @@ -55,6 +62,10 @@ export default class extends Endpoint { // eslint- throw new ApiError(meta.errors.noSuchUser); } + if (await this.roleService.isAdministrator(user) && me.id !== user.id) { + throw new ApiError(meta.errors.accessDenied); + } + await this.db.transaction(async (transactionalEntityManager) => { // パスキーを全て削除 await transactionalEntityManager.delete(MiUserSecurityKey, { userId: user.id }); diff --git a/packages/backend/src/server/api/endpoints/admin/update-meta.ts b/packages/backend/src/server/api/endpoints/admin/update-meta.ts index def6a0bda7..4a7df410bc 100644 --- a/packages/backend/src/server/api/endpoints/admin/update-meta.ts +++ b/packages/backend/src/server/api/endpoints/admin/update-meta.ts @@ -189,6 +189,12 @@ export const paramDef = { urlPreviewRequireContentLength: { type: 'boolean' }, urlPreviewUserAgent: { type: 'string', nullable: true }, urlPreviewSummaryProxyUrl: { type: 'string', nullable: true }, + urlPreviewSensitiveList: { + type: 'array', nullable: true, + items: { + type: 'string', + } + }, federation: { type: 'string', enum: ['all', 'none', 'specified'], @@ -734,6 +740,10 @@ export default class extends Endpoint { // eslint- set.urlPreviewSummaryProxyUrl = value === '' ? null : value; } + if (Array.isArray(ps.urlPreviewSensitiveList)) { + set.urlPreviewSensitiveList = ps.urlPreviewSensitiveList.filter(Boolean); + } + if (ps.federation !== undefined) { set.federation = ps.federation; } diff --git a/packages/backend/src/server/api/endpoints/channels/follow.ts b/packages/backend/src/server/api/endpoints/channels/follow.ts index 1812820ba2..b46552281f 100644 --- a/packages/backend/src/server/api/endpoints/channels/follow.ts +++ b/packages/backend/src/server/api/endpoints/channels/follow.ts @@ -8,6 +8,7 @@ import { Endpoint } from '@/server/api/endpoint-base.js'; import type { ChannelsRepository } from '@/models/_.js'; import { DI } from '@/di-symbols.js'; import { ChannelFollowingService } from '@/core/ChannelFollowingService.js'; +import { IdentifiableError } from '@/misc/identifiable-error.js'; import { ApiError } from '../../error.js'; export const meta = { @@ -25,6 +26,11 @@ export const meta = { code: 'NO_SUCH_CHANNEL', id: 'c0031718-d573-4e85-928e-10039f1fbb68', }, + alreadyFollowing: { + message: 'You are already following that channel.', + code: 'ALREADY_FOLLOWING', + id: '7db31665-651e-40c1-8e6e-28e9ad829a2d', + }, }, } as const; @@ -52,7 +58,14 @@ export default class extends Endpoint { // eslint- throw new ApiError(meta.errors.noSuchChannel); } - await this.channelFollowingService.follow(me, channel); + try { + await this.channelFollowingService.follow(me, channel); + } catch (e) { + if (e instanceof IdentifiableError) { + if (e.id === '6e335e39-0203-4418-a936-b3f2dc987845') throw new ApiError(meta.errors.alreadyFollowing); + } + throw e; + } }); } } diff --git a/packages/backend/src/server/http-access-log.ts b/packages/backend/src/server/http-access-log.ts new file mode 100644 index 0000000000..93b61aaf9d --- /dev/null +++ b/packages/backend/src/server/http-access-log.ts @@ -0,0 +1,182 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { Buffer } from 'node:buffer'; +import type { FastifyInstance } from 'fastify'; +import { logManager } from '@/logging/logging-runtime.js'; +import type { LogManager } from '@/logging/LogManager.js'; +import type { LogTraceContext } from '@/logging/types.js'; + +type AccessRequestState = { + traceContext?: LogTraceContext; + errorType?: string; + requestBody?: unknown; + responseBody?: unknown; +}; + +type CapturedBody = { + value: unknown; +}; + +/** Content-Typeから本文の種類だけを取り出します。 */ +function getMediaType(value: string | string[] | number | undefined): string | undefined { + const first = Array.isArray(value) ? value[0] : value; + return typeof first === 'string' ? first.split(';', 1)[0].trim().toLowerCase() : undefined; +} + +/** 秘匿処理を適用できるJSON・form・text本文か判定します。 */ +function isSupportedMediaType(value: string | undefined): boolean { + return value === 'application/json' + || (value?.startsWith('application/') === true && value.endsWith('+json')) + || value === 'application/x-www-form-urlencoded' + || (value?.startsWith('text/') === true && value !== 'text/event-stream'); +} + +/** application/jsonとapplication/*+jsonを構造化対象として判定します。 */ +function isJsonMediaType(value: string | undefined): boolean { + return value === 'application/json' + || (value?.startsWith('application/') === true && value.endsWith('+json')); +} + +/** Streamやバイナリを本文ログの対象から外します。 */ +function isStream(value: unknown): boolean { + if (typeof value !== 'object' || value === null) return false; + try { + const candidate = value as { + pipe?: unknown; + getReader?: unknown; + [Symbol.asyncIterator]?: unknown; + }; + return typeof candidate.pipe === 'function' + || typeof candidate.getReader === 'function' + || typeof candidate[Symbol.asyncIterator] === 'function'; + } catch { + // 本文のgetterが壊れていても、ログ処理が応答を失敗させないよう除外します。 + return true; + } +} + +/** JSON文字列を構造化し、解析できない場合は元の文字列を保持します。 */ +function parseJsonBody(value: string): unknown { + try { + return JSON.parse(value); + } catch { + return value; + } +} + +/** form本文を項目ごとの値へ分解し、キー単位の秘匿処理を可能にします。 */ +function parseFormBody(value: string): Record { + const result: Record = Object.create(null) as Record; + for (const [key, item] of new URLSearchParams(value)) { + const previous = result[key]; + result[key] = typeof previous === 'undefined' + ? item + : Array.isArray(previous) ? [...previous, item] : [previous, item]; + } + return result; +} + +/** 送受信本文から、ログへ渡せる値だけを取り出します。 */ +function captureBody(value: unknown, contentType: string | string[] | number | undefined): CapturedBody | undefined { + const mediaType = getMediaType(contentType); + if (!isSupportedMediaType(mediaType) || isStream(value)) return undefined; + + if (isJsonMediaType(mediaType)) { + if (typeof value === 'string') return { value: parseJsonBody(value) }; + if (Buffer.isBuffer(value)) return { value: parseJsonBody(value.toString('utf8')) }; + return { value }; + } + + if (mediaType === 'application/x-www-form-urlencoded') { + if (typeof value === 'string') return { value: parseFormBody(value) }; + if (Buffer.isBuffer(value)) return { value: parseFormBody(value.toString('utf8')) }; + } + if (typeof value === 'string') return { value }; + if (Buffer.isBuffer(value)) return { value: value.toString('utf8') }; + // form parserなどが返す構造化済みの本文も、通常の正規化処理へ渡します。 + if (typeof value === 'object' && value !== null) return { value }; + return undefined; +} + +/** Errorから本文を含めない型名だけを取り出します。 */ +function getErrorType(error: unknown): string { + if (typeof error === 'object' && error !== null && typeof (error as { name?: unknown }).name === 'string') { + const name = (error as { name: string }).name; + if (name.length > 0) return name; + } + return 'Error'; +} + +/** content-lengthを安全なバイト数へ変換し、未知の応答ではnullを返します。 */ +function getResponseSize(value: string | string[] | number | undefined): number | null { + const first = Array.isArray(value) ? value[0] : value; + const size = typeof first === 'number' ? first : typeof first === 'string' && /^\d+$/.test(first) ? Number(first) : NaN; + return Number.isSafeInteger(size) && size >= 0 ? size : null; +} + +/** Fastify全体へAccess log用のリクエスト・エラー・応答フックを登録します。 */ +export function registerHttpAccessLog(fastify: FastifyInstance, manager: LogManager = logManager): void { + if (!manager.isAccessLogEnabled()) return; + + const states = new WeakMap(); + + // HTTP計装の後に登録し、リクエスト開始時のactiveなTrace Contextを保存します。 + fastify.addHook('onRequest', (request, _reply, done) => { + states.set(request, { + traceContext: manager.getActiveTraceContext(), + }); + done(); + }); + + // Fastifyが応答へ変換したErrorから、型名だけをリクエストへ一時保存します。 + fastify.addHook('onError', (request, _reply, error, done) => { + const state = states.get(request) ?? {}; + state.errorType = getErrorType(error); + states.set(request, state); + done(); + }); + + // 送信前のpayloadを読み取りますが、元の値は変更せず、そのまま次のhookへ渡します。 + fastify.addHook('onSend', (request, reply, payload, done) => { + if (!manager.shouldWriteAccess(reply.statusCode)) { + done(null, payload); + return; + } + + const bodyConfiguration = manager.getAccessLogConfiguration().bodies; + const state = states.get(request) ?? {}; + if (bodyConfiguration.request && !('requestBody' in state)) { + // ActivityPubもFastifyが解析したrequest.bodyだけを対象にし、rawBodyやheaderは参照しません。 + const captured = captureBody(request.body, request.headers['content-type']); + if (captured !== undefined) state.requestBody = captured.value; + } + if (bodyConfiguration.response && !('responseBody' in state)) { + const captured = captureBody(payload, reply.getHeader('content-type')); + if (captured !== undefined) state.responseBody = captured.value; + } + states.set(request, state); + done(null, payload); + }); + + // 応答完了時にFastifyの値を集め、LogManagerへAccess logを渡します。 + fastify.addHook('onResponse', (request, reply, done) => { + const state = states.get(request); + const input = { + method: request.method, + route: request.routeOptions.url ?? null, + statusCode: reply.statusCode, + durationMs: reply.elapsedTime, + responseSizeBytes: getResponseSize(reply.getHeader('content-length')), + ...(state?.errorType !== undefined ? { errorType: state.errorType } : {}), + ...(state != null && 'requestBody' in state ? { requestBody: state.requestBody } : {}), + ...(state != null && 'responseBody' in state ? { responseBody: state.responseBody } : {}), + ...(state?.traceContext !== undefined ? { traceContext: state.traceContext } : {}), + }; + manager.writeAccess(input); + states.delete(request); + done(); + }); +} diff --git a/packages/backend/src/server/http-server-instrumentation.ts b/packages/backend/src/server/http-server-instrumentation.ts new file mode 100644 index 0000000000..c17f3c1bc0 --- /dev/null +++ b/packages/backend/src/server/http-server-instrumentation.ts @@ -0,0 +1,35 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import type { Config } from '@/config.js'; +import type { FastifyInstance } from 'fastify'; + +type TelemetryConfig = Pick; + +export function shouldRegisterHttpServerInstrumentation(config: TelemetryConfig): boolean { + // Sentryもリクエストspanを作成するため、両方を登録すると重複して出力される。 + return config.otelForBackend != null && config.sentryForBackend == null; +} + +/** + * すべてのルート・フックより前にリクエスト計装を登録し、ActivityPubや + * well-knownを含む全HTTP受信経路を1つのroot spanとして計測する。 + */ +export async function registerHttpServerInstrumentation(fastify: FastifyInstance, config: TelemetryConfig): Promise { + if (!shouldRegisterHttpServerInstrumentation(config)) return; + + const { FastifyOtelInstrumentation } = await import('@fastify/otel'); + const instrumentation = new FastifyOtelInstrumentation({ + requestHook: (span, request) => { + const route = request.routeOptions.url; + if (route != null) { + // デフォルトだとトレース名が「request」で固定されてしまうため、判別がつかなくなる。 + // ルート名をspan名に設定することで、トレースビューでルートごとの処理時間を確認できるようになる。 + span.updateName(`${request.method} ${route}`); + } + }, + }); + await fastify.register(instrumentation.plugin()); +} diff --git a/packages/backend/src/server/web/UrlPreviewService.ts b/packages/backend/src/server/web/UrlPreviewService.ts index 886e876c40..7b7ee2c425 100644 --- a/packages/backend/src/server/web/UrlPreviewService.ts +++ b/packages/backend/src/server/web/UrlPreviewService.ts @@ -3,22 +3,26 @@ * SPDX-License-Identifier: AGPL-3.0-only */ -import { Inject, Injectable } from '@nestjs/common'; +import { Inject, Injectable, OnApplicationShutdown } from '@nestjs/common'; import type { SummalyResult } from '@misskey-dev/summaly'; import { DI } from '@/di-symbols.js'; import type { Config } from '@/config.js'; import { HttpRequestService } from '@/core/HttpRequestService.js'; import type Logger from '@/logger.js'; +import { deepClone } from '@/misc/clone.js'; import { query } from '@/misc/prelude/url.js'; +import { MemoryKVCache } from '@/misc/cache.js'; import { LoggerService } from '@/core/LoggerService.js'; +import { UtilityService } from '@/core/UtilityService.js'; import { bindThis } from '@/decorators.js'; import { ApiError } from '@/server/api/error.js'; import { MiMeta } from '@/models/Meta.js'; import type { FastifyRequest, FastifyReply } from 'fastify'; @Injectable() -export class UrlPreviewService { +export class UrlPreviewService implements OnApplicationShutdown { private logger: Logger; + private summaryCache: MemoryKVCache; private readonly summalyDefaultUserAgent: string; constructor( @@ -29,9 +33,11 @@ export class UrlPreviewService { private meta: MiMeta, private httpRequestService: HttpRequestService, + private utilityService: UtilityService, private loggerService: LoggerService, ) { this.logger = this.loggerService.getLogger('url-preview'); + this.summaryCache = new MemoryKVCache(1000 * 60 * 60, 100); // 1h, 100件 this.summalyDefaultUserAgent = `SummalyBot/${_SUMMALY_VERSION_} (${this.config.url}; +https://github.com/misskey-dev/summaly/blob/master/README.md)`; } @@ -78,23 +84,39 @@ export class UrlPreviewService { : `Getting preview of ${url}@${lang} ...`); try { - const summary = this.meta.urlPreviewSummaryProxyUrl - ? await this.fetchSummaryFromProxy(url, this.meta, lang) - : await this.fetchSummary(url, this.meta, lang); + const fetcher = async () => { + const result = await ( + this.meta.urlPreviewSummaryProxyUrl + ? this.fetchSummaryFromProxy(url, lang) + : this.fetchSummary(url, lang) + ); + + if (!result.url.startsWith('http://') && !result.url.startsWith('https://')) { + return undefined; + } + + if (result.player.url && !result.player.url.startsWith('http://') && !result.player.url.startsWith('https://')) { + return undefined; + } + + return result; + }; + + const summary = deepClone(await this.summaryCache.fetchMaybe(`${url}@${lang ?? '_DEFAULT_'}`, fetcher)); + + if (summary == null) { + throw new Error('Invalid summary'); + } this.logger.succ(`Got preview of ${url}: ${summary.title}`); - if (!(summary.url.startsWith('http://') || summary.url.startsWith('https://'))) { - throw new Error('unsupported schema included'); - } - - if (summary.player.url && !(summary.player.url.startsWith('http://') || summary.player.url.startsWith('https://'))) { - throw new Error('unsupported schema included'); - } - summary.icon = this.wrap(summary.icon); summary.thumbnail = this.wrap(summary.thumbnail); + if (summary.sensitive !== true) { + summary.sensitive = this.utilityService.isKeyWordIncluded(summary.url, this.meta.urlPreviewSensitiveList); + } + // Cache 1day reply.header('Cache-Control', 'max-age=86400, immutable'); @@ -114,7 +136,8 @@ export class UrlPreviewService { } } - private async fetchSummary(url: string, meta: MiMeta, lang?: string): Promise { + @bindThis + private async fetchSummary(url: string, lang?: string): Promise { const { summaly } = await import('@misskey-dev/summaly'); return summaly(url, { @@ -124,25 +147,36 @@ export class UrlPreviewService { http: this.httpRequestService.httpAgent, https: this.httpRequestService.httpsAgent, }, - userAgent: meta.urlPreviewUserAgent ?? this.summalyDefaultUserAgent, - operationTimeout: meta.urlPreviewTimeout, - contentLengthLimit: meta.urlPreviewMaximumContentLength, - contentLengthRequired: meta.urlPreviewRequireContentLength, + userAgent: this.meta.urlPreviewUserAgent ?? this.summalyDefaultUserAgent, + operationTimeout: this.meta.urlPreviewTimeout, + contentLengthLimit: this.meta.urlPreviewMaximumContentLength, + contentLengthRequired: this.meta.urlPreviewRequireContentLength, }); } - private fetchSummaryFromProxy(url: string, meta: MiMeta, lang?: string): Promise { - const proxy = meta.urlPreviewSummaryProxyUrl!; + @bindThis + private fetchSummaryFromProxy(url: string, lang?: string): Promise { + const proxy = this.meta.urlPreviewSummaryProxyUrl!; const queryStr = query({ url: url, lang: lang ?? 'ja-JP', followRedirects: this.meta.urlPreviewAllowRedirect, - userAgent: meta.urlPreviewUserAgent ?? this.summalyDefaultUserAgent, - operationTimeout: meta.urlPreviewTimeout, - contentLengthLimit: meta.urlPreviewMaximumContentLength, - contentLengthRequired: meta.urlPreviewRequireContentLength, + userAgent: this.meta.urlPreviewUserAgent ?? this.summalyDefaultUserAgent, + operationTimeout: this.meta.urlPreviewTimeout, + contentLengthLimit: this.meta.urlPreviewMaximumContentLength, + contentLengthRequired: this.meta.urlPreviewRequireContentLength, }); return this.httpRequestService.getJson(`${proxy}?${queryStr}`, 'application/json, */*', undefined, true); } + + @bindThis + public dispose(): void { + this.summaryCache.dispose(); + } + + @bindThis + public onApplicationShutdown(): void { + this.dispose(); + } } diff --git a/packages/backend/test-federation/compose.yml b/packages/backend/test-federation/compose.yml index 53e297f867..8bc386c00c 100644 --- a/packages/backend/test-federation/compose.yml +++ b/packages/backend/test-federation/compose.yml @@ -139,7 +139,7 @@ services: bash -c " npm install -g pnpm pnpm -F backend i --frozen-lockfile - pnpm exec tsgo -p ./packages/backend/test-federation + pnpm exec tsc -p ./packages/backend/test-federation node ./packages/backend/test-federation/built/daemon.js " diff --git a/packages/backend/test/e2e/api-visibility.ts b/packages/backend/test/e2e/api-visibility.ts index 4f244c0cce..8bfca077ba 100644 --- a/packages/backend/test/e2e/api-visibility.ts +++ b/packages/backend/test/e2e/api-visibility.ts @@ -6,10 +6,12 @@ process.env.NODE_ENV = 'test'; import * as assert from 'assert'; -import { describe, beforeAll, beforeEach, test } from 'vitest'; +import { describe, beforeAll, beforeEach, test, vi } from 'vitest'; import { UserToken, api, post, signup } from '../utils.js'; import type * as misskey from 'misskey-js'; +const waitForPushToTlOptions = { timeout: 3000, interval: 25 }; + describe('API visibility', () => { describe('Note visibility', () => { //#region vars @@ -409,10 +411,12 @@ describe('API visibility', () => { //#region HTL test('[HTL] public-post が 自分が見れる', async () => { - const res = await api('notes/timeline', { limit: 100 }, alice); - assert.strictEqual(res.status, 200); - const notes = res.body.filter(n => n.id === pub.id); - assert.strictEqual(notes[0].text, 'x'); + await vi.waitFor(async () => { + const res = await api('notes/timeline', { limit: 100 }, alice); + assert.strictEqual(res.status, 200); + const notes = res.body.filter(n => n.id === pub.id); + assert.strictEqual(notes[0].text, 'x'); + }, waitForPushToTlOptions); }); test('[HTL] public-post が 非フォロワーから見れない', async () => { @@ -423,10 +427,12 @@ describe('API visibility', () => { }); test('[HTL] followers-post が フォロワーから見れる', async () => { - const res = await api('notes/timeline', { limit: 100 }, follower); - assert.strictEqual(res.status, 200); - const notes = res.body.filter(n => n.id === fol.id); - assert.strictEqual(notes[0].text, 'x'); + await vi.waitFor(async () => { + const res = await api('notes/timeline', { limit: 100 }, follower); + assert.strictEqual(res.status, 200); + const notes = res.body.filter(n => n.id === fol.id); + assert.strictEqual(notes[0].text, 'x'); + }, waitForPushToTlOptions); }); //#endregion diff --git a/packages/backend/test/e2e/block.ts b/packages/backend/test/e2e/block.ts index 9ef4dd8be9..86e27b461e 100644 --- a/packages/backend/test/e2e/block.ts +++ b/packages/backend/test/e2e/block.ts @@ -6,10 +6,12 @@ process.env.NODE_ENV = 'test'; import * as assert from 'assert'; -import { describe, beforeAll, test } from 'vitest'; +import { describe, beforeAll, test, vi } from 'vitest'; import { api, castAsError, post, signup } from '../utils.js'; import type * as misskey from 'misskey-js'; +const waitForPushToTlOptions = { timeout: 3000, interval: 25 }; + describe('Block', () => { // alice blocks bob let alice: misskey.entities.SignupResponse; @@ -75,13 +77,15 @@ describe('Block', () => { const bobNote = await post(bob, { text: 'hi' }); const carolNote = await post(carol, { text: 'hi' }); - const res = await api('notes/local-timeline', {}, bob); - const body = res.body as misskey.entities.Note[]; + await vi.waitFor(async () => { + const res = await api('notes/local-timeline', {}, bob); + const body = res.body as misskey.entities.Note[]; - assert.strictEqual(res.status, 200); - assert.strictEqual(Array.isArray(res.body), true); - assert.strictEqual(body.some(note => note.id === aliceNote.id), false); - assert.strictEqual(body.some(note => note.id === bobNote.id), true); - assert.strictEqual(body.some(note => note.id === carolNote.id), true); + assert.strictEqual(res.status, 200); + assert.strictEqual(Array.isArray(res.body), true); + assert.strictEqual(body.some(note => note.id === aliceNote.id), false); + assert.strictEqual(body.some(note => note.id === bobNote.id), true); + assert.strictEqual(body.some(note => note.id === carolNote.id), true); + }, waitForPushToTlOptions); }); }); diff --git a/packages/backend/test/e2e/channel.ts b/packages/backend/test/e2e/channel.ts new file mode 100644 index 0000000000..c2fd98cbf4 --- /dev/null +++ b/packages/backend/test/e2e/channel.ts @@ -0,0 +1,36 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +process.env.NODE_ENV = 'test'; + +import * as assert from 'assert'; +import { describe, beforeAll, test } from 'vitest'; +import { api, castAsError, signup } from '../utils.js'; +import type * as misskey from 'misskey-js'; + +describe('Channel', () => { + let alice: misskey.entities.SignupResponse; + beforeAll(async () => { + alice = await signup({ username: 'alice' }); + }); + + describe('Follow', () => { + let channel: misskey.entities.ChannelsCreateResponse; + + beforeAll(async () => { + const res = await api('channels/create', { name: 'follow-test-channel' }, alice); + channel = res.body; + }); + + test('フォローしているチャンネルを再度フォローするとALREADY_FOLLOWINGエラーになる', async () => { + const res1 = await api('channels/follow', { channelId: channel.id }, alice); + assert.strictEqual(res1.status, 204); + + const res2 = await api('channels/follow', { channelId: channel.id }, alice); + assert.strictEqual(res2.status, 400); + assert.strictEqual(castAsError(res2.body as any).error.code, 'ALREADY_FOLLOWING'); + }); + }); +}); diff --git a/packages/backend/test/e2e/endpoints.ts b/packages/backend/test/e2e/endpoints.ts index 09198384c4..402836ca47 100644 --- a/packages/backend/test/e2e/endpoints.ts +++ b/packages/backend/test/e2e/endpoints.ts @@ -6,7 +6,7 @@ process.env.NODE_ENV = 'test'; import * as assert from 'assert'; -import { describe, beforeAll, test, expect } from 'vitest'; +import { describe, beforeAll, test, expect, vi } from 'vitest'; // node-fetch only supports it's own Blob yet // https://github.com/node-fetch/node-fetch/pull/1664 import { Blob } from 'node-fetch'; @@ -14,6 +14,8 @@ import { api, castAsError, initTestDb, post, role, signup, simpleGet, uploadFile import type * as misskey from 'misskey-js'; import { MiUser } from '@/models/_.js'; +const waitForPushToTlOptions = { timeout: 3000, interval: 25 }; + describe('Endpoints', () => { let alice: misskey.entities.SignupResponse; let bob: misskey.entities.SignupResponse; @@ -1149,12 +1151,14 @@ describe('Endpoints', () => { visibility: 'followers', }); - const res = await api('notes/timeline', {}, dave); + await vi.waitFor(async () => { + const res = await api('notes/timeline', {}, dave); - assert.strictEqual(res.status, 200); - assert.strictEqual(Array.isArray(res.body), true); - assert.strictEqual(res.body.length, 1); - assert.strictEqual(res.body[0].id, carolPost.id); + assert.strictEqual(res.status, 200); + assert.strictEqual(Array.isArray(res.body), true); + assert.strictEqual(res.body.length, 1); + assert.strictEqual(res.body[0].id, carolPost.id); + }, waitForPushToTlOptions); }); }); diff --git a/packages/backend/test/e2e/mute.ts b/packages/backend/test/e2e/mute.ts index f5cc875e7c..c579c52a2b 100644 --- a/packages/backend/test/e2e/mute.ts +++ b/packages/backend/test/e2e/mute.ts @@ -6,10 +6,12 @@ process.env.NODE_ENV = 'test'; import * as assert from 'assert'; -import { beforeAll, describe, test } from 'vitest'; +import { beforeAll, describe, test, vi } from 'vitest'; import { api, post, react, signup, waitFire } from '../utils.js'; import type * as misskey from 'misskey-js'; +const waitForPushToTlOptions = { timeout: 3000, interval: 25 }; + describe('Mute', () => { // alice mutes carol let alice: misskey.entities.SignupResponse; @@ -67,13 +69,15 @@ describe('Mute', () => { const bobNote = await post(bob, { text: 'hi' }); const carolNote = await post(carol, { text: 'hi' }); - const res = await api('notes/local-timeline', {}, alice); + await vi.waitFor(async () => { + const res = await api('notes/local-timeline', {}, alice); - assert.strictEqual(res.status, 200); - assert.strictEqual(Array.isArray(res.body), true); - assert.strictEqual(res.body.some(note => note.id === aliceNote.id), true); - assert.strictEqual(res.body.some(note => note.id === bobNote.id), true); - assert.strictEqual(res.body.some(note => note.id === carolNote.id), false); + assert.strictEqual(res.status, 200); + assert.strictEqual(Array.isArray(res.body), true); + assert.strictEqual(res.body.some(note => note.id === aliceNote.id), true); + assert.strictEqual(res.body.some(note => note.id === bobNote.id), true); + assert.strictEqual(res.body.some(note => note.id === carolNote.id), false); + }, waitForPushToTlOptions); }); test('タイムラインにミュートしているユーザーの投稿のRenoteが含まれない', async () => { @@ -83,13 +87,15 @@ describe('Mute', () => { renoteId: carolNote.id, }); - const res = await api('notes/local-timeline', {}, alice); + await vi.waitFor(async () => { + const res = await api('notes/local-timeline', {}, alice); - assert.strictEqual(res.status, 200); - assert.strictEqual(Array.isArray(res.body), true); - assert.strictEqual(res.body.some(note => note.id === aliceNote.id), true); - assert.strictEqual(res.body.some(note => note.id === bobNote.id), false); - assert.strictEqual(res.body.some(note => note.id === carolNote.id), false); + assert.strictEqual(res.status, 200); + assert.strictEqual(Array.isArray(res.body), true); + assert.strictEqual(res.body.some(note => note.id === aliceNote.id), true); + assert.strictEqual(res.body.some(note => note.id === bobNote.id), false); + assert.strictEqual(res.body.some(note => note.id === carolNote.id), false); + }, waitForPushToTlOptions); }); }); diff --git a/packages/backend/test/e2e/renote-mute.ts b/packages/backend/test/e2e/renote-mute.ts index 785c9dff8b..555d7c2a20 100644 --- a/packages/backend/test/e2e/renote-mute.ts +++ b/packages/backend/test/e2e/renote-mute.ts @@ -6,11 +6,12 @@ process.env.NODE_ENV = 'test'; import * as assert from 'assert'; -import { beforeAll, describe, test } from 'vitest'; -import { setTimeout } from 'node:timers/promises'; +import { beforeAll, describe, test, vi } from 'vitest'; import { api, post, signup, waitFire } from '../utils.js'; import type * as misskey from 'misskey-js'; +const waitForPushToTlOptions = { timeout: 3000, interval: 25 }; + describe('Renote Mute', () => { // alice mutes carol let alice: misskey.entities.SignupResponse; @@ -36,16 +37,15 @@ describe('Renote Mute', () => { const carolRenote = await post(carol, { renoteId: bobNote.id }); const carolNote = await post(carol, { text: 'hi' }); - // redisに追加されるのを待つ - await setTimeout(100); + await vi.waitFor(async () => { + const res = await api('notes/local-timeline', {}, alice); - const res = await api('notes/local-timeline', {}, alice); - - assert.strictEqual(res.status, 200); - assert.strictEqual(Array.isArray(res.body), true); - assert.strictEqual(res.body.some(note => note.id === bobNote.id), true); - assert.strictEqual(res.body.some(note => note.id === carolRenote.id), false); - assert.strictEqual(res.body.some(note => note.id === carolNote.id), true); + assert.strictEqual(res.status, 200); + assert.strictEqual(Array.isArray(res.body), true); + assert.strictEqual(res.body.some(note => note.id === bobNote.id), true); + assert.strictEqual(res.body.some(note => note.id === carolRenote.id), false); + assert.strictEqual(res.body.some(note => note.id === carolNote.id), true); + }, waitForPushToTlOptions); }); test('タイムラインにリノートミュートしているユーザーの引用が含まれる', async () => { @@ -53,16 +53,15 @@ describe('Renote Mute', () => { const carolRenote = await post(carol, { renoteId: bobNote.id, text: 'kore' }); const carolNote = await post(carol, { text: 'hi' }); - // redisに追加されるのを待つ - await setTimeout(100); + await vi.waitFor(async () => { + const res = await api('notes/local-timeline', {}, alice); - const res = await api('notes/local-timeline', {}, alice); - - assert.strictEqual(res.status, 200); - assert.strictEqual(Array.isArray(res.body), true); - assert.strictEqual(res.body.some(note => note.id === bobNote.id), true); - assert.strictEqual(res.body.some(note => note.id === carolRenote.id), true); - assert.strictEqual(res.body.some(note => note.id === carolNote.id), true); + assert.strictEqual(res.status, 200); + assert.strictEqual(Array.isArray(res.body), true); + assert.strictEqual(res.body.some(note => note.id === bobNote.id), true); + assert.strictEqual(res.body.some(note => note.id === carolRenote.id), true); + assert.strictEqual(res.body.some(note => note.id === carolNote.id), true); + }, waitForPushToTlOptions); }); // #12956 @@ -70,15 +69,14 @@ describe('Renote Mute', () => { const carolNote = await post(carol, { text: 'hi' }); const bobRenote = await post(bob, { renoteId: carolNote.id }); - // redisに追加されるのを待つ - await setTimeout(100); + await vi.waitFor(async () => { + const res = await api('notes/local-timeline', {}, alice); - const res = await api('notes/local-timeline', {}, alice); - - assert.strictEqual(res.status, 200); - assert.strictEqual(Array.isArray(res.body), true); - assert.strictEqual(res.body.some(note => note.id === carolNote.id), true); - assert.strictEqual(res.body.some(note => note.id === bobRenote.id), true); + assert.strictEqual(res.status, 200); + assert.strictEqual(Array.isArray(res.body), true); + assert.strictEqual(res.body.some(note => note.id === carolNote.id), true); + assert.strictEqual(res.body.some(note => note.id === bobRenote.id), true); + }, waitForPushToTlOptions); }); test('ストリームにリノートミュートしているユーザーのリノートが流れない', async () => { diff --git a/packages/backend/test/e2e/streaming.ts b/packages/backend/test/e2e/streaming.ts index a051be6c3c..ec2e7592fa 100644 --- a/packages/backend/test/e2e/streaming.ts +++ b/packages/backend/test/e2e/streaming.ts @@ -510,7 +510,7 @@ describe('Streaming', () => { test('withReplies: true のとき自分のfollowers投稿に対するリプライが流れる', async () => { const erinNote = await post(erin, { text: 'hi', visibility: 'followers' }); const fired = await waitFire( - erin, 'homeTimeline', // erin:home + erin, 'hybridTimeline', // erin:Hybrid () => api('notes/create', { text: 'hello', replyId: erinNote.id }, ayano), // ayano reply to erin's followers post msg => msg.type === 'note' && msg.body.userId === ayano.id, // wait ayano ); @@ -521,7 +521,7 @@ describe('Streaming', () => { test('withReplies: false でも自分の投稿に対するリプライが流れる', async () => { const ayanoNote = await post(ayano, { text: 'hi', visibility: 'followers' }); const fired = await waitFire( - ayano, 'homeTimeline', // ayano:home + ayano, 'hybridTimeline', // ayano:Hybrid () => api('notes/create', { text: 'hello', replyId: ayanoNote.id }, erin), // erin reply to ayano's followers post msg => msg.type === 'note' && msg.body.userId === erin.id, // wait erin ); @@ -530,9 +530,12 @@ describe('Streaming', () => { }); test('withReplies: true のフォローしていない人のfollowersノートに対するリプライが流れない', async () => { + // ayano は kyoko をフォローしているため kyoko の followers 投稿にリプライできるが、 + // erin は kyoko をフォローしていないため、そのリプライは erin の Hybrid Timeline には流れないはず + const kyokoFollowersNote = await post(kyoko, { text: 'hi', visibility: 'followers' }); const fired = await waitFire( - erin, 'homeTimeline', // erin:home - () => api('notes/create', { text: 'hello', replyId: chitose.id }, ayano), // ayano reply to chitose's post + erin, 'hybridTimeline', // erin:Hybrid + () => api('notes/create', { text: 'hello', replyId: kyokoFollowersNote.id }, ayano), // ayano reply to kyoko's followers post msg => msg.type === 'note' && msg.body.userId === ayano.id, // wait ayano ); @@ -680,7 +683,7 @@ describe('Streaming', () => { const fired = await waitFire( chitose, 'userList', () => api('notes/create', { text: 'foo' }, takumi), - msg => msg.type === 'note' && msg.body.userId === kyoko.id, + msg => msg.type === 'note' && msg.body.userId === takumi.id, { listId: list.id }, ); diff --git a/packages/backend/test/e2e/telemetry-redis-instrumentation.ts b/packages/backend/test/e2e/telemetry-redis-instrumentation.ts new file mode 100644 index 0000000000..620f6adc69 --- /dev/null +++ b/packages/backend/test/e2e/telemetry-redis-instrumentation.ts @@ -0,0 +1,88 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { randomUUID } from 'node:crypto'; +import { describe, expect, test } from 'vitest'; +import { Queue, Worker } from 'bullmq'; +import { SpanKind, SpanStatusCode } from '@opentelemetry/api'; +import { InMemorySpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base'; +import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node'; +import { loadConfig } from '@/config.js'; +import { installRedisInstrumentation } from '@/core/telemetry/redis-instrumentation.js'; + +const config = loadConfig(); + +describe('Redis telemetry instrumentation', () => { + test('records Redis spans below HTTP and BullMQ worker spans without Redis arguments', async () => { + const exporter = new InMemorySpanExporter(); + const provider = new NodeTracerProvider({ + spanProcessors: [new SimpleSpanProcessor(exporter)], + }); + provider.register(); + + const tracer = provider.getTracer('telemetry-redis-instrumentation-test'); + const uninstall = installRedisInstrumentation(tracer, SpanKind.CLIENT, SpanStatusCode.ERROR, { + captureCommandSpans: true, + }); + const queueName = `telemetry-${randomUUID()}`; + const prefix = `telemetry-${randomUUID()}`; + const connection = { + host: config.redis.host, + port: config.redis.port, + ...(config.redis.password != null ? { password: config.redis.password } : {}), + }; + const queue = new Queue(queueName, { connection, prefix }); + let worker: Worker | undefined; + let httpSpanId: string | undefined; + let jobSpanId: string | undefined; + const secret = `secret-${randomUUID()}`; + + try { + const processed = new Promise((resolve, reject) => { + worker = new Worker(queueName, async job => { + return await tracer.startActiveSpan('Queue: telemetry test', async jobSpan => { + jobSpanId = jobSpan.spanContext().spanId; + try { + // updateData uses BullMQ's worker-side ioredis client. + await job.updateData({ secret }); + return 'ok'; + } finally { + jobSpan.end(); + } + }); + }, { connection, prefix }); + worker.once('completed', () => resolve()); + worker.once('failed', (_job, error) => reject(error)); + }); + + await tracer.startActiveSpan('HTTP POST /telemetry-test', async httpSpan => { + httpSpanId = httpSpan.spanContext().spanId; + try { + // Queue#add uses BullMQ's producer-side ioredis client. + await queue.add('probe', { secret }); + } finally { + httpSpan.end(); + } + }); + await processed; + await provider.forceFlush(); + + const redisSpans = exporter.getFinishedSpans().filter(span => span.attributes['db.system.name'] === 'redis'); + expect(redisSpans.some(span => span.parentSpanContext?.spanId === httpSpanId)).toBe(true); + expect(redisSpans.some(span => span.parentSpanContext?.spanId === jobSpanId)).toBe(true); + for (const span of redisSpans) { + expect(span.attributes).not.toHaveProperty('db.statement'); + expect(span.attributes).not.toHaveProperty('db.query.text'); + expect(Object.values(span.attributes)).not.toContain(secret); + } + } finally { + await worker?.close(); + await queue.obliterate({ force: true }); + await queue.close(); + uninstall(); + await provider.shutdown(); + } + }, 30000); +}); diff --git a/packages/backend/test/e2e/user-notes.ts b/packages/backend/test/e2e/user-notes.ts index 2f89ac54ce..c6fabca05f 100644 --- a/packages/backend/test/e2e/user-notes.ts +++ b/packages/backend/test/e2e/user-notes.ts @@ -6,10 +6,12 @@ process.env.NODE_ENV = 'test'; import * as assert from 'assert'; -import { beforeAll, describe, test } from 'vitest'; +import { beforeAll, describe, test, vi } from 'vitest'; import { api, post, signup, uploadUrl } from '../utils.js'; import type * as misskey from 'misskey-js'; +const waitForPushToTlOptions = { timeout: 3000, interval: 25 }; + describe('users/notes', () => { let alice: misskey.entities.SignupResponse; let jpgNote: misskey.entities.Note; @@ -32,16 +34,18 @@ describe('users/notes', () => { }, 1000 * 60 * 2); test('withFiles', async () => { - const res = await api('users/notes', { - userId: alice.id, - withFiles: true, - }, alice); + await vi.waitFor(async () => { + const res = await api('users/notes', { + userId: alice.id, + withFiles: true, + }, alice); - assert.strictEqual(res.status, 200); - assert.strictEqual(Array.isArray(res.body), true); - assert.strictEqual(res.body.length, 3); - assert.strictEqual(res.body.some((note: any) => note.id === jpgNote.id), true); - assert.strictEqual(res.body.some((note: any) => note.id === pngNote.id), true); - assert.strictEqual(res.body.some((note: any) => note.id === jpgPngNote.id), true); + assert.strictEqual(res.status, 200); + assert.strictEqual(Array.isArray(res.body), true); + assert.strictEqual(res.body.length, 3); + assert.strictEqual(res.body.some((note: any) => note.id === jpgNote.id), true); + assert.strictEqual(res.body.some((note: any) => note.id === pngNote.id), true); + assert.strictEqual(res.body.some((note: any) => note.id === jpgPngNote.id), true); + }, waitForPushToTlOptions); }); }); diff --git a/packages/backend/test/setup.e2e.ts b/packages/backend/test/setup.e2e.ts index 3141dc15ad..59bc6fcc62 100644 --- a/packages/backend/test/setup.e2e.ts +++ b/packages/backend/test/setup.e2e.ts @@ -7,6 +7,9 @@ import { beforeAll } from 'vitest'; import { initTestDb, sendEnvResetRequest } from './utils.js'; beforeAll(async () => { - await initTestDb(false); + // 前ファイルのNestJSアプリをdispose(env-reset)した後にスキーマをdrop & 再作成する。 + // 逆順だと、前ファイルの最後のテストが投げっぱなしにした非同期処理(cacheServiceのrefresh等)が + // dispose前のdrop中に発火し、Unhandled Rejection (relation does not exist) でクラッシュしうる。 await sendEnvResetRequest(); + await initTestDb(false); }); diff --git a/packages/backend/test/unit/boot/process-error-handler.ts b/packages/backend/test/unit/boot/process-error-handler.ts new file mode 100644 index 0000000000..32e341cc48 --- /dev/null +++ b/packages/backend/test/unit/boot/process-error-handler.ts @@ -0,0 +1,72 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import type { LogWriteInput } from '@/logging/types.js'; +import { installProcessErrorHandlers } from '@/boot/process-error-handler.js'; + +type ProcessListener = (...args: unknown[]) => void; + +/** テスト用に、登録されたプロセスイベントを呼び出せる処理対象を作成します。 */ +function createProcessLike(): { + readonly process: { on: (event: string, listener: ProcessListener) => void }; + readonly listeners: Map; +} { + const listeners = new Map(); + return { + process: { + on: (event, listener) => listeners.set(event, listener), + }, + listeners, + }; +} + +describe('installProcessErrorHandlers', () => { + test('records unhandled rejections and uncaught exceptions as structured errors', () => { + const { process, listeners } = createProcessLike(); + const write = vi.fn<(input: LogWriteInput) => void>(); + const rejection = new Error('rejected'); + const exception = new TypeError('uncaught'); + + installProcessErrorHandlers({ process: process as never, logger: { write }, quiet: false }); + // eslint-disable-next-line @typescript-eslint/no-non-null-assertion + listeners.get('unhandledRejection')!(rejection); + // eslint-disable-next-line @typescript-eslint/no-non-null-assertion + listeners.get('uncaughtException')!(exception); + + expect(write).toHaveBeenNthCalledWith(1, { + level: 'error', + eventName: 'process.unhandled_rejection', + message: 'Unhandled promise rejection', + error: rejection, + }); + expect(write).toHaveBeenNthCalledWith(2, { + level: 'error', + eventName: 'process.uncaught_exception', + message: 'Uncaught exception', + error: exception, + }); + }); + + test('does not register the unhandled rejection handler in quiet mode', () => { + const { process, listeners } = createProcessLike(); + + installProcessErrorHandlers({ process: process as never, logger: { write: vi.fn() }, quiet: true }); + + expect([...listeners.keys()]).toEqual(['uncaughtException']); + }); + + test('does not rethrow when the logger fails', () => { + const { process, listeners } = createProcessLike(); + const write = vi.fn(() => { + throw new Error('logger failed'); + }); + + installProcessErrorHandlers({ process: process as never, logger: { write }, quiet: false }); + + expect(() => listeners.get('unhandledRejection')!(new Error('rejected'))).not.toThrow(); + expect(() => listeners.get('uncaughtException')!(new Error('uncaught'))).not.toThrow(); + }); +}); diff --git a/packages/backend/test/unit/boot/shutdown-handler.ts b/packages/backend/test/unit/boot/shutdown-handler.ts new file mode 100644 index 0000000000..399820f08c --- /dev/null +++ b/packages/backend/test/unit/boot/shutdown-handler.ts @@ -0,0 +1,113 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; + +describe('shutdown-handler', () => { + test('runs shutdown tasks once and exits on SIGTERM or SIGINT', async () => { + vi.resetModules(); + const { installShutdownSignalHandlers, isShutdownInProgress } = await import('@/boot/shutdown-handler.js'); + const handlers = new Map Promise>(); + const processLike = { + once: vi.fn((event: string, handler: () => Promise) => { + handlers.set(event, handler); + return processLike; + }), + }; + const calls: string[] = []; + const shutdownTelemetry = vi.fn(async () => { + calls.push('telemetry'); + }); + const shutdownLogging = vi.fn(async () => { + calls.push('logging'); + }); + const exit = vi.fn(); + const onRegistered = vi.fn(); + + installShutdownSignalHandlers({ process: processLike, shutdownTasks: [shutdownTelemetry, shutdownLogging], exit, onRegistered }); + + expect(processLike.once).toHaveBeenCalledWith('SIGTERM', expect.any(Function)); + expect(processLike.once).toHaveBeenCalledWith('SIGINT', expect.any(Function)); + expect(onRegistered).toHaveBeenCalledOnce(); + expect(isShutdownInProgress()).toBe(false); + + await handlers.get('SIGTERM')!(); + await handlers.get('SIGINT')!(); + + expect(isShutdownInProgress()).toBe(true); + expect(calls).toEqual(['telemetry', 'logging']); + expect(shutdownTelemetry).toHaveBeenCalledTimes(1); + expect(shutdownLogging).toHaveBeenCalledTimes(1); + expect(exit).toHaveBeenCalledTimes(1); + expect(exit).toHaveBeenCalledWith(0); + }); + + test('continues with later shutdown tasks when an earlier task rejects', async () => { + vi.resetModules(); + const { installShutdownSignalHandlers } = await import('@/boot/shutdown-handler.js'); + const handlers = new Map Promise>(); + const processLike = { + once: vi.fn((event: string, handler: () => Promise) => { + handlers.set(event, handler); + return processLike; + }), + }; + const exit = vi.fn(); + const shutdownLogging = vi.fn().mockResolvedValue(undefined); + const shutdownError = new Error('flush failed'); + const consoleError = vi.spyOn(console, 'error').mockImplementation(() => {}); + + try { + installShutdownSignalHandlers({ + process: processLike, + shutdownTasks: [vi.fn().mockRejectedValue(shutdownError), shutdownLogging], + exit, + }); + + await handlers.get('SIGINT')!(); + + expect(exit).toHaveBeenCalledWith(0); + expect(shutdownLogging).toHaveBeenCalledOnce(); + expect(consoleError).toHaveBeenCalledWith('Shutdown task failed:', shutdownError); + } finally { + consoleError.mockRestore(); + } + }); + + test('exits after the shutdown deadline when a task remains pending', async () => { + vi.resetModules(); + vi.useFakeTimers(); + const { installShutdownSignalHandlers } = await import('@/boot/shutdown-handler.js'); + const handlers = new Map Promise>(); + const processLike = { + once: vi.fn((event: string, handler: () => Promise) => { + handlers.set(event, handler); + return processLike; + }), + }; + const exit = vi.fn(); + const consoleError = vi.spyOn(console, 'error').mockImplementation(() => {}); + + try { + installShutdownSignalHandlers({ + process: processLike, + shutdownTasks: [() => new Promise(() => {})], + exit, + }); + + const signalPromise = handlers.get('SIGTERM')!(); + expect(exit).not.toHaveBeenCalled(); + + await vi.advanceTimersByTimeAsync(10_000); + await signalPromise; + + expect(consoleError).toHaveBeenCalledWith('Shutdown tasks timed out after 10000ms.'); + expect(exit).toHaveBeenCalledWith(0); + } finally { + consoleError.mockRestore(); + vi.useRealTimers(); + } + }); +}); diff --git a/packages/backend/test/unit/core/telemetry/adapters/OpenTelemetryAdapter.ts b/packages/backend/test/unit/core/telemetry/adapters/OpenTelemetryAdapter.ts new file mode 100644 index 0000000000..ad6a559250 --- /dev/null +++ b/packages/backend/test/unit/core/telemetry/adapters/OpenTelemetryAdapter.ts @@ -0,0 +1,379 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { beforeEach, describe, expect, test, vi } from 'vitest'; +import { SpanStatusCode } from '@opentelemetry/api'; +import { defaultResource, detectResources, envDetector, resourceFromAttributes } from '@opentelemetry/resources'; +import { ParentBasedSampler, TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-base'; +import { ATTR_SERVICE_INSTANCE_ID, ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions'; +import type { Context, SpanContext } from '@opentelemetry/api'; +import { OpenTelemetryAdapter, createResource, createSampler, getMisskeyProcessRole } from '@/core/telemetry/adapters/OpenTelemetryAdapter.js'; + +const mocks = vi.hoisted(() => { + return { + envOption: { + disableClustering: false, + onlyServer: false, + onlyQueue: false, + }, + isPrimary: false, + }; +}); + +vi.mock('@/env.js', () => ({ + envOption: mocks.envOption, +})); + +vi.mock('node:cluster', () => ({ + default: { + get isPrimary() { + return mocks.isPrimary; + }, + }, +})); + +const samplerDeps = { + ParentBasedSampler, + TraceIdRatioBasedSampler, +}; + +describe('OpenTelemetryAdapter', () => { + test('wraps async work in an active span and ends it after success', async () => { + const span = { + end: vi.fn(), + recordException: vi.fn(), + setStatus: vi.fn(), + }; + const tracer = { + startActiveSpan: vi.fn(async (_name: string, fn: (spanArg: any) => Promise) => fn(span)), + } as any; + const provider = { + shutdown: vi.fn(), + }; + const adapter = new OpenTelemetryAdapter({ + tracer, + provider, + getActiveSpan: () => undefined, + spanStatusCodeError: SpanStatusCode.ERROR, + shutdownTimeout: 10, + }); + + await expect(adapter.startSpan('API: test', async () => 'ok')).resolves.toBe('ok'); + + expect(tracer.startActiveSpan).toHaveBeenCalledWith('API: test', expect.any(Function)); + expect(span.recordException).not.toHaveBeenCalled(); + expect(span.setStatus).not.toHaveBeenCalled(); + expect(span.end).toHaveBeenCalledTimes(1); + }); + + test('records thrown errors on the active span before rethrowing', async () => { + const error = new Error('boom'); + const span = { + end: vi.fn(), + recordException: vi.fn(), + setStatus: vi.fn(), + }; + const tracer = { + startActiveSpan: vi.fn(async (_name: string, fn: (spanArg: any) => Promise) => fn(span)), + } as any; + const adapter = new OpenTelemetryAdapter({ + tracer, + provider: { shutdown: vi.fn() }, + getActiveSpan: () => undefined, + spanStatusCodeError: SpanStatusCode.ERROR, + shutdownTimeout: 10, + }); + + await expect(adapter.startSpan('Queue: test', async () => { + throw error; + })).rejects.toThrow(error); + + expect(span.recordException).toHaveBeenCalledWith(error); + expect(span.setStatus).toHaveBeenCalledWith({ + code: SpanStatusCode.ERROR, + message: error.message, + }); + expect(span.end).toHaveBeenCalledTimes(1); + }); + + test('creates a root worker span linked to the enqueue span by default', () => { + const span = { + end: vi.fn(), + recordException: vi.fn(), + setStatus: vi.fn(), + }; + const rootContext = {} as Context; + const extractedContext = {} as Context; + const sourceSpanContext = { + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 1, + isRemote: true, + } as SpanContext; + const propagation = { + isPropagationApi: true, + inject: vi.fn(), + extract(this: { isPropagationApi: boolean }) { + if (!this.isPropagationApi) throw new Error('lost propagation API receiver'); + return extractedContext; + }, + }; + const tracer = { + startActiveSpan: vi.fn((_name: string, _options: unknown, _context: unknown, fn: (spanArg: typeof span) => string) => fn(span)), + } as any; + const adapter = new OpenTelemetryAdapter({ + tracer, + provider: { shutdown: vi.fn() }, + getActiveSpan: () => undefined, + spanStatusCodeError: SpanStatusCode.ERROR, + shutdownTimeout: 10, + queueTraceContext: { + tracer, + propagation: propagation as any, + trace: { getSpanContext: () => sourceSpanContext }, + getActiveContext: () => rootContext, + rootContext, + mode: 'link', + spanStatusCodeError: SpanStatusCode.ERROR, + }, + }); + + expect(adapter.startSpanWithTraceContext('Queue: Deliver', { + __misskeyTraceContext: { + traceparent: '00-0123456789abcdef0123456789abcdef-0123456789abcdef-01', + }, + }, () => 'ok')).toBe('ok'); + + expect(tracer.startActiveSpan).toHaveBeenCalledWith('Queue: Deliver', { + root: true, + links: [{ context: sourceSpanContext }], + }, rootContext, expect.any(Function)); + expect(span.end).toHaveBeenCalledTimes(1); + }); + + test('returns the active span context for log enrichment', () => { + const adapter = new OpenTelemetryAdapter({ + tracer: { startActiveSpan: vi.fn() }, + provider: { shutdown: vi.fn() }, + getActiveSpan: () => ({ + spanContext: () => ({ + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 0, + }), + } as any), + spanStatusCodeError: SpanStatusCode.ERROR, + shutdownTimeout: 10, + }); + + expect(adapter.getActiveTraceContext()).toEqual({ + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 0, + }); + }); + + test('bridges captureMessage to the active span when one exists', () => { + const activeSpan = { + recordException: vi.fn(), + setStatus: vi.fn(), + }; + const adapter = new OpenTelemetryAdapter({ + tracer: { startActiveSpan: vi.fn() }, + provider: { shutdown: vi.fn() }, + getActiveSpan: () => activeSpan as any, + spanStatusCodeError: SpanStatusCode.ERROR, + shutdownTimeout: 10, + }); + + adapter.captureMessage('Queue failed', { + level: 'error', + extra: { queue: 'deliver' }, + }); + + expect(activeSpan.recordException).toHaveBeenCalledWith(expect.objectContaining({ + message: 'Queue failed', + })); + expect(activeSpan.setStatus).toHaveBeenCalledWith({ + code: SpanStatusCode.ERROR, + message: 'Queue failed', + }); + }); + + test('times out shutdown instead of waiting forever', async () => { + vi.useFakeTimers(); + const adapter = new OpenTelemetryAdapter({ + tracer: { startActiveSpan: vi.fn() }, + provider: { shutdown: vi.fn(() => new Promise(() => {})) }, + getActiveSpan: () => undefined, + spanStatusCodeError: SpanStatusCode.ERROR, + shutdownTimeout: 50, + }); + + const shutdown = adapter.shutdown(); + await vi.advanceTimersByTimeAsync(50); + + await expect(shutdown).resolves.toBeUndefined(); + vi.useRealTimers(); + }); + + test('clears the shutdown timeout timer once provider.shutdown() resolves first', async () => { + vi.useFakeTimers(); + const clearTimeoutSpy = vi.spyOn(global, 'clearTimeout'); + const adapter = new OpenTelemetryAdapter({ + tracer: { startActiveSpan: vi.fn() }, + provider: { shutdown: vi.fn().mockResolvedValue(undefined) }, + getActiveSpan: () => undefined, + spanStatusCodeError: SpanStatusCode.ERROR, + shutdownTimeout: 5000, + }); + + await adapter.shutdown(); + + expect(clearTimeoutSpy).toHaveBeenCalled(); + clearTimeoutSpy.mockRestore(); + vi.useRealTimers(); + }); + + test('captureMessage starts a standalone span to report the error when there is no active span', () => { + const reportSpan = { + end: vi.fn(), + recordException: vi.fn(), + setStatus: vi.fn(), + }; + const tracer = { + startActiveSpan: vi.fn((_name: string, fn: (spanArg: typeof reportSpan) => void) => fn(reportSpan)), + }; + const adapter = new OpenTelemetryAdapter({ + tracer: tracer as any, + provider: { shutdown: vi.fn() }, + getActiveSpan: () => undefined, + spanStatusCodeError: SpanStatusCode.ERROR, + shutdownTimeout: 10, + }); + + adapter.captureMessage('Queue: Deliver failed', { + level: 'error', + extra: { queue: 'deliver' }, + }); + + expect(tracer.startActiveSpan).toHaveBeenCalledWith('captureMessage', expect.any(Function)); + expect(reportSpan.recordException).toHaveBeenCalledWith(expect.objectContaining({ + message: 'Queue: Deliver failed', + })); + expect(reportSpan.setStatus).toHaveBeenCalledWith({ + code: SpanStatusCode.ERROR, + message: 'Queue: Deliver failed', + }); + expect(reportSpan.end).toHaveBeenCalledTimes(1); + }); +}); + +describe('createSampler', () => { + test('accepts sample rates within [0, 1]', () => { + expect(() => createSampler(0, samplerDeps)).not.toThrow(); + expect(() => createSampler(0.5, samplerDeps)).not.toThrow(); + expect(() => createSampler(1, samplerDeps)).not.toThrow(); + }); + + test('rejects sample rates outside [0, 1]', () => { + expect(() => createSampler(-0.1, samplerDeps)).toThrow(); + expect(() => createSampler(1.1, samplerDeps)).toThrow(); + }); + + test('rejects NaN instead of silently disabling sampling', () => { + expect(() => createSampler(Number.NaN, samplerDeps)).toThrow(); + }); + + test('rejects non-number values that pass through YAML as strings', () => { + expect(() => createSampler('0.5' as unknown as number, samplerDeps)).toThrow(); + }); +}); + +describe('createResource', () => { + test('lets explicit config override OTEL resource env, and env override Misskey defaults', () => { + const previousServiceName = process.env['OTEL_SERVICE_NAME']; + const previousResourceAttributes = process.env['OTEL_RESOURCE_ATTRIBUTES']; + process.env['OTEL_SERVICE_NAME'] = 'env-service'; + process.env['OTEL_RESOURCE_ATTRIBUTES'] = [ + 'deployment.environment=staging', + 'misskey.process.role=env-role', + 'service.instance.id=env-instance', + 'env.only=value', + ].join(','); + + try { + const resource = createResource({ + serviceVersion: '2026.1.0', + resourceAttributes: { + [ATTR_SERVICE_NAME]: 'config-service', + 'deployment.environment': 'production', + 'config.only': 'value', + }, + }, { + defaultResource, + resourceFromAttributes, + detectResources, + envDetector, + serviceNameAttribute: ATTR_SERVICE_NAME, + serviceInstanceIdAttribute: ATTR_SERVICE_INSTANCE_ID, + serviceVersionAttribute: ATTR_SERVICE_VERSION, + serviceVersion: '2026.1.0', + }); + + expect(resource.attributes[ATTR_SERVICE_NAME]).toBe('config-service'); + expect(resource.attributes[ATTR_SERVICE_INSTANCE_ID]).toBe('env-instance'); + expect(resource.attributes[ATTR_SERVICE_VERSION]).toBe('2026.1.0'); + expect(resource.attributes['deployment.environment']).toBe('production'); + expect(resource.attributes['misskey.process.role']).toBe('env-role'); + expect(resource.attributes['env.only']).toBe('value'); + expect(resource.attributes['config.only']).toBe('value'); + } finally { + if (previousServiceName == null) { + delete process.env['OTEL_SERVICE_NAME']; + } else { + process.env['OTEL_SERVICE_NAME'] = previousServiceName; + } + if (previousResourceAttributes == null) { + delete process.env['OTEL_RESOURCE_ATTRIBUTES']; + } else { + process.env['OTEL_RESOURCE_ATTRIBUTES'] = previousResourceAttributes; + } + } + }); +}); + +describe('getMisskeyProcessRole', () => { + beforeEach(() => { + mocks.envOption.disableClustering = false; + mocks.envOption.onlyServer = false; + mocks.envOption.onlyQueue = false; + mocks.isPrimary = false; + }); + + test('labels non-clustered onlyServer as primary-server', () => { + mocks.envOption.disableClustering = true; + mocks.envOption.onlyServer = true; + expect(getMisskeyProcessRole()).toBe('primary-server'); + }); + + test('labels clustered primary with onlyServer as fork-only', () => { + mocks.isPrimary = true; + mocks.envOption.onlyServer = true; + expect(getMisskeyProcessRole()).toBe('fork-only'); + }); + + test('labels clustered worker running the HTTP server (onlyServer) as worker-server, not worker-queue', () => { + mocks.isPrimary = false; + mocks.envOption.onlyServer = true; + expect(getMisskeyProcessRole()).toBe('worker-server'); + }); + + test('labels clustered worker without onlyServer as worker-queue', () => { + mocks.isPrimary = false; + mocks.envOption.onlyServer = false; + expect(getMisskeyProcessRole()).toBe('worker-queue'); + }); +}); diff --git a/packages/backend/test/unit/core/telemetry/adapters/SentryTelemetryAdapter.ts b/packages/backend/test/unit/core/telemetry/adapters/SentryTelemetryAdapter.ts new file mode 100644 index 0000000000..9423aebe51 --- /dev/null +++ b/packages/backend/test/unit/core/telemetry/adapters/SentryTelemetryAdapter.ts @@ -0,0 +1,288 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import { SentryTelemetryAdapter, buildSentryIntegrations, buildSentryNodeOptions, buildSentryOtlpInitOptions } from '@/core/telemetry/adapters/SentryTelemetryAdapter.js'; + +type TestIntegration = Parameters>[0][number]; + +function testIntegration(name: string): TestIntegration { + return { name }; +} + +describe('SentryTelemetryAdapter', () => { + test('removes disabled integrations from Sentry defaults', () => { + const integrations = buildSentryIntegrations({ + disabledIntegrations: ['Postgres'], + enableNodeProfiling: false, + }); + + const result = integrations([ + testIntegration('Http'), + testIntegration('Postgres'), + testIntegration('Redis'), + ]); + + expect(result.map((integration: TestIntegration) => integration.name)).toEqual(['Http', 'Redis']); + }); + + test('keeps profiling integration when enabled', () => { + const integrations = buildSentryIntegrations({ + disabledIntegrations: [], + enableNodeProfiling: true, + nodeProfilingIntegration: () => testIntegration('ProfilingIntegration'), + }); + + const result = integrations([ + testIntegration('Http'), + ]); + + expect(result.map((integration: TestIntegration) => integration.name)).toEqual(['Http', 'ProfilingIntegration']); + }); + + test('warns about unknown disabled integration names without removing defaults', () => { + const warn = vi.fn(); + const integrations = buildSentryIntegrations({ + disabledIntegrations: ['Unknown'], + enableNodeProfiling: false, + warn, + }); + + const result = integrations([ + testIntegration('Http'), + ]); + + expect(result.map((integration: TestIntegration) => integration.name)).toEqual(['Http']); + expect(warn).toHaveBeenCalledWith('Unknown Sentry integration configured in sentryForBackend.disabledIntegrations: Unknown'); + }); + + test('disables outbound trace propagation by default', () => { + const options = buildSentryNodeOptions({ + enableNodeProfiling: false, + options: {}, + }); + + expect(options.tracePropagationTargets).toEqual([]); + }); + + test('allows explicit tracePropagationTargets to override the default', () => { + const options = buildSentryNodeOptions({ + enableNodeProfiling: false, + options: { + tracePropagationTargets: ['^https://internal\\.example/'], + }, + }); + + expect(options.tracePropagationTargets).toEqual(['^https://internal\\.example/']); + }); + + test('builds Sentry options that export spans to both Sentry and OTLP', () => { + const existingProcessor = { name: 'existingProcessor' }; + const otlpProcessor = { name: 'otlpProcessor' }; + + const result = buildSentryOtlpInitOptions({ + sentryConfig: { + enableNodeProfiling: false, + disabledIntegrations: ['Redis'], + options: { + openTelemetrySpanProcessors: [existingProcessor as any], + tracesSampleRate: 0.25, + }, + }, + otelConfig: { serviceVersion: '2026.1.0' }, + otlpProcessor, + }); + + expect(result.tracesSampleRate).toBe(0.25); + expect(result.openTelemetrySpanProcessors).toEqual([existingProcessor, otlpProcessor]); + // OTel併存時もremoteへtrace headerを漏らさないデフォルトはSentry単体時と揃える。 + expect(result.tracePropagationTargets).toEqual([]); + expect((result.integrations as any)([ + testIntegration('Http'), + testIntegration('Redis'), + testIntegration('Postgres'), + ]).map((integration: TestIntegration) => integration.name)).toEqual(['Http', 'Postgres']); + }); + + test('does not disable Sentry trace propagation when explicitly enabled for OTel coexistence', () => { + const result = buildSentryOtlpInitOptions({ + sentryConfig: { + enableNodeProfiling: false, + options: {}, + }, + otelConfig: { + serviceVersion: '2026.1.0', + propagateTraceToRemote: true, + }, + otlpProcessor: { name: 'otlpProcessor' }, + }); + + expect(result.tracePropagationTargets).toBeUndefined(); + }); + + test('honors explicit tracePropagationTargets for OTel coexistence even without propagateTraceToRemote', () => { + const result = buildSentryOtlpInitOptions({ + sentryConfig: { + enableNodeProfiling: false, + options: { + tracePropagationTargets: ['^https://internal\\.example/'], + }, + }, + otelConfig: { serviceVersion: '2026.1.0' }, + otlpProcessor: { name: 'otlpProcessor' }, + }); + + expect(result.tracePropagationTargets).toEqual(['^https://internal\\.example/']); + }); + + test('warns when OTel-only options are ignored in Sentry coexistence mode', () => { + const warn = vi.fn(); + + buildSentryOtlpInitOptions({ + sentryConfig: { + enableNodeProfiling: false, + options: {}, + }, + otelConfig: { + serviceVersion: '2026.1.0', + sampleRate: 0.25, + resourceAttributes: { + 'deployment.environment': 'production', + }, + }, + otlpProcessor: { name: 'otlpProcessor' }, + warn, + }); + + expect(warn).toHaveBeenCalledWith(expect.stringContaining('otelForBackend.sampleRate is ignored')); + expect(warn).toHaveBeenCalledWith(expect.stringContaining('otelForBackend.resourceAttributes is ignored')); + }); +}); + +describe('SentryTelemetryAdapter trace context', () => { + test('returns the active span context for log enrichment', async () => { + const activeSpan = { + spanContext: () => ({ + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 0, + }), + }; + vi.doMock('@sentry/node', () => ({ + init: vi.fn(), + close: vi.fn(), + getActiveSpan: vi.fn(() => activeSpan), + })); + vi.doMock('@sentry/profiling-node', () => ({ + nodeProfilingIntegration: vi.fn(), + })); + + const adapter = await SentryTelemetryAdapter.create({ + enableNodeProfiling: false, + options: {}, + }); + + expect(adapter.getActiveTraceContext()).toEqual({ + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 0, + }); + + vi.doUnmock('@sentry/node'); + vi.doUnmock('@sentry/profiling-node'); + }); +}); + +describe('SentryTelemetryAdapter.shutdown', () => { + test('bounds Sentry.close() with a timeout so a stuck transport cannot hang process shutdown', async () => { + const close = vi.fn().mockResolvedValue(true); + vi.doMock('@sentry/node', () => ({ + init: vi.fn(), + close, + })); + vi.doMock('@sentry/profiling-node', () => ({ + nodeProfilingIntegration: vi.fn(), + })); + + const adapter = await SentryTelemetryAdapter.create({ + enableNodeProfiling: false, + options: {}, + }); + await adapter.shutdown(); + + expect(close).toHaveBeenCalledTimes(1); + expect(close).toHaveBeenCalledWith(expect.any(Number)); + expect(close.mock.calls[0][0]).toBeGreaterThan(0); + + vi.doUnmock('@sentry/node'); + vi.doUnmock('@sentry/profiling-node'); + }); +}); + +describe('SentryTelemetryAdapter.createWithOtlpExport', () => { + test('registers the OTel diag logger before creating the OTLP exporter', async () => { + const init = vi.fn(); + const close = vi.fn(); + const setLogger = vi.fn(); + const nodeProfilingIntegration = vi.fn(); + const BatchSpanProcessor = vi.fn(function (this: { exporter: unknown }, exporter: unknown) { + this.exporter = exporter; + }); + const OTLPTraceExporter = vi.fn(function (this: { options: unknown }, options: unknown) { + this.options = options; + }); + + vi.doMock('@sentry/node', () => ({ + init, + close, + })); + vi.doMock('@sentry/profiling-node', () => ({ + nodeProfilingIntegration, + })); + vi.doMock('@opentelemetry/api', () => ({ + context: { active: vi.fn() }, + diag: { setLogger }, + DiagLogLevel: { WARN: 50 }, + propagation: { inject: vi.fn(), extract: vi.fn() }, + ROOT_CONTEXT: {}, + SpanStatusCode: { ERROR: 2 }, + trace: { getTracer: vi.fn(), getSpanContext: vi.fn() }, + })); + vi.doMock('@opentelemetry/sdk-trace-base', () => ({ + BatchSpanProcessor, + })); + vi.doMock('@opentelemetry/exporter-trace-otlp-proto', () => ({ + OTLPTraceExporter, + })); + + await SentryTelemetryAdapter.createWithOtlpExport({ + enableNodeProfiling: false, + options: {}, + }, { + serviceVersion: '2026.1.0', + endpoint: 'http://collector:4318/v1/traces', + }); + + expect(setLogger).toHaveBeenCalledWith(expect.objectContaining({ + error: expect.any(Function), + warn: expect.any(Function), + }), { + logLevel: 50, + suppressOverrideMessage: true, + }); + expect(OTLPTraceExporter).toHaveBeenCalledWith({ + url: 'http://collector:4318/v1/traces', + }); + expect(init).toHaveBeenCalledWith(expect.objectContaining({ + openTelemetrySpanProcessors: [expect.any(Object)], + })); + + vi.doUnmock('@sentry/node'); + vi.doUnmock('@sentry/profiling-node'); + vi.doUnmock('@opentelemetry/api'); + vi.doUnmock('@opentelemetry/sdk-trace-base'); + vi.doUnmock('@opentelemetry/exporter-trace-otlp-proto'); + }); +}); diff --git a/packages/backend/test/unit/core/telemetry/http-client-instrumentation.ts b/packages/backend/test/unit/core/telemetry/http-client-instrumentation.ts new file mode 100644 index 0000000000..e05f4407e4 --- /dev/null +++ b/packages/backend/test/unit/core/telemetry/http-client-instrumentation.ts @@ -0,0 +1,108 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import { SpanKind, SpanStatusCode } from '@opentelemetry/api'; +import { createHttpClientInstrumentation } from '@/core/telemetry/http-client-instrumentation.js'; + +function request() { + return { + method: 'POST', + protocol: 'https:', + path: '/inbox?token=secret', + host: 'remote.example', + getHeader: vi.fn((name: string) => name === 'host' ? 'user:password@remote.example:8443' : undefined), + }; +} + +describe('http-client-instrumentation', () => { + test('creates and completes a sanitized CLIENT span from diagnostics channels', () => { + const listeners = new Map void>(); + const span = { + end: vi.fn(), + recordException: vi.fn(), + setAttribute: vi.fn(), + setStatus: vi.fn(), + }; + const tracer = { startSpan: vi.fn(() => span) } as any; + const unsubscribe = createHttpClientInstrumentation({ + tracer, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + subscribe: (name, listener) => { + listeners.set(name, listener); + return () => listeners.delete(name); + }, + }); + const clientRequest = request(); + + listeners.get('http.client.request.created')!({ request: clientRequest }); + listeners.get('http.client.response.finish')!({ + request: clientRequest, + response: { statusCode: 201, httpVersion: '1.1' }, + }); + + expect(tracer.startSpan).toHaveBeenCalledWith('POST', { + kind: SpanKind.CLIENT, + attributes: { + 'http.request.method': 'POST', + 'url.full': 'https://remote.example:8443/inbox', + 'server.address': 'remote.example', + 'server.port': 8443, + }, + }); + expect(span.setAttribute).toHaveBeenCalledWith('http.response.status_code', 201); + expect(span.setAttribute).toHaveBeenCalledWith('network.protocol.version', '1.1'); + expect(span.end).toHaveBeenCalledTimes(1); + unsubscribe(); + expect(listeners).toHaveLength(0); + }); + + test('records a request error and ends the span once', () => { + const listeners = new Map void>(); + const span = { end: vi.fn(), recordException: vi.fn(), setAttribute: vi.fn(), setStatus: vi.fn() }; + const error = Object.assign(new Error('connection refused'), { code: 'ECONNREFUSED' }); + const clientRequest = request(); + createHttpClientInstrumentation({ + tracer: { startSpan: vi.fn(() => span) } as any, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + subscribe: (name, listener) => { + listeners.set(name, listener); + return () => listeners.delete(name); + }, + }); + + listeners.get('http.client.request.created')!({ request: clientRequest }); + listeners.get('http.client.request.error')!({ request: clientRequest, error }); + listeners.get('http.client.response.finish')!({ request: clientRequest, response: { statusCode: 200 } }); + + expect(span.recordException).toHaveBeenCalledWith(error); + expect(span.setAttribute).toHaveBeenCalledWith('error.type', 'ECONNREFUSED'); + expect(span.setStatus).toHaveBeenCalledWith({ code: SpanStatusCode.ERROR }); + expect(span.end).toHaveBeenCalledTimes(1); + }); + + test('records the response status code as error.type for an error response', () => { + const listeners = new Map void>(); + const span = { end: vi.fn(), recordException: vi.fn(), setAttribute: vi.fn(), setStatus: vi.fn() }; + const clientRequest = request(); + createHttpClientInstrumentation({ + tracer: { startSpan: vi.fn(() => span) } as any, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + subscribe: (name, listener) => { + listeners.set(name, listener); + return () => listeners.delete(name); + }, + }); + + listeners.get('http.client.request.created')!({ request: clientRequest }); + listeners.get('http.client.response.finish')!({ request: clientRequest, response: { statusCode: 502 } }); + + expect(span.setAttribute).toHaveBeenCalledWith('error.type', '502'); + expect(span.setStatus).toHaveBeenCalledWith({ code: SpanStatusCode.ERROR }); + }); +}); diff --git a/packages/backend/test/unit/core/telemetry/queue-instrumentation.ts b/packages/backend/test/unit/core/telemetry/queue-instrumentation.ts new file mode 100644 index 0000000000..ac73df3c13 --- /dev/null +++ b/packages/backend/test/unit/core/telemetry/queue-instrumentation.ts @@ -0,0 +1,57 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { beforeEach, describe, expect, test, vi } from 'vitest'; +import type * as Bull from 'bullmq'; +import { instrumentQueue } from '@/core/telemetry/queue-instrumentation.js'; + +const mocks = vi.hoisted(() => ({ + injectTraceContext: vi.fn((carrier: Record) => { + carrier['traceparent'] = '00-0123456789abcdef0123456789abcdef-0123456789abcdef-01'; + }), +})); + +vi.mock('@/core/telemetry/telemetry-registry.js', () => ({ + injectTraceContext: mocks.injectTraceContext, +})); + +describe('queue-instrumentation', () => { + beforeEach(() => { + mocks.injectTraceContext.mockClear(); + }); + + test('injects the active trace context for add()', () => { + const add = vi.fn(); + const queue = instrumentQueue({ add, addBulk: vi.fn() } as unknown as Bull.Queue<{ noteId: string }>); + const data = { noteId: '9d6b9a65-46c9-4e1b-a640-9589693893c9' }; + + queue.add('endedPollNotification', data); + + expect(mocks.injectTraceContext).toHaveBeenCalledTimes(1); + expect(data).toMatchObject({ + __misskeyTraceContext: { + traceparent: '00-0123456789abcdef0123456789abcdef-0123456789abcdef-01', + }, + }); + expect(add).toHaveBeenCalledWith('endedPollNotification', data, undefined); + }); + + test('injects every job passed to addBulk()', () => { + const addBulk = vi.fn(); + const queue = instrumentQueue({ add: vi.fn(), addBulk } as unknown as Bull.Queue<{ to: string }>); + const jobs = [ + { name: 'deliver', data: { to: 'https://remote.example/inbox' } }, + { name: 'deliver', data: { to: 'https://remote2.example/inbox' } }, + ]; + + queue.addBulk(jobs); + + expect(mocks.injectTraceContext).toHaveBeenCalledTimes(2); + expect(jobs).toEqual(expect.arrayContaining([ + expect.objectContaining({ data: expect.objectContaining({ __misskeyTraceContext: expect.any(Object) }) }), + ])); + expect(addBulk).toHaveBeenCalledWith(jobs); + }); +}); diff --git a/packages/backend/test/unit/core/telemetry/queue-trace-context.ts b/packages/backend/test/unit/core/telemetry/queue-trace-context.ts new file mode 100644 index 0000000000..3ecf4fe270 --- /dev/null +++ b/packages/backend/test/unit/core/telemetry/queue-trace-context.ts @@ -0,0 +1,132 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import type { Context, SpanContext } from '@opentelemetry/api'; +import { getQueueSpanContext, getQueueTraceContextMode, injectActiveTraceContext, injectQueueTraceContext } from '@/core/telemetry/queue-trace-context.js'; + +const rootContext = {} as Context; +const extractedContext = {} as Context; +const sourceSpanContext: SpanContext = { + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 1, + isRemote: true, +}; + +function jobData() { + return { + name: 'deliver', + __misskeyTraceContext: { + traceparent: '00-0123456789abcdef0123456789abcdef-0123456789abcdef-01', + }, + }; +} + +describe('queue-trace-context', () => { + test('stores only a non-empty carrier in the job data', () => { + const data = { noteId: '9d6b9a65-46c9-4e1b-a640-9589693893c9' }; + + injectQueueTraceContext(data, carrier => { + carrier['traceparent'] = '00-0123456789abcdef0123456789abcdef-0123456789abcdef-01'; + }); + + expect(data).toMatchObject({ + __misskeyTraceContext: { + traceparent: '00-0123456789abcdef0123456789abcdef-0123456789abcdef-01', + }, + }); + }); + + test('does not store an empty carrier when no active trace exists', () => { + const data = { noteId: '9d6b9a65-46c9-4e1b-a640-9589693893c9' }; + + injectQueueTraceContext(data, () => {}); + + expect(data).not.toHaveProperty('__misskeyTraceContext'); + }); + + test('ignores non-object job data', () => { + const inject = vi.fn(); + + injectQueueTraceContext(null, inject); + injectQueueTraceContext('not a job object', inject); + + expect(inject).not.toHaveBeenCalled(); + }); + + test('injects the active context with the configured propagator', () => { + const activeContext = {} as Context; + const carrier = {}; + const inject = vi.fn(); + + injectActiveTraceContext({ + tracer: { startActiveSpan: vi.fn() } as any, + propagation: { inject, extract: vi.fn() } as any, + trace: { getSpanContext: vi.fn() }, + getActiveContext: () => activeContext, + rootContext, + mode: 'link', + spanStatusCodeError: 2 as any, + }, carrier); + + expect(inject).toHaveBeenCalledWith(activeContext, carrier); + }); + + test('starts a new root trace with a link by default', () => { + const extract = vi.fn(() => extractedContext); + const getSpanContext = vi.fn(() => sourceSpanContext); + + const result = getQueueSpanContext(jobData(), { + rootContext, + propagation: { inject: vi.fn(), extract }, + trace: { getSpanContext }, + mode: 'link', + }); + + expect(extract).toHaveBeenCalledWith(rootContext, jobData().__misskeyTraceContext); + expect(result).toEqual({ + options: { + root: true, + links: [{ context: sourceSpanContext }], + }, + parentContext: rootContext, + }); + }); + + test('uses the extracted context as the parent when parent mode is selected', () => { + const result = getQueueSpanContext(jobData(), { + rootContext, + propagation: { inject: vi.fn(), extract: () => extractedContext }, + trace: { getSpanContext: () => sourceSpanContext }, + mode: 'parent', + }); + + expect(result).toEqual({ + options: {}, + parentContext: extractedContext, + }); + }); + + test('ignores malformed or missing carriers', () => { + const extract = vi.fn(() => extractedContext); + const deps = { + rootContext, + propagation: { inject: vi.fn(), extract }, + trace: { getSpanContext: () => sourceSpanContext }, + mode: 'link' as const, + }; + + expect(getQueueSpanContext({}, deps)).toBeUndefined(); + expect(getQueueSpanContext({ __misskeyTraceContext: { traceparent: 1 } }, deps)).toBeUndefined(); + expect(extract).not.toHaveBeenCalled(); + }); + + test('defaults to link mode and rejects invalid configuration', () => { + expect(getQueueTraceContextMode(undefined)).toBe('link'); + expect(getQueueTraceContextMode('parent')).toBe('parent'); + expect(() => getQueueTraceContextMode('children')).toThrow('otelForBackend.jobTraceContextMode'); + }); +}); diff --git a/packages/backend/test/unit/logging/BootstrapConsoleBackend.ts b/packages/backend/test/unit/logging/BootstrapConsoleBackend.ts new file mode 100644 index 0000000000..43545a0693 --- /dev/null +++ b/packages/backend/test/unit/logging/BootstrapConsoleBackend.ts @@ -0,0 +1,54 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import type { LogRecord } from '@/logging/types.js'; +import { BootstrapConsoleBackend } from '@/logging/BootstrapConsoleBackend.js'; + +function createRecord(overrides: Partial = {}): LogRecord { + return { + level: 'error', + message: 'configuration failed', + context: [{ name: 'core' }, { name: 'boot' }], + timestamp: '2025-01-02T03:04:05.678Z', + loggerName: 'core.boot', + processId: 1234, + isPrimary: true, + workerId: null, + ...overrides, + }; +} + +describe('BootstrapConsoleBackend', () => { + test('writes a minimal line with legacy data', () => { + const output = vi.fn(); + const backend = new BootstrapConsoleBackend({ output }); + const data = { detail: 'failed' }; + + backend.write(createRecord({ compatibility: { data } })); + + expect(output).toHaveBeenCalledWith('2025-01-02T03:04:05.678Z ERROR *\t[core.boot]\tconfiguration failed', data); + }); + + test('writes structured details without depending on Pretty formatting', () => { + const output = vi.fn(); + const backend = new BootstrapConsoleBackend({ output }); + + backend.write(createRecord({ + eventName: 'config.load.failed', + attributes: { path: '.config/default.yml' }, + error: { type: 'Error', message: 'not found' }, + })); + + expect(output).toHaveBeenCalledWith( + '2025-01-02T03:04:05.678Z ERROR *\t[core.boot]\tconfiguration failed', + { + eventName: 'config.load.failed', + attributes: { path: '.config/default.yml' }, + error: { type: 'Error', message: 'not found' }, + }, + ); + }); +}); diff --git a/packages/backend/test/unit/logging/JsonConsoleBackend.ts b/packages/backend/test/unit/logging/JsonConsoleBackend.ts new file mode 100644 index 0000000000..67ca71fcc3 --- /dev/null +++ b/packages/backend/test/unit/logging/JsonConsoleBackend.ts @@ -0,0 +1,135 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import type { AccessLogRecord, LogRecord } from '@/logging/types.js'; +import { JsonConsoleBackend } from '@/logging/JsonConsoleBackend.js'; + +/** JSON形式のテストで使う共通のログを作成します。 */ +function createRecord(overrides: Partial = {}): LogRecord { + return { + level: 'error', + message: 'delivery failed', + context: [{ name: 'queue' }, { name: 'deliver' }], + timestamp: '2025-01-02T03:04:05.678Z', + loggerName: 'queue.deliver', + processId: 1234, + isPrimary: false, + workerId: 7, + ...overrides, + }; +} + +describe('JsonConsoleBackend', () => { + test('writes Access log as a separate one-line schema', () => { + const output = vi.fn<(line: string) => void>(); + const backend = new JsonConsoleBackend({ output }); + const record: AccessLogRecord = { + type: 'access', + timestamp: '2025-01-02T03:04:05.678Z', + method: 'GET', + route: '/items/:id', + statusCode: 500, + durationMs: 12.5, + responseSizeBytes: null, + errorType: 'TypeError', + requestBody: { token: '[REDACTED]' }, + processId: 1234, + isPrimary: true, + workerId: null, + traceId: 'trace', + spanId: 'span', + traceFlags: 1, + }; + + backend.writeAccess(record); + + expect(JSON.parse(output.mock.calls[0][0])).toEqual({ + type: 'access', + timestamp: '2025-01-02T03:04:05.678Z', + method: 'GET', + route: '/items/:id', + statusCode: 500, + durationMs: 12.5, + responseSizeBytes: null, + errorType: 'TypeError', + requestBody: { token: '[REDACTED]' }, + processId: 1234, + isPrimary: true, + workerId: null, + trace_id: 'trace', + span_id: 'span', + trace_flags: 1, + }); + }); + + test('writes a stable one-line JSON record', () => { + const output = vi.fn<(line: string) => void>(); + const backend = new JsonConsoleBackend({ output }); + + backend.write(createRecord({ + eventName: 'queue.job.failed', + attributes: { jobId: '123', attempt: 2 }, + error: { type: 'TypeError', message: 'broken', stack: 'stack' }, + })); + + expect(output).toHaveBeenCalledOnce(); + expect(output.mock.calls[0][0]).toMatchInlineSnapshot(`"{\"timestamp\":\"2025-01-02T03:04:05.678Z\",\"level\":\"error\",\"message\":\"delivery failed\",\"loggerName\":\"queue.deliver\",\"eventName\":\"queue.job.failed\",\"attributes\":{\"jobId\":\"123\",\"attempt\":2},\"error\":{\"type\":\"TypeError\",\"message\":\"broken\",\"stack\":\"stack\"},\"processId\":1234,\"isPrimary\":false,\"workerId\":7}"`); + }); + + test('omits pretty-only compatibility data and context colors', () => { + const output = vi.fn<(line: string) => void>(); + const backend = new JsonConsoleBackend({ output }); + const record = createRecord({ + context: [{ name: 'queue', color: 'red' }], + compatibility: { + legacyLevel: 'success', + important: true, + data: { secret: 'must not be written' }, + }, + }); + + backend.write(record); + + expect(JSON.parse(output.mock.calls[0][0])).toEqual({ + timestamp: '2025-01-02T03:04:05.678Z', + level: 'error', + message: 'delivery failed', + loggerName: 'queue.deliver', + processId: 1234, + isPrimary: false, + workerId: 7, + }); + }); + + test('writes trace context using the standard JSON field names', () => { + const output = vi.fn<(line: string) => void>(); + const backend = new JsonConsoleBackend({ output }); + + backend.write(createRecord({ + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 0, + })); + + expect(JSON.parse(output.mock.calls[0][0])).toMatchObject({ + trace_id: '0123456789abcdef0123456789abcdef', + span_id: '0123456789abcdef', + trace_flags: 0, + }); + }); + + test('escapes newlines so each record remains one physical line', () => { + const output = vi.fn<(line: string) => void>(); + const backend = new JsonConsoleBackend({ output }); + const message = 'first line\nsecond line "quoted"'; + + backend.write(createRecord({ message })); + + const line = output.mock.calls[0][0]; + expect(line).not.toContain('\n'); + expect(JSON.parse(line).message).toBe(message); + }); +}); diff --git a/packages/backend/test/unit/logging/LogManager.ts b/packages/backend/test/unit/logging/LogManager.ts new file mode 100644 index 0000000000..94880856a1 --- /dev/null +++ b/packages/backend/test/unit/logging/LogManager.ts @@ -0,0 +1,412 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import type { LogBackend } from '@/logging/LogBackend.js'; +import type { AccessLogRecord, AccessLogRecordInput, LogRecordInput, LogTraceContext } from '@/logging/types.js'; +import { LogManager } from '@/logging/LogManager.js'; + +/** テストで使う最小構成のログ入力を作成します。 */ +function createInput(level: LogRecordInput['level'] = 'info'): LogRecordInput { + return { + level, + message: 'message', + context: [ + { name: 'queue', color: 'red' }, + { name: 'deliver', color: 'blue' }, + ], + }; +} + +/** 実行環境を固定したLogManagerと、出力確認用の関数を作成します。 */ +function createManager(options: { + quiet?: boolean; + verbose?: boolean; + nodeEnv?: string; + isPrimary?: boolean; + workerId?: number | null; + normalizationProfile?: 'standard' | 'detailed'; + configuration?: { + level?: 'debug' | 'info' | 'warn' | 'error' | 'fatal' | 'off'; + domains?: Record; + access?: { + statusClasses?: ('2xx' | '3xx' | '4xx' | '5xx')[]; + bodies?: { request?: boolean; response?: boolean; maxBytes?: number }; + }; + }; +} = {}) { + const write = vi.fn(); + const writeAccess = vi.fn<(record: AccessLogRecord) => void>(); + const manager = new LogManager({ write, writeAccess }, { + now: () => new Date('2025-01-02T03:04:05.678Z'), + getProcessInfo: () => ({ + processId: 1234, + isPrimary: options.isPrimary ?? true, + workerId: options.workerId ?? null, + }), + isQuiet: () => options.quiet ?? false, + isVerbose: () => options.verbose ?? false, + getNodeEnv: () => options.nodeEnv ?? 'development', + }, { + normalizationProfile: options.normalizationProfile, + }); + if (options.configuration) manager.configure(options.configuration); + + return { manager, write, writeAccess }; +} + +describe('LogManager', () => { + test('adds logger and process metadata while preserving root-to-leaf context order', () => { + const { manager, write } = createManager(); + const input = createInput(); + + manager.write(input); + + expect(write).toHaveBeenCalledOnce(); + expect(write).toHaveBeenCalledWith({ + ...input, + context: [ + { name: 'queue', color: 'red' }, + { name: 'deliver', color: 'blue' }, + ], + timestamp: '2025-01-02T03:04:05.678Z', + loggerName: 'queue.deliver', + processId: 1234, + isPrimary: true, + workerId: null, + }); + expect(write.mock.calls[0][0].context).not.toBe(input.context); + }); + + test('records worker process metadata', () => { + const { manager, write } = createManager({ isPrimary: false, workerId: 7 }); + + manager.write(createInput()); + + expect(write.mock.calls[0][0]).toMatchObject({ + processId: 1234, + isPrimary: false, + workerId: 7, + }); + }); + + test('adds active trace context to the record after filtering', () => { + const { manager, write } = createManager(); + const traceContext: LogTraceContext = { + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 0, + }; + const provider = vi.fn(() => traceContext); + manager.setTraceContextProvider(provider); + + manager.write(createInput()); + + expect(provider).toHaveBeenCalledOnce(); + expect(write.mock.calls[0][0]).toMatchObject(traceContext); + }); + + test('does not get trace context for logs that will not be written', () => { + const provider = vi.fn(() => ({ + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 1, + })); + const filtered = createManager({ configuration: { level: 'warn' } }); + filtered.manager.setTraceContextProvider(provider); + filtered.manager.write(createInput('info')); + + const quiet = createManager({ quiet: true }); + quiet.manager.setTraceContextProvider(provider); + quiet.manager.write(createInput('error')); + + expect(provider).not.toHaveBeenCalled(); + }); + + test('does not call the backend in quiet mode', () => { + const { manager, write } = createManager({ quiet: true, verbose: true }); + + manager.write(createInput('fatal')); + manager.write(createInput('debug')); + + expect(write).not.toHaveBeenCalled(); + }); + + test('writes debug logs outside production', () => { + const { manager, write } = createManager({ nodeEnv: 'development' }); + + manager.write(createInput('debug')); + + expect(write).toHaveBeenCalledOnce(); + }); + + test('suppresses debug logs in production by default', () => { + const { manager, write } = createManager({ nodeEnv: 'production' }); + + manager.write(createInput('debug')); + + expect(write).not.toHaveBeenCalled(); + }); + + test('writes debug logs in verbose production mode', () => { + const { manager, write } = createManager({ nodeEnv: 'production', verbose: true }); + + manager.write(createInput('debug')); + + expect(write).toHaveBeenCalledOnce(); + }); + + test('applies the configured global level', () => { + const { manager, write } = createManager({ configuration: { level: 'warn' } }); + + manager.write(createInput('info')); + manager.write(createInput('warn')); + + expect(write).toHaveBeenCalledOnce(); + expect(write.mock.calls[0][0].level).toBe('warn'); + }); + + test('uses the longest matching domain and supports child re-enablement', () => { + const { manager, write } = createManager({ + configuration: { + level: 'info', + domains: { + queue: 'off', + 'queue.deliver': 'debug', + }, + }, + }); + + manager.write({ ...createInput('info'), context: [{ name: 'queue' }, { name: 'inbox' }] }); + manager.write({ ...createInput('debug'), context: [{ name: 'queue' }, { name: 'deliver' }] }); + manager.write({ ...createInput('debug'), context: [{ name: 'queueing' }] }); + + expect(write).toHaveBeenCalledOnce(); + expect(write.mock.calls[0][0].loggerName).toBe('queue.deliver'); + }); + + test('lowers configured levels to debug in verbose mode', () => { + const { manager, write } = createManager({ + nodeEnv: 'production', + verbose: true, + configuration: { level: 'info' }, + }); + + manager.write(createInput('debug')); + + expect(write).toHaveBeenCalledOnce(); + }); + + test('keeps explicit off levels disabled in verbose mode', () => { + const { manager, write } = createManager({ + verbose: true, + configuration: { + level: 'off', + domains: { + queue: 'off', + 'queue.deliver': 'warn', + }, + }, + }); + + manager.write({ ...createInput('fatal'), context: [{ name: 'system' }] }); + manager.write({ ...createInput('debug'), context: [{ name: 'queue' }] }); + manager.write({ ...createInput('debug'), context: [{ name: 'queue' }, { name: 'deliver' }] }); + + expect(write).toHaveBeenCalledOnce(); + expect(write.mock.calls[0][0].loggerName).toBe('queue.deliver'); + }); + + test('rejects invalid logging configuration', () => { + const { manager } = createManager(); + + expect(() => manager.configure({ domains: null })).not.toThrow(); + expect(() => manager.configure({ level: 'notice' as never })).toThrow('logging.level'); + expect(() => manager.configure({ domains: { queue: 'notice' as never } })).toThrow('logging.domains.queue'); + expect(() => manager.configure({ domains: { 'queue.': 'info' } })).toThrow('invalid domain name'); + expect(() => manager.configure({ access: { statusClasses: ['1xx' as never] } })).toThrow('statusClasses'); + expect(() => manager.configure({ access: { statusClasses: '4xx' as never } })).toThrow('statusClasses'); + expect(() => manager.configure({ access: { bodies: null as never } })).toThrow('bodies'); + expect(() => manager.configure({ access: { bodies: { maxBytes: 0 } } })).toThrow('maxBytes'); + expect(() => manager.configure({ access: { bodies: { maxBytes: 1.5 } } })).toThrow('maxBytes'); + expect(() => manager.configure({ access: { bodies: { maxBytes: 128 * 1024 + 1 } } })).toThrow('maxBytes'); + }); + + test('filters Access log by status class and keeps it independent from application level', () => { + const { manager, write, writeAccess } = createManager({ configuration: { level: 'off', access: { statusClasses: ['4xx', '5xx'] } } }); + const input: AccessLogRecordInput = { + method: 'GET', + route: '/items/:id', + statusCode: 200, + durationMs: 1, + responseSizeBytes: 10, + }; + + manager.writeAccess(input); + manager.writeAccess({ ...input, statusCode: 404 }); + manager.writeAccess({ ...input, statusCode: 500 }); + + expect(write).not.toHaveBeenCalled(); + expect(writeAccess).toHaveBeenCalledTimes(2); + expect(writeAccess.mock.calls[0][0]).toMatchObject({ type: 'access', statusCode: 404, processId: 1234, workerId: null }); + }); + + test('normalizes Access log bodies and keeps captured Trace Context', () => { + const { manager, writeAccess } = createManager({ + configuration: { + access: { + statusClasses: ['2xx'], + bodies: { request: true, response: true, maxBytes: 1024 }, + }, + }, + }); + const input: AccessLogRecordInput = { + method: 'POST', + route: '/body', + statusCode: 200, + durationMs: 2, + responseSizeBytes: 20, + requestBody: { i: 'secret', nested: { password: 'secret' } }, + responseBody: { token: 'secret', value: 'visible' }, + traceContext: { traceId: 'trace', spanId: 'span', traceFlags: 1 }, + }; + + manager.writeAccess(input); + + expect(writeAccess).toHaveBeenCalledWith(expect.objectContaining({ + requestBody: { i: '[REDACTED]', nested: { password: '[REDACTED]' } }, + responseBody: { token: '[REDACTED]', value: 'visible' }, + traceId: 'trace', + spanId: 'span', + })); + }); + + test('does not write Access log by default and preserves the default body limit', () => { + const { manager, writeAccess } = createManager(); + + manager.writeAccess({ + method: 'GET', + route: '/disabled', + statusCode: 200, + durationMs: 1, + responseSizeBytes: null, + }); + + expect(manager.getAccessLogConfiguration().bodies.maxBytes).toBe(16 * 1024); + expect(writeAccess).not.toHaveBeenCalled(); + }); + + test('accepts the maximum configured body limit', () => { + const { manager } = createManager({ + configuration: { + access: { + statusClasses: ['2xx'], + bodies: { maxBytes: 128 * 1024 }, + }, + }, + }); + + expect(manager.getAccessLogConfiguration().bodies.maxBytes).toBe(128 * 1024); + }); + + test('uses a replaced backend for subsequent records', () => { + const { manager, write } = createManager(); + const replacementWrite = vi.fn(); + + manager.setBackend({ write: replacementWrite }); + manager.write(createInput()); + + expect(write).not.toHaveBeenCalled(); + expect(replacementWrite).toHaveBeenCalledOnce(); + }); + + test('flushes and closes the backend once during shutdown', async () => { + const write = vi.fn(); + const flush = vi.fn>().mockResolvedValue(undefined); + const close = vi.fn>().mockResolvedValue(undefined); + const manager = new LogManager({ write, flush, close }); + + await Promise.all([manager.shutdown(), manager.shutdown()]); + + expect(flush).toHaveBeenCalledOnce(); + expect(close).toHaveBeenCalledOnce(); + expect(flush.mock.invocationCallOrder[0]).toBeLessThan(close.mock.invocationCallOrder[0]); + }); + + test('normalizes structured attributes and errors before writing', () => { + const { manager, write } = createManager(); + const error = new TypeError('broken'); + + manager.write({ + ...createInput('error'), + eventName: 'api.endpoint.failed', + attributes: { i: 'secret', safe: 'value' }, + error, + }); + + expect(write.mock.calls[0][0]).toMatchObject({ + eventName: 'api.endpoint.failed', + attributes: { i: '[REDACTED]', safe: 'value' }, + error: { type: 'TypeError', message: 'broken' }, + }); + }); + + test('does not pass raw structured values when normalization omits an error', () => { + const { manager, write } = createManager(); + + manager.write({ + ...createInput('error'), + attributes: { detail: 'value' }, + error: null, + }); + + expect(write.mock.calls[0][0].attributes).toEqual({ detail: 'value' }); + expect(write.mock.calls[0][0]).not.toHaveProperty('error'); + }); + + test('keeps legacy data for the pretty output while serializing its Error separately', () => { + const { manager, write } = createManager(); + const error = new Error('legacy failure'); + const data = { detail: 'legacy', e: error }; + + manager.write({ + ...createInput('error'), + compatibility: { data }, + }); + + expect(write.mock.calls[0][0].compatibility?.data).toBe(data); + expect(write.mock.calls[0][0]).toMatchObject({ + error: { type: 'Error', message: 'legacy failure' }, + }); + expect(write.mock.calls[0][0].attributes).toBeUndefined(); + }); + + test('supports the detailed normalization profile', () => { + const { manager, write } = createManager({ normalizationProfile: 'detailed' }); + + manager.write({ + ...createInput(), + attributes: { nested: { level1: { level2: { level3: { value: 'kept' } } } } }, + }); + + expect(write.mock.calls[0][0].attributes).toEqual({ + nested: { level1: { level2: { level3: { value: 'kept' } } } }, + }); + }); + + test('switches the normalization profile for existing loggers', () => { + const { manager, write } = createManager(); + const nested = { level1: { level2: { level3: { level4: { level5: { level6: { value: 'kept' } } } } } } }; + + manager.write({ ...createInput(), attributes: nested }); + manager.setNormalizationProfile('detailed'); + manager.write({ ...createInput(), attributes: nested }); + + expect(write.mock.calls[0][0].attributes).toMatchObject({ + level1: { level2: { level3: { level4: { level5: { level6: '[Truncated]' } } } } }, + }); + expect(write.mock.calls[1][0].attributes).toEqual(nested); + }); +}); diff --git a/packages/backend/test/unit/logging/LogNormalizer.ts b/packages/backend/test/unit/logging/LogNormalizer.ts new file mode 100644 index 0000000000..b183f03fcd --- /dev/null +++ b/packages/backend/test/unit/logging/LogNormalizer.ts @@ -0,0 +1,146 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test } from 'vitest'; +import { + findLegacyLogError, + normalizeLogAttributes, + normalizeLogValue, + serializeLogError, +} from '@/logging/LogNormalizer.js'; + +describe('LogNormalizer', () => { + test('normalizes common non-JSON values', () => { + expect(normalizeLogAttributes({ + date: new Date('2025-01-02T03:04:05.678Z'), + bigint: 123n, + infinity: Infinity, + undefinedValue: undefined, + })).toEqual({ + bigint: '123', + date: '2025-01-02T03:04:05.678Z', + infinity: 'Infinity', + undefinedValue: '[Unsupported]: undefined', + }); + }); + + test('normalizes a standalone body value with the same redaction and size rules', () => { + const normalized = normalizeLogValue({ token: 'secret', text: 'x'.repeat(100) }, { limits: { maxBytes: 256 } }); + + expect(normalized).toMatchObject({ token: '[REDACTED]' }); + expect(Buffer.byteLength(JSON.stringify(normalized), 'utf8')).toBeLessThanOrEqual(256); + }); + + test('marks unsupported objects and invalid dates without throwing', () => { + expect(normalizeLogAttributes({ + map: new Map([['key', 'value']]), + invalidDate: new Date('invalid'), + })).toEqual({ + invalidDate: '[Unsupported]: object access failed', + map: '[Unsupported]: object', + }); + }); + + test('preserves __proto__ as a normal attribute key', () => { + const value = JSON.parse('{"__proto__":{"nested":"value"},"large":"' + 'x'.repeat(100) + '"}') as Record; + const normalized = normalizeLogAttributes(value, { limits: { maxBytes: 64 } }); + + expect(Object.keys(normalized)).toContain('__proto__'); + expect(normalized['__proto__']).toEqual({ nested: 'value' }); + expect(Object.getPrototypeOf(normalized)).toBeNull(); + }); + + test('redacts sensitive fields recursively, including the Misskey i token', () => { + expect(normalizeLogAttributes({ + i: 'top-level-token', + request: { + Authorization: 'bearer-token', + captcha: 'captcha-value', + password: 'password', + nested: [{ api_key: 'api-key' }], + }, + visible: 'value', + })).toEqual({ + i: '[REDACTED]', + request: { + Authorization: '[REDACTED]', + captcha: '[REDACTED]', + password: '[REDACTED]', + nested: [{ api_key: '[REDACTED]' }], + }, + visible: 'value', + }); + }); + + test('keeps cycles finite and marks depth and entry truncation', () => { + const cycle: Record = { value: 'ok' }; + cycle.self = cycle; + + expect(normalizeLogAttributes({ + cycle, + deep: { level1: { level2: { level3: 'value' } } }, + }, { + limits: { maxDepth: 2, maxEntries: 10 }, + })).toEqual({ + cycle: { self: '[Circular]', value: 'ok' }, + deep: { level1: '[Truncated]' }, + }); + expect(normalizeLogAttributes({ entries: { a: 1, b: 2, c: 3 } }, { limits: { maxEntries: 2 } })).toEqual({ + entries: { a: 1, b: 2, '[Truncated]': '[Truncated]' }, + }); + }); + + test('uses the detailed profile while retaining the redaction policy', () => { + const value = { level1: { level2: { level3: { level4: 'value' } } }, token: 'secret' }; + + expect(normalizeLogAttributes(value, { limits: { maxDepth: 3 } })).toMatchObject({ level1: { level2: { level3: '[Truncated]' } }, token: '[REDACTED]' }); + expect(normalizeLogAttributes(value, { profile: 'detailed' })).toEqual({ + level1: { level2: { level3: { level4: 'value' } } }, + token: '[REDACTED]', + }); + }); + + test('keeps normalized attributes within the configured byte limit', () => { + const normalized = normalizeLogAttributes({ first: 'a'.repeat(100), second: 'b'.repeat(100) }, { limits: { maxBytes: 32 } }); + + expect(Buffer.byteLength(JSON.stringify(normalized), 'utf8')).toBeLessThanOrEqual(32); + }); + + test('truncates long multibyte strings without splitting characters', () => { + const normalized = normalizeLogAttributes({ value: '😀'.repeat(100) }, { limits: { maxStringBytes: 16, maxBytes: 100 } }); + + expect(Buffer.byteLength(JSON.stringify(normalized.value), 'utf8')).toBeLessThanOrEqual(16); + expect(JSON.stringify(normalized.value)).not.toContain('�'); + }); + + test('serializes Error and its cause consistently', () => { + const cause = new Error('root cause'); + const error = new TypeError('outer error', { cause }); + + expect(serializeLogError(error)).toMatchObject({ + type: 'TypeError', + message: 'outer error', + cause: { + type: 'Error', + message: 'root cause', + }, + }); + }); + + test('keeps serialized Error output within its byte limit', () => { + const serialized = serializeLogError(new Error('a long message'), { limits: { maxBytes: 32 } }); + + expect(serialized).toBeDefined(); + expect(Buffer.byteLength(JSON.stringify(serialized), 'utf8')).toBeLessThanOrEqual(32); + expect(serializeLogError(new Error('message'), { limits: { maxBytes: 20 } })).toBeUndefined(); + }); + + test('finds Error values in legacy data', () => { + const error = new Error('legacy'); + + expect(findLegacyLogError({ e: error })).toBe(error); + expect(findLegacyLogError({ stack: 'not-an-error' })).toBeUndefined(); + }); +}); diff --git a/packages/backend/test/unit/logging/Logger.ts b/packages/backend/test/unit/logging/Logger.ts new file mode 100644 index 0000000000..dca5cab773 --- /dev/null +++ b/packages/backend/test/unit/logging/Logger.ts @@ -0,0 +1,205 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { beforeEach, describe, expect, test, vi } from 'vitest'; +import type { LogRecordInput } from '@/logging/types.js'; + +const mocks = vi.hoisted(() => ({ + write: vi.fn<(input: LogRecordInput) => void>(), +})); + +vi.mock('@/logging/logging-runtime.js', () => ({ + logManager: { + write: mocks.write, + }, +})); + +import Logger from '@/logger.js'; + +describe('Logger', () => { + beforeEach(() => { + mocks.write.mockReset(); + }); + + test('passes immutable multi-level root-to-leaf context to the manager', () => { + const root = new Logger('root', 'red'); + const child = root.createSubLogger('child', 'green'); + const leaf = child.createSubLogger('leaf', 'blue'); + + leaf.info('from leaf'); + root.info('from root'); + + expect(mocks.write.mock.calls[0][0]).toMatchObject({ + level: 'info', + message: 'from leaf', + context: [ + { name: 'root', color: 'red' }, + { name: 'child', color: 'green' }, + { name: 'leaf', color: 'blue' }, + ], + }); + expect(mocks.write.mock.calls[1][0].context).toEqual([ + { name: 'root', color: 'red' }, + ]); + }); + + test('maps succ to info with the legacy success presentation', () => { + new Logger('root').succ('completed', { count: 1 }, true); + + expect(mocks.write).toHaveBeenCalledWith({ + level: 'info', + message: 'completed', + context: [{ name: 'root', color: undefined }], + compatibility: { + legacyLevel: 'success', + important: true, + data: { count: 1 }, + }, + }); + }); + + test('supports structured log input while preserving the context hierarchy', () => { + new Logger('root').createSubLogger('child').write({ + level: 'error', + eventName: 'example.failed', + message: 'failed', + attributes: { id: 'id' }, + error: new Error('broken'), + }); + + expect(mocks.write).toHaveBeenCalledWith(expect.objectContaining({ + level: 'error', + eventName: 'example.failed', + context: [{ name: 'root', color: undefined }, { name: 'child', color: undefined }], + })); + }); + + test('supports structured input through every level-specific method', () => { + const logger = new Logger('root').createSubLogger('child'); + const error = new Error('broken'); + const input = { + eventName: 'example.failed', + message: 'failed', + attributes: { id: 'id' }, + error, + }; + + logger.debug(input); + logger.info(input); + logger.warn(input); + logger.error(input); + logger.fatal(input); + + expect(mocks.write.mock.calls.map(([entry]) => entry.level)).toEqual([ + 'debug', + 'info', + 'warn', + 'error', + 'fatal', + ]); + for (const [entry] of mocks.write.mock.calls) { + expect(entry).toMatchObject({ + ...input, + context: [ + { name: 'root', color: undefined }, + { name: 'child', color: undefined }, + ], + }); + expect(entry).not.toHaveProperty('compatibility'); + } + }); + + test('level-specific methods own the level even for runtime-invalid input', () => { + const logger = new Logger('root'); + + // @ts-expect-error level is selected by the method rather than the input object + logger.warn({ level: 'error', message: 'warning' }); + + expect(mocks.write.mock.calls[0][0]).toMatchObject({ + level: 'warn', + message: 'warning', + }); + }); + + test('preserves the legacy string signatures for non-error levels', () => { + const logger = new Logger('root'); + logger.debug('debug', { source: 'debug' }, true); + logger.info('info', null, true); + logger.warn('warn', { source: 'warn' }); + + expect(mocks.write.mock.calls.map(([entry]) => entry.compatibility)).toEqual([ + { legacyLevel: undefined, important: true, data: { source: 'debug' } }, + { legacyLevel: undefined, important: true, data: null }, + { legacyLevel: undefined, important: false, data: { source: 'warn' } }, + ]); + }); + + test('records a fatal string through the structured API', () => { + new Logger('root').fatal('fatal message'); + + expect(mocks.write).toHaveBeenCalledWith({ + level: 'fatal', + message: 'fatal message', + context: [{ name: 'root', color: undefined }], + }); + }); + + test('preserves the legacy error string signature', () => { + new Logger('root').error('failed', { requestId: 'request' }, true); + + expect(mocks.write).toHaveBeenCalledWith({ + level: 'error', + message: 'failed', + context: [{ name: 'root', color: undefined }], + compatibility: { + legacyLevel: undefined, + important: true, + data: { requestId: 'request' }, + }, + }); + }); + + test('uses Error.toString and adds the Error to existing data', () => { + const logger = new Logger('root'); + const error = new TypeError('broken'); + const data: Record = { requestId: 'request' }; + + logger.error(error, data, true); + + expect(data).toEqual({ requestId: 'request', e: error }); + expect(mocks.write).toHaveBeenCalledWith({ + level: 'error', + message: 'TypeError: broken', + context: [{ name: 'root', color: undefined }], + error, + compatibility: { + legacyLevel: undefined, + important: true, + data, + }, + }); + }); + + test('creates data containing the Error when none is supplied', () => { + const error = new Error('broken'); + + new Logger('root').error(error); + + expect(mocks.write.mock.calls[0][0].compatibility?.data).toEqual({ e: error }); + }); + + test('keeps public methods bound when called separately', () => { + const logger = new Logger('root'); + const info = logger.info; + + info('bound'); + + expect(mocks.write).toHaveBeenCalledWith(expect.objectContaining({ + level: 'info', + message: 'bound', + context: [{ name: 'root', color: undefined }], + })); + }); +}); diff --git a/packages/backend/test/unit/logging/PrettyConsoleBackend.ts b/packages/backend/test/unit/logging/PrettyConsoleBackend.ts new file mode 100644 index 0000000000..004c60f783 --- /dev/null +++ b/packages/backend/test/unit/logging/PrettyConsoleBackend.ts @@ -0,0 +1,203 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import type { AccessLogRecord, LogRecord } from '@/logging/types.js'; + +type Formatter = ((value: unknown) => string) & { white?: Formatter }; + +const chalkMock = vi.hoisted(() => { + const format = (name: string): Formatter => (value: unknown) => `<${name}>${String(value)}`; + const bgRed = format('bgRed'); + bgRed.white = format('bgRed.white'); + const bgGreen = format('bgGreen'); + bgGreen.white = format('bgGreen.white'); + + return { + red: format('red'), + yellow: format('yellow'), + green: format('green'), + gray: format('gray'), + blue: format('blue'), + white: format('white'), + bold: format('bold'), + bgRed, + bgGreen, + rgb: (red: number, green: number, blue: number) => format(`rgb:${red},${green},${blue}`), + }; +}); + +vi.mock('chalk', () => ({ + default: chalkMock, +})); + +import { PrettyConsoleBackend } from '@/logging/PrettyConsoleBackend.js'; + +/** 見やすい形式のテストで使う共通のログを作成します。 */ +function createRecord(overrides: Partial = {}): LogRecord { + return { + level: 'info', + message: 'message', + context: [{ name: 'root' }], + timestamp: '2025-01-02T03:04:05.678Z', + loggerName: 'root', + processId: 1234, + isPrimary: true, + workerId: null, + ...overrides, + }; +} + +/** Access logの表示確認用に、正規化後と同じnull prototypeの本文を作成します。 */ +function createAccessRecord(requestBody: AccessLogRecord['requestBody']): AccessLogRecord { + return { + type: 'access', + timestamp: '2025-01-02T03:04:05.678Z', + method: 'POST', + route: '/api/test', + statusCode: 200, + durationMs: 1, + responseSizeBytes: 10, + requestBody, + processId: 1234, + isPrimary: true, + workerId: null, + }; +} + +describe('PrettyConsoleBackend', () => { + test.each([ + { level: 'error', label: 'ERR ', message: 'message' }, + { level: 'warn', label: 'WARN', message: 'message' }, + { level: 'debug', label: 'VERB', message: 'message' }, + { level: 'info', label: 'INFO', message: 'message' }, + ] as const)('formats the $level label and message', ({ level, label, message }) => { + const output = vi.fn(); + const backend = new PrettyConsoleBackend({ output, withLogTime: () => false }); + + backend.write(createRecord({ level })); + + expect(output).toHaveBeenCalledWith(`${label} *\t[root]\t${message}`); + }); + + test('formats legacy success as DONE while retaining the info severity', () => { + const output = vi.fn(); + const backend = new PrettyConsoleBackend({ output, withLogTime: () => false }); + + backend.write(createRecord({ + level: 'info', + compatibility: { legacyLevel: 'success' }, + })); + + expect(output).toHaveBeenCalledWith('DONE *\t[root]\tmessage'); + }); + + test('formats contexts from root to leaf using their configured colors', () => { + const output = vi.fn(); + const backend = new PrettyConsoleBackend({ output, withLogTime: () => false }); + + backend.write(createRecord({ + context: [ + { name: 'root', color: 'red' }, + { name: 'child' }, + { name: 'leaf', color: 'blue' }, + ], + })); + + expect(output).toHaveBeenCalledWith('INFO *\t[root child leaf]\tmessage'); + }); + + test('prefixes the time derived from the record timestamp when enabled', () => { + const output = vi.fn(); + const backend = new PrettyConsoleBackend({ output, withLogTime: () => true }); + + backend.write(createRecord()); + + expect(output).toHaveBeenCalledWith('03:04:05 INFO *\t[root]\tmessage'); + }); + + test('uses the worker id for a worker process', () => { + const output = vi.fn(); + const backend = new PrettyConsoleBackend({ output, withLogTime: () => false }); + + backend.write(createRecord({ isPrimary: false, workerId: 7 })); + + expect(output).toHaveBeenCalledWith('INFO 7\t[root]\tmessage'); + }); + + test('bolds an important record and passes data as the second output argument', () => { + const output = vi.fn(); + const data = { detail: 'value' }; + const backend = new PrettyConsoleBackend({ output, withLogTime: () => false }); + + backend.write(createRecord({ + level: 'error', + compatibility: { important: true, data }, + })); + + expect(output).toHaveBeenCalledWith( + 'ERR *\t[root]\tmessage', + data, + ); + }); + + test('treats fatal records as important errors', () => { + const output = vi.fn(); + const backend = new PrettyConsoleBackend({ output, withLogTime: () => false }); + + backend.write(createRecord({ level: 'fatal' })); + + expect(output).toHaveBeenCalledWith('ERR *\t[root]\tmessage'); + }); + + test('omits null data from output arguments', () => { + const output = vi.fn(); + const backend = new PrettyConsoleBackend({ output, withLogTime: () => false }); + + backend.write(createRecord({ compatibility: { data: null } })); + + expect(output).toHaveBeenCalledTimes(1); + expect(output.mock.calls[0]).toHaveLength(1); + }); + + test('passes structured event details as normalized output data', () => { + const output = vi.fn(); + const backend = new PrettyConsoleBackend({ output, withLogTime: () => false }); + + backend.write(createRecord({ + eventName: 'api.endpoint.failed', + attributes: { 'api.endpoint': 'notes/show' }, + error: { type: 'TypeError', message: 'broken' }, + })); + + expect(output).toHaveBeenCalledWith( + 'INFO *\t[root]\tmessage', + { + eventName: 'api.endpoint.failed', + attributes: { 'api.endpoint': 'notes/show' }, + error: { type: 'TypeError', message: 'broken' }, + }, + ); + }); + + test('shows normalized body maps as regular objects in pretty output', () => { + const output = vi.fn(); + const backend = new PrettyConsoleBackend({ output, withLogTime: () => false }); + const nested = Object.create(null) as Record; + nested.visible = 'value'; + const requestBody = Object.create(null) as Record; + requestBody.nested = nested; + Object.defineProperty(requestBody, '__proto__', { value: 'safe', enumerable: true }); + + backend.writeAccess(createAccessRecord(requestBody as AccessLogRecord['requestBody'])); + + const details = output.mock.calls[0][1] as { requestBody: Record }; + expect(Object.getPrototypeOf(details.requestBody)).toBe(Object.prototype); + expect(Object.getPrototypeOf(details.requestBody.nested)).toBe(Object.prototype); + expect(details.requestBody).toHaveProperty('__proto__', 'safe'); + expect(Object.getPrototypeOf(requestBody)).toBe(null); + expect(Object.getPrototypeOf(nested)).toBe(null); + }); +}); diff --git a/packages/backend/test/unit/logging/logging-runtime.ts b/packages/backend/test/unit/logging/logging-runtime.ts new file mode 100644 index 0000000000..9e8b2c2903 --- /dev/null +++ b/packages/backend/test/unit/logging/logging-runtime.ts @@ -0,0 +1,40 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { JsonConsoleBackend } from '@/logging/JsonConsoleBackend.js'; +import { logManager, configureLogging } from '@/logging/logging-runtime.js'; +import { PrettyConsoleBackend } from '@/logging/PrettyConsoleBackend.js'; + +describe('logging-runtime', () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + test('uses Pretty backend when the format is omitted', () => { + const setBackend = vi.spyOn(logManager, 'setBackend'); + + configureLogging(); + + expect(setBackend).toHaveBeenCalledWith(expect.any(PrettyConsoleBackend)); + }); + + test('uses JSON backend when the format is json', () => { + const setBackend = vi.spyOn(logManager, 'setBackend'); + + configureLogging({ format: 'json' }); + + expect(setBackend).toHaveBeenCalledWith(expect.any(JsonConsoleBackend)); + }); + + test('rejects an unknown format before changing logging configuration', () => { + const setBackend = vi.spyOn(logManager, 'setBackend'); + const configure = vi.spyOn(logManager, 'configure'); + + expect(() => configureLogging({ format: 'xml' as never })).toThrow('logging.format'); + expect(setBackend).not.toHaveBeenCalled(); + expect(configure).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/backend/test/unit/misc/cache.ts b/packages/backend/test/unit/misc/cache.ts index 297378304e..80085cf148 100644 --- a/packages/backend/test/unit/misc/cache.ts +++ b/packages/backend/test/unit/misc/cache.ts @@ -38,6 +38,29 @@ describe('misc:MemoryKVCache', () => { cache.dispose(); }); + test('keeps current behavior when limit is omitted', () => { + const cache = new MemoryKVCache(1000 * 60); + cache.set('a', 1); + cache.set('b', 2); + cache.set('c', 3); + expect(cache.get('a')).toBe(1); + expect(cache.get('b')).toBe(2); + expect(cache.get('c')).toBe(3); + cache.dispose(); + }); + + test('evicts least recently used entry when limit is reached', () => { + const cache = new MemoryKVCache(1000 * 60, 2); + cache.set('a', 1); + cache.set('b', 2); + expect(cache.get('a')).toBe(1); + cache.set('c', 3); + expect(cache.get('a')).toBe(1); + expect(cache.get('b')).toBeUndefined(); + expect(cache.get('c')).toBe(3); + cache.dispose(); + }); + describe('gc()', () => { test('removes expired entries', () => { const cache = new MemoryKVCache(1000); diff --git a/packages/backend/test/unit/queue/queue-job-runner.ts b/packages/backend/test/unit/queue/queue-job-runner.ts new file mode 100644 index 0000000000..3a587b0623 --- /dev/null +++ b/packages/backend/test/unit/queue/queue-job-runner.ts @@ -0,0 +1,56 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import { runQueueJobWithTraceContext } from '@/queue/queue-job-runner.js'; +import { TelemetryService } from '@/core/telemetry/TelemetryService.js'; + +describe('runQueueJobWithTraceContext', () => { + test('returns the processor result without invoking the error handler', async () => { + let spanActive = false; + const startSpanWithTraceContext = vi.fn((_name: string, _jobData: object, fn: () => T): T => { + spanActive = true; + const result = fn(); + if (result instanceof Promise) return result.finally(() => { spanActive = false; }) as T; + spanActive = false; + return result; + }); + const telemetryService = { + startSpanWithTraceContext, + } as unknown as TelemetryService; + const onError = vi.fn(); + + await expect(runQueueJobWithTraceContext(telemetryService, 'Queue: test', {}, () => 'ok', onError)).resolves.toBe('ok'); + + expect(onError).not.toHaveBeenCalled(); + expect(spanActive).toBe(false); + }); + + test('handles failures while the processor span is active and rethrows the original error', async () => { + let spanActive = false; + const startSpanWithTraceContext = vi.fn((_name: string, _jobData: object, fn: () => T): T => { + spanActive = true; + const result = fn(); + if (result instanceof Promise) return result.finally(() => { spanActive = false; }) as T; + spanActive = false; + return result; + }); + const telemetryService = { + startSpanWithTraceContext, + } as unknown as TelemetryService; + const onError = vi.fn((error: Error) => { + expect(spanActive).toBe(true); + expect(error).toBeInstanceOf(Error); + }); + const originalError = new Error('failed'); + + await expect(runQueueJobWithTraceContext(telemetryService, 'Queue: test', {}, async () => { + throw originalError; + }, onError)).rejects.toBe(originalError); + + expect(onError).toHaveBeenCalledOnce(); + expect(spanActive).toBe(false); + }); +}); diff --git a/packages/backend/test/unit/server/ApiCallService.ts b/packages/backend/test/unit/server/ApiCallService.ts new file mode 100644 index 0000000000..6c1f520207 --- /dev/null +++ b/packages/backend/test/unit/server/ApiCallService.ts @@ -0,0 +1,98 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import { ApiCallService } from '@/server/api/ApiCallService.js'; +import Logger from '@/logger.js'; +import { envOption } from '@/env.js'; +import { logManager } from '@/logging/logging-runtime.js'; +import { PrettyConsoleBackend } from '@/logging/PrettyConsoleBackend.js'; +import type { LogBackend } from '@/logging/LogBackend.js'; +import type { LogRecord } from '@/logging/types.js'; + +/** API失敗ログを確認するための最小Fastify応答を作成します。 */ +function createReply() { + return { + code: vi.fn(), + header: vi.fn(), + send: vi.fn(), + }; +} + +/** APIサービスの依存関係を最小限の仮実装へ差し替えます。 */ +function createService() { + const authenticateService = { + authenticate: vi.fn().mockResolvedValue([null, null]), + }; + const telemetryService = { + startSpan: vi.fn((_name: string, callback: () => unknown) => callback()), + captureMessage: vi.fn(), + }; + const apiLoggerService = { logger: new Logger('api') }; + + const service = new ApiCallService( + {} as never, + {} as never, + {} as never, + authenticateService as never, + {} as never, + {} as never, + apiLoggerService as never, + telemetryService as never, + ); + return { service, telemetryService }; +} + +describe('ApiCallService structured error logging', () => { + test('redacts API credentials and serializes the endpoint error', async () => { + const write = vi.fn(); + logManager.setBackend({ write }); + const previousQuiet = envOption.quiet; + envOption.quiet = false; + const { service, telemetryService } = createService(); + try { + const reply = createReply(); + const endpoint = { + name: 'notes/show', + meta: {}, + params: {}, + exec: vi.fn().mockRejectedValue(new TypeError('broken endpoint')), + }; + const request = { + method: 'POST', + body: { + i: 'native-token', + password: 'password', + options: { visible: true }, + }, + query: {}, + headers: {}, + ip: '127.0.0.1', + }; + + await service.handleRequest(endpoint as never, request as never, reply as never); + + const record = write.mock.calls[0][0] as LogRecord; + expect(record).toMatchObject({ + eventName: 'api.endpoint.failed', + attributes: { + 'api.endpoint': 'notes/show', + 'api.params': { + i: '[REDACTED]', + password: '[REDACTED]', + options: { visible: true }, + }, + }, + error: { type: 'TypeError', message: 'broken endpoint' }, + }); + expect(record.attributes?.['error.id']).toEqual(expect.any(String)); + expect(telemetryService.captureMessage.mock.calls[0][1].extra).not.toHaveProperty('ps'); + } finally { + service.dispose(); + envOption.quiet = previousQuiet; + logManager.setBackend(new PrettyConsoleBackend({ output: () => undefined })); + } + }); +}); diff --git a/packages/backend/test/unit/server/http-access-log.ts b/packages/backend/test/unit/server/http-access-log.ts new file mode 100644 index 0000000000..ddc235a076 --- /dev/null +++ b/packages/backend/test/unit/server/http-access-log.ts @@ -0,0 +1,249 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { Readable } from 'node:stream'; +import Fastify, { type FastifyInstance } from 'fastify'; +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { LogManager } from '@/logging/LogManager.js'; +import type { AccessLogRecord, AccessLogStatusClass } from '@/logging/types.js'; +import { registerHttpAccessLog } from '@/server/http-access-log.js'; + +type TestServer = { + readonly fastify: FastifyInstance; + readonly manager: LogManager; + readonly writeAccess: ReturnType; +}; + +type TestManager = { + readonly manager: LogManager; + readonly writeAccess: ReturnType; +}; + +/** Access logの動作確認用に固定時刻・プロセス情報を持つManagerを作成します。 */ +function createManager(options: { + statusClasses?: AccessLogStatusClass[]; + requestBody?: boolean; + responseBody?: boolean; + maxBytes?: number; + nodeEnv?: string; + quiet?: boolean; +} = {}): TestManager { + const writeAccess = vi.fn<(record: AccessLogRecord) => void>(); + const manager = new LogManager({ write: vi.fn(), writeAccess }, { + now: () => new Date('2026-07-22T00:00:00.000Z'), + getProcessInfo: () => ({ processId: 123, isPrimary: true, workerId: null }), + isQuiet: () => options.quiet ?? false, + isVerbose: () => false, + getNodeEnv: () => options.nodeEnv ?? 'development', + }); + manager.configure({ + access: { + statusClasses: options.statusClasses ?? ['2xx', '3xx', '4xx', '5xx'], + bodies: { + request: options.requestBody ?? false, + response: options.responseBody ?? false, + maxBytes: options.maxBytes, + }, + }, + }); + return { manager, writeAccess }; +} + +/** Access logフックを登録したテスト用Fastifyを作成します。 */ +async function createServer(options: Parameters[0] = {}): Promise { + const { manager, writeAccess } = createManager(options); + const fastify = Fastify({ logger: false }); + registerHttpAccessLog(fastify, manager); + fastify.get('/items/:id', async () => ({ ok: true })); + fastify.get('/bad', async (_request, reply) => reply.code(400).send({ error: 'bad' })); + fastify.get('/fail', async () => { + throw new TypeError('failure'); + }); + fastify.get('/redirect', async (_request, reply) => reply.redirect('/items/redirect')); + fastify.post('/body', async (request) => ({ echo: request.body, token: 'response-secret' })); + fastify.get('/text', async (_request, reply) => reply.type('text/plain').send('response text')); + fastify.get('/form', async (_request, reply) => reply.type('application/x-www-form-urlencoded').send('i=form-token&password=form-password&visible=yes')); + fastify.get('/binary', async (_request, reply) => reply.type('application/octet-stream').send(Buffer.from('binary'))); + fastify.get('/stream', async (_request, reply) => reply.type('text/plain').send(Readable.from(['stream body']))); + await fastify.ready(); + return { fastify, manager, writeAccess }; +} + +const servers: FastifyInstance[] = []; + +afterEach(async () => { + await Promise.all(servers.splice(0).map(server => server.close())); +}); + +describe('registerHttpAccessLog', () => { + test('filters responses by configured status classes and keeps the route template', async () => { + const server = await createServer({ statusClasses: ['4xx', '5xx'] }); + servers.push(server.fastify); + + await server.fastify.inject({ method: 'GET', url: '/items/secret?id=hidden' }); + await server.fastify.inject({ method: 'GET', url: '/bad' }); + await server.fastify.inject({ method: 'GET', url: '/fail' }); + await server.fastify.inject({ method: 'GET', url: '/missing?token=hidden' }); + + expect(server.writeAccess).toHaveBeenCalledTimes(3); + expect(server.writeAccess.mock.calls.map(call => call[0])).toEqual(expect.arrayContaining([ + expect.objectContaining({ route: '/bad', statusCode: 400 }), + expect.objectContaining({ route: '/fail', statusCode: 500, errorType: 'TypeError' }), + expect.objectContaining({ route: null, statusCode: 404 }), + ])); + expect(server.writeAccess.mock.calls[0][0]).not.toHaveProperty('requestUrl'); + expect(server.writeAccess.mock.calls.find(call => call[0].statusCode === 404)?.[0]).not.toHaveProperty('errorType'); + }); + + test('records redirects and response size when the status class is selected', async () => { + const server = await createServer({ statusClasses: ['3xx'] }); + servers.push(server.fastify); + + await server.fastify.inject({ method: 'GET', url: '/redirect' }); + + expect(server.writeAccess).toHaveBeenCalledWith(expect.objectContaining({ + method: 'GET', + route: '/redirect', + statusCode: 302, + responseSizeBytes: expect.any(Number), + })); + }); + + test('captures and redacts JSON request and response bodies in development', async () => { + const server = await createServer({ requestBody: true, responseBody: true }); + servers.push(server.fastify); + + await server.fastify.inject({ + method: 'POST', + url: '/body', + headers: { 'content-type': 'application/json' }, + payload: { i: 'request-token', nested: { password: 'request-password' }, value: 'visible' }, + }); + + expect(server.writeAccess).toHaveBeenCalledWith(expect.objectContaining({ + requestBody: { + i: '[REDACTED]', + nested: { password: '[REDACTED]' }, + value: 'visible', + }, + responseBody: { + echo: { + i: '[REDACTED]', + nested: { password: '[REDACTED]' }, + value: 'visible', + }, + token: '[REDACTED]', + }, + })); + }); + + test('captures text but omits binary and stream bodies', async () => { + const server = await createServer({ statusClasses: ['2xx'], responseBody: true }); + servers.push(server.fastify); + + await server.fastify.inject({ method: 'GET', url: '/text' }); + await server.fastify.inject({ method: 'GET', url: '/binary' }); + await server.fastify.inject({ method: 'GET', url: '/stream' }); + + expect(server.writeAccess.mock.calls[0][0]).toHaveProperty('responseBody', 'response text'); + expect(server.writeAccess.mock.calls[1][0]).not.toHaveProperty('responseBody'); + expect(server.writeAccess.mock.calls[2][0]).not.toHaveProperty('responseBody'); + }); + + test('parses form bodies before redaction', async () => { + const server = await createServer({ statusClasses: ['2xx'], responseBody: true }); + servers.push(server.fastify); + + await server.fastify.inject({ method: 'GET', url: '/form' }); + + expect(server.writeAccess).toHaveBeenCalledWith(expect.objectContaining({ + responseBody: { + i: '[REDACTED]', + password: '[REDACTED]', + visible: 'yes', + }, + })); + }); + + test('truncates normalized bodies to the configured limit', async () => { + const server = await createServer({ requestBody: true, responseBody: true, maxBytes: 1024 }); + servers.push(server.fastify); + + await server.fastify.inject({ + method: 'POST', + url: '/body', + headers: { 'content-type': 'application/json' }, + payload: { value: 'x'.repeat(20_000) }, + }); + + const record = server.writeAccess.mock.calls[0][0]; + expect(Buffer.byteLength(JSON.stringify(record.requestBody), 'utf8')).toBeLessThanOrEqual(1024); + expect(Buffer.byteLength(JSON.stringify(record.responseBody), 'utf8')).toBeLessThanOrEqual(1024); + }); + + test('preserves the response payload and reports an unknown stream size', async () => { + const server = await createServer({ statusClasses: ['2xx'], responseBody: true }); + servers.push(server.fastify); + + const response = await server.fastify.inject({ method: 'GET', url: '/stream' }); + + expect(response.body).toBe('stream body'); + expect(server.writeAccess).toHaveBeenCalledWith(expect.objectContaining({ responseSizeBytes: null })); + }); + + test('does not capture bodies in production and returns a warning', async () => { + const { manager, writeAccess } = createManager({ nodeEnv: 'production', requestBody: true, responseBody: true }); + const warnings = manager.configure({ access: { statusClasses: ['2xx'], bodies: { request: true, response: true } } }); + const fastify = Fastify({ logger: false }); + registerHttpAccessLog(fastify, manager); + fastify.post('/body', async request => ({ body: request.body })); + await fastify.ready(); + servers.push(fastify); + + await fastify.inject({ method: 'POST', url: '/body', headers: { 'content-type': 'application/json' }, payload: { token: 'hidden' } }); + + expect(warnings).toEqual(['logging.access.bodies is disabled in production mode']); + expect(writeAccess).toHaveBeenCalledWith(expect.not.objectContaining({ requestBody: expect.anything(), responseBody: expect.anything() })); + }); + + test('keeps the request Trace Context through response completion', async () => { + const server = await createServer({ statusClasses: ['2xx'] }); + servers.push(server.fastify); + const traceContext = { traceId: 'trace', spanId: 'span', traceFlags: 1 }; + const provider = vi.fn(() => traceContext); + server.manager.setTraceContextProvider(provider); + + await server.fastify.inject({ method: 'GET', url: '/items/trace' }); + + expect(provider).toHaveBeenCalledOnce(); + expect(server.writeAccess).toHaveBeenCalledWith(expect.objectContaining(traceContext)); + }); + + test('omits a Trace Context that was not active at request start', async () => { + const server = await createServer({ statusClasses: ['2xx'] }); + servers.push(server.fastify); + const provider = vi.fn() + .mockReturnValueOnce(undefined) + .mockReturnValue({ traceId: 'late-trace', spanId: 'late-span', traceFlags: 1 }); + server.manager.setTraceContextProvider(provider); + + await server.fastify.inject({ method: 'GET', url: '/items/no-trace' }); + + expect(server.writeAccess.mock.calls[0][0]).not.toHaveProperty('traceId'); + expect(provider).toHaveBeenCalledOnce(); + }); + + test('does not write in quiet mode', async () => { + const server = await createServer({ quiet: true }); + servers.push(server.fastify); + const provider = vi.fn(() => ({ traceId: 'trace', spanId: 'span', traceFlags: 1 })); + server.manager.setTraceContextProvider(provider); + + await server.fastify.inject({ method: 'GET', url: '/items/quiet' }); + + expect(provider).not.toHaveBeenCalled(); + expect(server.writeAccess).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/backend/test/unit/server/http-server-instrumentation.ts b/packages/backend/test/unit/server/http-server-instrumentation.ts new file mode 100644 index 0000000000..3e4baae14a --- /dev/null +++ b/packages/backend/test/unit/server/http-server-instrumentation.ts @@ -0,0 +1,53 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import { shouldRegisterHttpServerInstrumentation, registerHttpServerInstrumentation } from '@/server/http-server-instrumentation.js'; + +const mocks = vi.hoisted(() => ({ + plugin: vi.fn(), + instrumentation: vi.fn(), +})); + +vi.mock('@fastify/otel', () => ({ + FastifyOtelInstrumentation: class { + public plugin = mocks.plugin; + + public constructor(options: unknown) { + mocks.instrumentation(options); + } + }, +})); + +describe('http-server-instrumentation', () => { + test('registers Fastify instrumentation when only OpenTelemetry is configured', async () => { + const plugin = vi.fn(); + const fastify = { register: vi.fn().mockResolvedValue(undefined) }; + mocks.plugin.mockReturnValue(plugin); + + await registerHttpServerInstrumentation(fastify as any, { otelForBackend: {} } as any); + + expect(mocks.instrumentation).toHaveBeenCalledTimes(1); + expect(fastify.register).toHaveBeenCalledWith(plugin); + + const requestHook = mocks.instrumentation.mock.calls[0][0].requestHook; + const span = { updateName: vi.fn() }; + requestHook(span, { method: 'POST', routeOptions: { url: '/notes/create' } }); + expect(span.updateName).toHaveBeenCalledWith('POST /notes/create'); + }); + + test('does not register duplicate request instrumentation with Sentry', async () => { + const fastify = { register: vi.fn() }; + + await registerHttpServerInstrumentation(fastify as any, { otelForBackend: {}, sentryForBackend: {} } as any); + + expect(fastify.register).not.toHaveBeenCalled(); + expect(shouldRegisterHttpServerInstrumentation({ otelForBackend: {}, sentryForBackend: {} } as any)).toBe(false); + }); + + test('does not register instrumentation without OpenTelemetry', () => { + expect(shouldRegisterHttpServerInstrumentation({} as any)).toBe(false); + }); +}); diff --git a/packages/backend/test/unit/telemetry-database-instrumentation.ts b/packages/backend/test/unit/telemetry-database-instrumentation.ts new file mode 100644 index 0000000000..5ac31fd0e6 --- /dev/null +++ b/packages/backend/test/unit/telemetry-database-instrumentation.ts @@ -0,0 +1,108 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import { installDatabaseInstrumentation, installInstrumentation } from '@/core/telemetry/database-instrumentation.js'; + +describe('database-instrumentation', () => { + test('does not install PostgreSQL instrumentation when disabled', async () => { + const uninstall = await installDatabaseInstrumentation({} as any, { + capturePgSpans: false, + capturePgStatement: false, + capturePgConnectionSpans: false, + }); + + expect(uninstall).toBeTypeOf('function'); + expect(() => uninstall()).not.toThrow(); + }); + + test('registers pg instrumentation with the active provider', () => { + const provider = {}; + const pg = { setTracerProvider: vi.fn(), enable: vi.fn(), disable: vi.fn() }; + const config = vi.fn(); + const PgInstrumentation = class { + public constructor(options: unknown) { + config(options); + return pg as any; + } + }; + + const uninstall = installInstrumentation(provider as any, { + PgInstrumentation: PgInstrumentation as any, + }, { + capturePgStatement: false, + capturePgConnectionSpans: false, + }); + + expect(pg.setTracerProvider).toHaveBeenCalledWith(provider); + expect(pg.enable).toHaveBeenCalledOnce(); + expect(config).toHaveBeenCalledWith(expect.objectContaining({ + enhancedDatabaseReporting: false, + requireParentSpan: true, + ignoreConnectSpans: true, + requestHook: expect.any(Function), + })); + + const span = { setAttribute: vi.fn() }; + (config.mock.calls[0][0] as { requestHook: (span: any) => void }).requestHook(span); + expect(span.setAttribute).toHaveBeenCalledWith('db.statement', '[REDACTED]'); + expect(span.setAttribute).toHaveBeenCalledWith('db.query.text', '[REDACTED]'); + + uninstall(); + + expect(pg.disable).toHaveBeenCalledOnce(); + }); + + test('keeps SQL statement attributes when explicitly enabled', () => { + const pg = { setTracerProvider: vi.fn(), enable: vi.fn(), disable: vi.fn() }; + const config = vi.fn(); + + installInstrumentation({} as any, { + PgInstrumentation: class { + public constructor(options: unknown) { + config(options); + return pg as any; + } + } as any, + }, { + capturePgStatement: true, + capturePgConnectionSpans: false, + }); + + expect(config.mock.calls[0][0]).not.toHaveProperty('requestHook'); + }); + + test('enables connection spans when explicitly configured', () => { + const pg = { setTracerProvider: vi.fn(), enable: vi.fn(), disable: vi.fn() }; + const config = vi.fn(); + + installInstrumentation({} as any, { + PgInstrumentation: class { + public constructor(options: unknown) { + config(options); + return pg as any; + } + } as any, + }, { + capturePgStatement: false, + capturePgConnectionSpans: true, + }); + + expect(config).toHaveBeenCalledWith(expect.objectContaining({ + ignoreConnectSpans: false, + })); + }); + + test('cleans up both instrumentations when initialization fails', () => { + const pg = { setTracerProvider: vi.fn(), enable: vi.fn(), disable: vi.fn() }; + pg.enable.mockImplementation(() => { throw new Error('failed'); }); + + expect(() => installInstrumentation({} as any, { + PgInstrumentation: class { public constructor() { return pg as any; } } as any, + })).toThrow('failed'); + + expect(pg.disable).toHaveBeenCalledOnce(); + }); +}); diff --git a/packages/backend/test/unit/telemetry-redis-instrumentation.ts b/packages/backend/test/unit/telemetry-redis-instrumentation.ts new file mode 100644 index 0000000000..85a7df3dca --- /dev/null +++ b/packages/backend/test/unit/telemetry-redis-instrumentation.ts @@ -0,0 +1,148 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { describe, expect, test, vi } from 'vitest'; +import { SpanKind, SpanStatusCode } from '@opentelemetry/api'; +import { createRedisInstrumentation } from '@/core/telemetry/redis-instrumentation.js'; + +describe('redis-instrumentation', () => { + test('creates and completes a span for an ioredis command', () => { + let subscribers: any; + const unsubscribe = vi.fn(); + const span = { end: vi.fn(), recordException: vi.fn(), setStatus: vi.fn(), setAttribute: vi.fn() }; + const tracer = { startSpan: vi.fn(() => span) }; + const uninstall = createRedisInstrumentation({ + tracingChannel: () => ({ subscribe: (value) => { subscribers = value; }, unsubscribe }), + tracer: tracer as any, + getActiveSpan: () => ({}) as any, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + }, { captureCommandSpans: true }); + const command = { command: 'get', args: ['key'], database: 0, serverAddress: 'redis', serverPort: 6379 }; + + subscribers.start(command); + subscribers.asyncEnd(command); + + expect(tracer.startSpan).toHaveBeenCalledWith('get', expect.objectContaining({ + kind: SpanKind.CLIENT, + attributes: expect.objectContaining({ + 'db.system.name': 'redis', + 'db.operation.name': 'get', + 'server.address': 'redis', + 'server.port': 6379, + }), + })); + expect(span.end).toHaveBeenCalledOnce(); + + uninstall(); + expect(unsubscribe).toHaveBeenCalledWith(subscribers); + }); + + test('records rejected Redis commands as errors', () => { + let subscribers: any; + const span = { end: vi.fn(), recordException: vi.fn(), setStatus: vi.fn(), setAttribute: vi.fn() }; + createRedisInstrumentation({ + tracingChannel: () => ({ subscribe: (value) => { subscribers = value; }, unsubscribe: vi.fn() }), + tracer: { startSpan: vi.fn(() => span) } as any, + getActiveSpan: () => ({}) as any, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + }, { captureCommandSpans: true }); + const command = { command: 'get', args: ['key'], database: 0, serverAddress: 'redis', serverPort: undefined }; + const error = Object.assign(new Error('ERR Redis failed'), { code: 'ERR' }); + + subscribers.start(command); + Object.assign(command, { error }); + subscribers.error(command); + subscribers.asyncEnd(command); + + expect(span.recordException).toHaveBeenCalledWith(error); + expect(span.recordException).toHaveBeenCalledOnce(); + expect(span.setStatus).toHaveBeenCalledWith({ code: SpanStatusCode.ERROR, message: 'ERR Redis failed' }); + expect(span.setAttribute).toHaveBeenCalledWith('error.type', 'ERR'); + expect(span.setAttribute).toHaveBeenCalledWith('db.response.status_code', 'ERR'); + expect(span.end).toHaveBeenCalledOnce(); + }); + + test('does not create a root span when no parent span is active', () => { + let subscribers: any; + const tracer = { startSpan: vi.fn() }; + createRedisInstrumentation({ + tracingChannel: () => ({ subscribe: (value) => { subscribers = value; }, unsubscribe: vi.fn() }), + tracer: tracer as any, + getActiveSpan: () => undefined, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + }, { captureCommandSpans: true }); + + subscribers.start({ command: 'get', args: ['key'], database: 0, serverAddress: 'redis', serverPort: 6379 }); + + expect(tracer.startSpan).not.toHaveBeenCalled(); + }); + + test('creates a root span when explicitly enabled', () => { + let subscribers: any; + const span = { end: vi.fn(), recordException: vi.fn(), setStatus: vi.fn(), setAttribute: vi.fn() }; + const tracer = { startSpan: vi.fn(() => span) }; + createRedisInstrumentation({ + tracingChannel: () => ({ subscribe: (value) => { subscribers = value; }, unsubscribe: vi.fn() }), + tracer: tracer as any, + getActiveSpan: () => undefined, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + }, { captureCommandSpans: true, requireParentSpan: false }); + + const command = { command: 'get', args: ['key'], database: 0, serverAddress: 'redis', serverPort: 6379 }; + subscribers.start(command); + subscribers.asyncEnd(command); + + expect(tracer.startSpan).toHaveBeenCalledOnce(); + expect(span.end).toHaveBeenCalledOnce(); + }); + + test('records connection spans only when explicitly enabled', () => { + const subscribers = new Map(); + const unsubscribe = vi.fn(); + const span = { end: vi.fn(), recordException: vi.fn(), setStatus: vi.fn(), setAttribute: vi.fn() }; + const tracer = { startSpan: vi.fn(() => span) }; + const uninstall = createRedisInstrumentation({ + tracingChannel: (name) => ({ subscribe: (value) => { subscribers.set(name, value); }, unsubscribe }), + tracer: tracer as any, + getActiveSpan: () => undefined, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + }, { captureConnectionSpans: true }); + + const connection = { serverAddress: 'redis', serverPort: 6379 }; + subscribers.get('ioredis:connect').start(connection); + subscribers.get('ioredis:connect').asyncEnd(connection); + + expect(tracer.startSpan).toHaveBeenCalledWith('connect', expect.objectContaining({ + kind: SpanKind.CLIENT, + attributes: expect.objectContaining({ + 'db.operation.name': 'connect', + 'server.address': 'redis', + 'server.port': 6379, + }), + })); + expect(span.end).toHaveBeenCalledOnce(); + + uninstall(); + expect(unsubscribe).toHaveBeenCalledWith(subscribers.get('ioredis:connect')); + }); + + test('does not subscribe to Redis command diagnostics unless explicitly enabled', () => { + const tracingChannel = vi.fn(); + createRedisInstrumentation({ + tracingChannel, + tracer: { startSpan: vi.fn() } as any, + getActiveSpan: () => undefined, + spanKindClient: SpanKind.CLIENT, + spanStatusCodeError: SpanStatusCode.ERROR, + }); + + expect(tracingChannel).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/backend/test/unit/telemetry-registry.ts b/packages/backend/test/unit/telemetry-registry.ts new file mode 100644 index 0000000000..dbf10c9176 --- /dev/null +++ b/packages/backend/test/unit/telemetry-registry.ts @@ -0,0 +1,180 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { beforeEach, describe, expect, test, vi } from 'vitest'; +import type { Config } from '@/config.js'; + +const mocks = vi.hoisted(() => { + return { + sentryCreate: vi.fn(), + sentryCreateWithOtlpExport: vi.fn(), + otelCreate: vi.fn(), + setLogTraceContextProvider: vi.fn(), + }; +}); + +vi.mock('@/logging/logging-runtime.js', () => ({ + setLogTraceContextProvider: mocks.setLogTraceContextProvider, +})); + +vi.mock('@/core/telemetry/adapters/SentryTelemetryAdapter.js', () => ({ + SentryTelemetryAdapter: { + create: mocks.sentryCreate, + createWithOtlpExport: mocks.sentryCreateWithOtlpExport, + }, +})); + +vi.mock('@/core/telemetry/adapters/OpenTelemetryAdapter.js', () => ({ + OpenTelemetryAdapter: { + create: mocks.otelCreate, + }, +})); + +function config(overrides: Partial): Config { + return { + version: '2026.1.0', + ...overrides, + } as Config; +} + +describe('telemetry-registry', () => { + beforeEach(() => { + vi.resetModules(); + mocks.sentryCreate.mockReset(); + mocks.sentryCreateWithOtlpExport.mockReset(); + mocks.otelCreate.mockReset(); + mocks.setLogTraceContextProvider.mockReset(); + mocks.sentryCreate.mockResolvedValue({ shutdown: vi.fn(), captureMessage: vi.fn(), startSpan: vi.fn() }); + mocks.sentryCreateWithOtlpExport.mockResolvedValue({ shutdown: vi.fn(), captureMessage: vi.fn(), startSpan: vi.fn() }); + mocks.otelCreate.mockResolvedValue({ shutdown: vi.fn(), captureMessage: vi.fn(), startSpan: vi.fn() }); + }); + + test('uses OpenTelemetryAdapter when only otelForBackend is configured', async () => { + const { initTelemetry } = await import('@/core/telemetry/telemetry-registry.js'); + const otelForBackend = { endpoint: 'http://collector:4318/v1/traces' }; + + await initTelemetry(config({ otelForBackend })); + + expect(mocks.otelCreate).toHaveBeenCalledWith({ + ...otelForBackend, + serviceVersion: '2026.1.0', + }); + expect(mocks.sentryCreate).not.toHaveBeenCalled(); + expect(mocks.sentryCreateWithOtlpExport).not.toHaveBeenCalled(); + }); + + test('registers the adapter trace context provider after telemetry initialization', async () => { + const { initTelemetry } = await import('@/core/telemetry/telemetry-registry.js'); + const getActiveTraceContext = vi.fn(() => ({ + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 0, + })); + mocks.otelCreate.mockResolvedValue({ + shutdown: vi.fn(), + captureMessage: vi.fn(), + startSpan: vi.fn(), + getActiveTraceContext, + }); + + await initTelemetry(config({ otelForBackend: { endpoint: 'http://collector:4318/v1/traces' } })); + + expect(mocks.setLogTraceContextProvider).toHaveBeenCalledWith(expect.any(Function)); + const provider = mocks.setLogTraceContextProvider.mock.calls[0][0] as () => unknown; + expect(provider()).toEqual({ + traceId: '0123456789abcdef0123456789abcdef', + spanId: '0123456789abcdef', + traceFlags: 0, + }); + expect(getActiveTraceContext).toHaveBeenCalledOnce(); + }); + + test('adds OTLP export to the Sentry provider when both Sentry and OTel are configured', async () => { + const { initTelemetry } = await import('@/core/telemetry/telemetry-registry.js'); + const sentryForBackend = { options: {}, enableNodeProfiling: false }; + const otelForBackend = { endpoint: 'http://collector:4318/v1/traces' }; + + await initTelemetry(config({ sentryForBackend, otelForBackend })); + + expect(mocks.sentryCreateWithOtlpExport).toHaveBeenCalledWith(sentryForBackend, { + ...otelForBackend, + serviceVersion: '2026.1.0', + }); + expect(mocks.sentryCreate).not.toHaveBeenCalled(); + expect(mocks.otelCreate).not.toHaveBeenCalled(); + }); + + test('startSpan runs fn directly when no adapter is registered', async () => { + const { startSpan } = await import('@/core/telemetry/telemetry-registry.js'); + + const fn = vi.fn().mockReturnValue('result'); + expect(startSpan('test', fn)).toBe('result'); + expect(fn).toHaveBeenCalledTimes(1); + }); + + test('startSpan delegates directly to the single registered adapter without extra wrapping', async () => { + const { initTelemetry, startSpan } = await import('@/core/telemetry/telemetry-registry.js'); + const otelForBackend = { endpoint: 'http://collector:4318/v1/traces' }; + const adapterStartSpan = vi.fn((_name: string, fn: () => string) => fn()); + mocks.otelCreate.mockResolvedValue({ shutdown: vi.fn(), captureMessage: vi.fn(), startSpan: adapterStartSpan }); + + await initTelemetry(config({ otelForBackend })); + + const fn = vi.fn().mockReturnValue('result'); + expect(startSpan('test', fn)).toBe('result'); + expect(adapterStartSpan).toHaveBeenCalledWith('test', fn); + }); + + test('startSpan wraps work through multiple registered adapters in order for future adapter combinations', async () => { + const { initTelemetry, startSpan } = await import('@/core/telemetry/telemetry-registry.js'); + const calls: string[] = []; + mocks.sentryCreate.mockResolvedValue({ + shutdown: vi.fn(), + captureMessage: vi.fn(), + startSpan: vi.fn((_name: string, fn: () => string) => { + calls.push('sentry:start'); + const result = fn(); + calls.push('sentry:end'); + return result; + }), + }); + mocks.otelCreate.mockResolvedValue({ + shutdown: vi.fn(), + captureMessage: vi.fn(), + startSpan: vi.fn((_name: string, fn: () => string) => { + calls.push('otel:start'); + const result = fn(); + calls.push('otel:end'); + return result; + }), + }); + + await initTelemetry(config({ sentryForBackend: { options: {}, enableNodeProfiling: false } })); + await initTelemetry(config({ otelForBackend: { endpoint: 'http://collector:4318/v1/traces' } })); + + const fn = vi.fn(() => { + calls.push('work'); + return 'result'; + }); + + expect(startSpan('test', fn)).toBe('result'); + expect(calls).toEqual(['sentry:start', 'otel:start', 'work', 'otel:end', 'sentry:end']); + }); + + test('shutdownTelemetry waits for every adapter even when one shutdown rejects', async () => { + const { initTelemetry, shutdownTelemetry } = await import('@/core/telemetry/telemetry-registry.js'); + const sentryShutdown = vi.fn().mockRejectedValue(new Error('sentry failed')); + const otelShutdown = vi.fn().mockResolvedValue(undefined); + mocks.sentryCreate.mockResolvedValue({ shutdown: sentryShutdown, captureMessage: vi.fn(), startSpan: vi.fn() }); + mocks.otelCreate.mockResolvedValue({ shutdown: otelShutdown, captureMessage: vi.fn(), startSpan: vi.fn() }); + + await initTelemetry(config({ sentryForBackend: { options: {}, enableNodeProfiling: false } })); + await initTelemetry(config({ otelForBackend: { endpoint: 'http://collector:4318/v1/traces' } })); + + await expect(shutdownTelemetry()).resolves.toBeUndefined(); + expect(sentryShutdown).toHaveBeenCalledTimes(1); + expect(otelShutdown).toHaveBeenCalledTimes(1); + }); +}); diff --git a/packages/frontend-builder/package.json b/packages/frontend-builder/package.json index ceb33a230d..047f8b4e73 100644 --- a/packages/frontend-builder/package.json +++ b/packages/frontend-builder/package.json @@ -3,7 +3,7 @@ "type": "module", "scripts": { "eslint": "eslint './**/*.{js,jsx,ts,tsx}'", - "typecheck": "tsgo --noEmit", + "typecheck": "tsc --noEmit", "lint": "pnpm typecheck && pnpm eslint" }, "exports": { @@ -11,16 +11,16 @@ }, "devDependencies": { "@types/estree": "1.0.9", - "@types/node": "26.0.0", - "@typescript-eslint/eslint-plugin": "8.61.1", - "@typescript-eslint/parser": "8.61.1", + "@types/node": "26.1.1", + "@typescript-eslint/eslint-plugin": "8.63.0", + "@typescript-eslint/parser": "8.63.0", "rollup": "4.62.2" }, "dependencies": { "i18n": "workspace:*", "magic-string": "0.30.21", "oxc-walker": "1.0.0", - "rolldown": "1.1.2", - "vite": "8.1.0" + "rolldown": "1.1.5", + "vite": "8.1.4" } } diff --git a/packages/frontend-builder/tsconfig.json b/packages/frontend-builder/tsconfig.json index ab943fded4..e2d1f7ada0 100644 --- a/packages/frontend-builder/tsconfig.json +++ b/packages/frontend-builder/tsconfig.json @@ -9,6 +9,7 @@ "sourceMap": false, "noEmit": true, "removeComments": true, + "skipLibCheck": true, "resolveJsonModule": true, "strict": true, "strictFunctionTypes": true, diff --git a/packages/frontend-builder/utils.ts b/packages/frontend-builder/utils.ts index f85ae7ea0c..3bb528e25f 100644 --- a/packages/frontend-builder/utils.ts +++ b/packages/frontend-builder/utils.ts @@ -8,5 +8,5 @@ export function assertNever(x: never): never { throw new Error(`Unexpected type: ${(x as any)?.type ?? x}`); } -export function assertType(_node: unknown): asserts node is T { +export function assertType(_node: unknown): asserts _node is T { } diff --git a/packages/frontend-embed/@types/global.d.ts b/packages/frontend-embed/@types/global.d.ts index 8a067a78ec..7e2f8f5567 100644 --- a/packages/frontend-embed/@types/global.d.ts +++ b/packages/frontend-embed/@types/global.d.ts @@ -13,8 +13,3 @@ declare const _PERF_PREFIX_: string; // for dev-mode declare const _LANGS_FULL_: string[][]; - -// TagCanvas -interface Window { - TagCanvas: any; -} diff --git a/packages/frontend-embed/@types/theme.d.ts b/packages/frontend-embed/@types/theme.d.ts index 6ac1037493..6a010889d6 100644 --- a/packages/frontend-embed/@types/theme.d.ts +++ b/packages/frontend-embed/@types/theme.d.ts @@ -4,7 +4,7 @@ */ declare module '@@/themes/*.json5' { - import { Theme } from '@/theme.js'; + import { Theme } from '@@/js/theme.js'; const theme: Theme; diff --git a/packages/frontend-embed/package.json b/packages/frontend-embed/package.json index 1026a36144..5c07e795f3 100644 --- a/packages/frontend-embed/package.json +++ b/packages/frontend-embed/package.json @@ -10,9 +10,6 @@ "lint": "pnpm typecheck && pnpm eslint" }, "dependencies": { - "@rollup/plugin-json": "6.1.0", - "@rollup/pluginutils": "5.4.0", - "@vitejs/plugin-vue": "6.0.7", "buraha": "0.0.1", "frontend-shared": "workspace:*", "i18n": "workspace:*", @@ -21,44 +18,37 @@ "mfm-js": "0.26.0", "misskey-js": "workspace:*", "punycode.js": "2.3.1", - "rollup": "4.62.2", - "shiki": "4.2.0", + "shiki": "4.3.1", "tinycolor2": "1.6.0", - "uuid": "14.0.0", - "vue": "3.5.38" + "uuid": "14.0.1", + "vue": "3.5.39" }, "devDependencies": { "@misskey-dev/emoji-assets": "17.0.3", "@misskey-dev/summaly": "5.5.1", + "@rollup/plugin-json": "6.1.0", + "@rollup/pluginutils": "5.4.0", "@tabler/icons-webfont": "3.35.0", "@testing-library/vue": "8.1.0", - "@types/estree": "1.0.9", - "@types/micromatch": "4.0.10", - "@types/node": "26.0.0", + "@types/node": "26.1.1", "@types/punycode.js": "npm:@types/punycode@2.1.4", "@types/tinycolor2": "1.4.6", "@types/ws": "8.18.1", - "@typescript-eslint/eslint-plugin": "8.61.1", - "@typescript-eslint/parser": "8.61.1", - "@vitest/coverage-v8": "4.1.9", - "@vue/runtime-core": "3.5.38", - "acorn": "8.17.0", - "cross-env": "10.1.0", + "@typescript-eslint/eslint-plugin": "8.63.0", + "@typescript-eslint/parser": "8.63.0", + "@vitest/coverage-v8": "4.1.10", + "@vitejs/plugin-vue": "6.0.7", + "@vue/runtime-core": "3.5.39", "eslint-plugin-import": "2.32.0", "eslint-plugin-vue": "10.9.2", - "happy-dom": "20.10.6", "intersection-observer": "0.12.2", "lightningcss": "1.32.0", - "micromatch": "4.0.8", - "msw": "2.14.6", - "prettier": "3.8.4", "sass-embedded": "1.100.0", - "start-server-and-test": "3.0.11", - "tsx": "4.22.4", - "vite": "8.1.0", - "vite-plugin-turbosnap": "1.0.3", - "vue-component-type-helpers": "3.3.5", + "tsx": "4.23.0", + "typescript": "6.0.2", + "vite": "8.1.4", + "vue-component-type-helpers": "3.3.7", "vue-eslint-parser": "10.4.1", - "vue-tsc": "3.3.5" + "vue-tsc": "3.3.7" } } diff --git a/packages/frontend-embed/tsconfig.json b/packages/frontend-embed/tsconfig.json index 24fa71de19..d0d7591f23 100644 --- a/packages/frontend-embed/tsconfig.json +++ b/packages/frontend-embed/tsconfig.json @@ -13,6 +13,7 @@ "module": "ES2022", "moduleResolution": "Bundler", "removeComments": false, + "skipLibCheck": true, "noLib": false, "strict": true, "strictNullChecks": true, diff --git a/packages/frontend-embed/vite.config.ts b/packages/frontend-embed/vite.config.ts index ff00a7d385..81e5d3a2c6 100644 --- a/packages/frontend-embed/vite.config.ts +++ b/packages/frontend-embed/vite.config.ts @@ -1,7 +1,7 @@ import path from 'path'; import pluginVue from '@vitejs/plugin-vue'; import { defineConfig, type UserConfig } from 'vite'; -import * as yaml from 'js-yaml'; +import { load as loadYaml } from 'js-yaml'; import { promises as fsp } from 'fs'; import locales from 'i18n'; @@ -11,7 +11,7 @@ import pluginJson5 from './lib/vite-plugin-json5.js'; import { pluginRemoveUnrefI18n } from '../frontend-builder/rollup-plugin-remove-unref-i18n'; import { Features } from 'lightningcss'; -const url = process.env.NODE_ENV === 'development' ? (yaml.load(await fsp.readFile('../../.config/default.yml', 'utf-8')) as any).url : null; +const url = process.env.NODE_ENV === 'development' ? (loadYaml(await fsp.readFile('../../.config/default.yml', 'utf-8')) as any).url : null; const host = url ? (new URL(url)).hostname : undefined; const extensions = ['.ts', '.tsx', '.js', '.jsx', '.mjs', '.json', '.json5', '.svg', '.sass', '.scss', '.css', '.vue']; diff --git a/packages/frontend-shared/@types/global.d.ts b/packages/frontend-shared/@types/global.d.ts index 52081d07b3..165ca27bfe 100644 --- a/packages/frontend-shared/@types/global.d.ts +++ b/packages/frontend-shared/@types/global.d.ts @@ -14,9 +14,3 @@ declare const _PERF_PREFIX_: string; // for dev-mode declare const _LANGS_FULL_: string[][]; - -// TagCanvas -interface Window { - // eslint-disable-next-line @typescript-eslint/no-explicit-any - TagCanvas: any; -} diff --git a/packages/frontend-shared/js/interval.ts b/packages/frontend-shared/js/interval.ts new file mode 100644 index 0000000000..37abf8bb51 --- /dev/null +++ b/packages/frontend-shared/js/interval.ts @@ -0,0 +1,59 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +/** ドキュメントがアクティブになっていない間は実行を止めるsetInterval。戻り値の関数でdisposeする */ +export function createVisibilityAwareInterval(fn: () => void, interval: number, options: { + immediate?: boolean; +} = {}): () => void { + let intervalId: number | null = null; + let lastCalledAt: number | null = null; + + const tick = () => { + lastCalledAt = Date.now(); + fn(); + }; + + const start = () => { + if (intervalId != null) return; + if (lastCalledAt == null) { + if (options.immediate) { + tick(); + } else { + lastCalledAt = Date.now(); + } + } else if (Date.now() - lastCalledAt >= interval) { + // もし前回の呼び出しからinterval以上の時間が経過していたら、即座に呼び出す + // （非アクティブ→アクティブに戻った場合など） + tick(); + } + intervalId = window.setInterval(tick, interval); + }; + + const stop = () => { + if (intervalId != null) { + window.clearInterval(intervalId); + intervalId = null; + } + }; + + const onVisibilityChange = () => { + if (window.document.visibilityState === 'visible') { + start(); + } else { + stop(); + } + }; + + window.document.addEventListener('visibilitychange', onVisibilityChange); + + if (window.document.visibilityState === 'visible') { + start(); + } + + return () => { + window.document.removeEventListener('visibilitychange', onVisibilityChange); + stop(); + }; +} diff --git a/packages/frontend-shared/js/use-interval.ts b/packages/frontend-shared/js/use-interval.ts index b50e78c3cc..d9d00450e0 100644 --- a/packages/frontend-shared/js/use-interval.ts +++ b/packages/frontend-shared/js/use-interval.ts @@ -4,34 +4,47 @@ */ import { onActivated, onDeactivated, onMounted, onUnmounted } from 'vue'; +import { createVisibilityAwareInterval } from './interval.js'; export function useInterval(fn: () => void, interval: number, options: { immediate: boolean; afterMounted: boolean; + keepRunningWhenHidden?: boolean; }): (() => void) | undefined { if (Number.isNaN(interval)) return; - let intervalId: number | null = null; + let disposer: (() => void) | null = null; + + const start = () => { + if (options.keepRunningWhenHidden) { + if (options.immediate) fn(); + const intervalId = window.setInterval(fn, interval); + disposer = () => { + window.clearInterval(intervalId); + }; + } else { + disposer = createVisibilityAwareInterval(fn, interval, { immediate: options.immediate }); + } + }; + + const clear = () => { + if (disposer) { + disposer(); + disposer = null; + } + }; if (options.afterMounted) { onMounted(() => { - if (options.immediate) fn(); - intervalId = window.setInterval(fn, interval); + start(); }); } else { - if (options.immediate) fn(); - intervalId = window.setInterval(fn, interval); + start(); } - const clear = () => { - if (intervalId) window.clearInterval(intervalId); - intervalId = null; - }; - onActivated(() => { - if (intervalId) return; - if (options.immediate) fn(); - intervalId = window.setInterval(fn, interval); + if (disposer) return; + start(); }); onDeactivated(() => { diff --git a/packages/frontend-shared/package.json b/packages/frontend-shared/package.json index d35e380588..24ba6bddb2 100644 --- a/packages/frontend-shared/package.json +++ b/packages/frontend-shared/package.json @@ -4,14 +4,14 @@ "private": true, "scripts": { "eslint": "eslint './**/*.{js,jsx,ts,tsx}'", - "typecheck": "tsgo --noEmit", + "typecheck": "tsc --noEmit", "lint": "pnpm typecheck && pnpm eslint" }, "devDependencies": { - "@types/node": "26.0.0", + "@types/node": "26.1.1", "@types/tinycolor2": "1.4.6", - "@typescript-eslint/eslint-plugin": "8.61.1", - "@typescript-eslint/parser": "8.61.1", + "@typescript-eslint/eslint-plugin": "8.63.0", + "@typescript-eslint/parser": "8.63.0", "eslint-plugin-vue": "10.9.2", "vue-eslint-parser": "10.4.1" }, @@ -23,8 +23,8 @@ "i18n": "workspace:*", "json5": "2.2.3", "misskey-js": "workspace:*", - "shiki": "4.2.0", + "shiki": "4.3.1", "tinycolor2": "1.6.0", - "vue": "3.5.38" + "vue": "3.5.39" } } diff --git a/packages/frontend-shared/tsconfig.json b/packages/frontend-shared/tsconfig.json index c2600dc45e..e038b9ca23 100644 --- a/packages/frontend-shared/tsconfig.json +++ b/packages/frontend-shared/tsconfig.json @@ -9,6 +9,7 @@ "sourceMap": false, "outDir": "./js-built", "removeComments": true, + "skipLibCheck": true, "resolveJsonModule": true, "strict": true, "strictFunctionTypes": true, diff --git a/packages/frontend/.storybook/generate.tsx b/packages/frontend/.storybook/generate.tsx index 6005049dde..01ddefa544 100644 --- a/packages/frontend/.storybook/generate.tsx +++ b/packages/frontend/.storybook/generate.tsx @@ -458,7 +458,6 @@ function toStories(component: string): Promise { globSync('src/components/MkSignupServerRules.vue'), globSync('src/components/MkUserSetupDialog.vue'), globSync('src/components/MkUserSetupDialog.*.vue'), - globSync('src/components/MkImgPreviewDialog.vue'), globSync('src/components/MkInstanceCardMini.vue'), globSync('src/components/MkInviteCode.vue'), globSync('src/components/MkTagItem.vue'), diff --git a/packages/frontend/@types/global.d.ts b/packages/frontend/@types/global.d.ts index 8a067a78ec..7e2f8f5567 100644 --- a/packages/frontend/@types/global.d.ts +++ b/packages/frontend/@types/global.d.ts @@ -13,8 +13,3 @@ declare const _PERF_PREFIX_: string; // for dev-mode declare const _LANGS_FULL_: string[][]; - -// TagCanvas -interface Window { - TagCanvas: any; -} diff --git a/packages/frontend/@types/theme.d.ts b/packages/frontend/@types/theme.d.ts index 6ac1037493..6a010889d6 100644 --- a/packages/frontend/@types/theme.d.ts +++ b/packages/frontend/@types/theme.d.ts @@ -4,7 +4,7 @@ */ declare module '@@/themes/*.json5' { - import { Theme } from '@/theme.js'; + import { Theme } from '@@/js/theme.js'; const theme: Theme; diff --git a/packages/frontend/assets/artist_palette_3d.png b/packages/frontend/assets/artist_palette_3d.png deleted file mode 100644 index e815352ece..0000000000 Binary files a/packages/frontend/assets/artist_palette_3d.png and /dev/null differ diff --git a/packages/frontend/assets/bell_3d.png b/packages/frontend/assets/bell_3d.png deleted file mode 100644 index 2598cdd82b..0000000000 Binary files a/packages/frontend/assets/bell_3d.png and /dev/null differ diff --git a/packages/frontend/assets/cloud_3d.png b/packages/frontend/assets/cloud_3d.png deleted file mode 100644 index a3a1de12dd..0000000000 Binary files a/packages/frontend/assets/cloud_3d.png and /dev/null differ diff --git a/packages/frontend/assets/desktop_computer_3d.png b/packages/frontend/assets/desktop_computer_3d.png deleted file mode 100644 index 85e92a02c0..0000000000 Binary files a/packages/frontend/assets/desktop_computer_3d.png and /dev/null differ diff --git a/packages/frontend/assets/electric_plug_3d.png b/packages/frontend/assets/electric_plug_3d.png deleted file mode 100644 index 431ef68c85..0000000000 Binary files a/packages/frontend/assets/electric_plug_3d.png and /dev/null differ diff --git a/packages/frontend/assets/gear_3d.png b/packages/frontend/assets/gear_3d.png deleted file mode 100644 index 050340b76c..0000000000 Binary files a/packages/frontend/assets/gear_3d.png and /dev/null differ diff --git a/packages/frontend/assets/link_3d.png b/packages/frontend/assets/link_3d.png deleted file mode 100644 index b1cb23080a..0000000000 Binary files a/packages/frontend/assets/link_3d.png and /dev/null differ diff --git a/packages/frontend/assets/locked_with_key_3d.png b/packages/frontend/assets/locked_with_key_3d.png deleted file mode 100644 index aae99b982d..0000000000 Binary files a/packages/frontend/assets/locked_with_key_3d.png and /dev/null differ diff --git a/packages/frontend/assets/mens_room_3d.png b/packages/frontend/assets/mens_room_3d.png deleted file mode 100644 index 8b85ca8782..0000000000 Binary files a/packages/frontend/assets/mens_room_3d.png and /dev/null differ diff --git a/packages/frontend/assets/musical_note_3d.png b/packages/frontend/assets/musical_note_3d.png deleted file mode 100644 index 0b520311f6..0000000000 Binary files a/packages/frontend/assets/musical_note_3d.png and /dev/null differ diff --git a/packages/frontend/assets/package_3d.png b/packages/frontend/assets/package_3d.png deleted file mode 100644 index 582134fd2f..0000000000 Binary files a/packages/frontend/assets/package_3d.png and /dev/null differ diff --git a/packages/frontend/assets/prohibited_3d.png b/packages/frontend/assets/prohibited_3d.png deleted file mode 100644 index 1f071edd06..0000000000 Binary files a/packages/frontend/assets/prohibited_3d.png and /dev/null differ diff --git a/packages/frontend/assets/speaker_high_volume_3d.png b/packages/frontend/assets/speaker_high_volume_3d.png deleted file mode 100644 index b25aaa91d6..0000000000 Binary files a/packages/frontend/assets/speaker_high_volume_3d.png and /dev/null differ diff --git a/packages/frontend/assets/tagcanvas.min.js b/packages/frontend/assets/tagcanvas.min.js deleted file mode 100644 index bcee46e682..0000000000 --- a/packages/frontend/assets/tagcanvas.min.js +++ /dev/null @@ -1,21 +0,0 @@ -/** - * Copyright (C) 2010-2021 Graham Breach - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the GNU Lesser General Public License as published by - * the Free Software Foundation, either version 3 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public License - * along with this program. If not, see . - */ -/** - * TagCanvas 2.11 - * For more information, please contact - */ - (function(){"use strict";var r,C,p=Math.abs,o=Math.sin,l=Math.cos,g=Math.max,h=Math.min,af=Math.ceil,E=Math.sqrt,w=Math.pow,I={},D={},R={0:"0,",1:"17,",2:"34,",3:"51,",4:"68,",5:"85,",6:"102,",7:"119,",8:"136,",9:"153,",a:"170,",A:"170,",b:"187,",B:"187,",c:"204,",C:"204,",d:"221,",D:"221,",e:"238,",E:"238,",f:"255,",F:"255,"},f,d,b,T,z,F,M,c=document,v,e,P,j={};for(r=0;r<256;++r)C=r.toString(16),r<16&&(C='0'+C),D[C]=D[C.toUpperCase()]=r.toString()+',';function n(a){return typeof a!='undefined'}function B(a){return typeof a=='object'&&a!=null}function G(a,c,b){return isNaN(a)?b:h(b,g(c,a))}function x(){return!1}function q(){return(new Date).valueOf()}function ak(c,d){var b=[],e=c.length,a;for(a=0;a=1)?0:a<=-1?Math.PI:Math.acos(a)},z.unit=function(){var a=this.length();return new s(this.x/a,this.y/a,this.z/a)};function ay(b,a){a=a*Math.PI/180,b=b*Math.PI/180;var c=o(b)*l(a),d=-o(a),e=-l(b)*l(a);return new s(c,d,e)}function m(a){this[1]={1:a[0],2:a[1],3:a[2]},this[2]={1:a[3],2:a[4],3:a[5]},this[3]={1:a[6],2:a[7],3:a[8]}}T=m.prototype,m.Identity=function(){return new m([1,0,0,0,1,0,0,0,1])},m.Rotation=function(e,a){var c=o(e),d=l(e),b=1-d;return new m([d+w(a.x,2)*b,a.x*a.y*b-a.z*c,a.x*a.z*b+a.y*c,a.y*a.x*b+a.z*c,d+w(a.y,2)*b,a.y*a.z*b-a.x*c,a.z*a.x*b-a.y*c,a.z*a.y*b+a.x*c,d+w(a.z,2)*b])},T.mul=function(c){var d=[],a,b,e=c.xform?1:0;for(a=1;a<=3;++a)for(b=1;b<=3;++b)e?d.push(this[a][1]*c[1][b]+this[a][2]*c[2][b]+this[a][3]*c[3][b]):d.push(this[a][b]*c);return new m(d)},T.xform=function(b){var a={},c=b.x,d=b.y,e=b.z;return a.x=c*this[1][1]+d*this[2][1]+e*this[3][1],a.y=c*this[1][2]+d*this[2][2]+e*this[3][2],a.z=c*this[1][3]+d*this[2][3]+e*this[3][3],a};function aB(g,j,k,m,f){var a,b,c,d,e=[],h=2/g,i;i=Math.PI*(3-E(5)+(parseFloat(f)?parseFloat(f):0));for(a=0;a0)}function aC(a,c,f,d){var e=a.createLinearGradient(0,0,c,0),b;for(b in d)e.addColorStop(1-b,d[b]);a.fillStyle=e,a.fillRect(0,f,c,1)}function L(a,m,j){var l=1024,d=1,e=a.weightGradient,i,f,b,c;if(a.gCanvas)f=a.gCanvas.getContext('2d'),d=a.gCanvas.height;else{if(B(e[0])?d=e.length:e=[e],a.gCanvas=i=k(l,d),!i)return null;f=i.getContext('2d');for(b=0;b0?b=i*b/100:b=b*j,a=e.getContext('2d'),a.globalCompositeOperation='source-over',a.fillStyle='#fff',b>=i/2?(b=h(c,d)/2,a.beginPath(),a.moveTo(c/2,d/2),a.arc(c/2,d/2,b,0,2*Math.PI,!1),a.fill(),a.closePath()):(b=h(c/2,d/2,b),y(a,0,0,c,d,b,!0),a.fill()),a.globalCompositeOperation='source-in',a.drawImage(l,0,0,c,d),e)}function ao(q,m,i,b,h,a,c){var g=p(c[0]),f=p(c[1]),j=m+(g>a?g+a:a*2)*b,l=i+(f>a?f+a:a*2)*b,n=b*((a||0)+(c[0]<0?g:0)),o=b*((a||0)+(c[1]<0?f:0)),e,d;return e=k(j,l),!e?null:(d=e.getContext('2d'),h&&(d.shadowColor=h),a&&(d.shadowBlur=a*b),c&&(d.shadowOffsetX=c[0]*b,d.shadowOffsetY=c[1]*b),d.drawImage(q,n,o,m,i),{image:e,width:j/b,height:l/b})}function ae(m,o,l){var c=parseInt(m.toString().length*l),h=parseInt(l*2*m.length),j=k(c,h),g,i,e,f,b,d,n,a;if(!j)return null;g=j.getContext('2d'),g.fillStyle='#000',g.fillRect(0,0,c,h),Y(g,l+'px '+o,'#fff',m,0,0,0,0,[],'centre'),i=g.getImageData(0,0,c,h),e=i.width,f=i.height,a={min:{x:e,y:f},max:{x:-1,y:-1}};for(d=0;d0&&(ba.max.x&&(a.max.x=b),da.max.y&&(a.max.y=d));return e!=c&&(a.min.x*=c/e,a.max.x*=c/e),f!=h&&(a.min.y*=c/f,a.max.y*=c/f),j=null,a}function Q(a){return"'"+a.replace(/(\'|\")/g,'').replace(/\s*,\s*/g,"', '")+"'"}function t(b,d,a){a=a||c,a.addEventListener?a.addEventListener(b,d,!1):a.attachEvent('on'+b,d)}function am(b,d,a){a=a||c,a.removeEventListener?a.removeEventListener(b,d):a.detachEvent('on'+b,d)}function A(g,e,j,a,b){var l=b.imageScale,h,c,k,m,f,d;if(!e.complete)return t('load',function(){A(g,e,j,a,b)},e);if(!g.complete)return t('load',function(){A(g,e,j,a,b)},g);if(j&&!j.complete)return t('load',function(){A(g,e,j,a,b)},j);e.width=e.width,e.height=e.height,l&&(g.width=e.width*l,g.height=e.height*l),a.iw=g.width,a.ih=g.height,b.txtOpt&&(c=g,h=b.zoomMax*b.txtScale,f=a.iw*h,d=a.ih*h,f0?(a.iw+=2*b.outlineIncrease,a.ih+=2*b.outlineIncrease,f=h*a.iw,d=h*a.ih,c=S(a.fimage,f,d),a.oimage=c,a.fimage=H(a.fimage,a.oimage.width,a.oimage.height)):(f=h*(a.iw+2*b.outlineIncrease),d=h*(a.ih+2*b.outlineIncrease),c=S(a.fimage,f,d),a.oimage=H(c,a.fimage.width,a.fimage.height))))),a.alt=j,a.Init()}function i(a,d){var b=c.defaultView,e=d.replace(/\-([a-z])/g,function(a){return a.charAt(1).toUpperCase()});return b&&b.getComputedStyle&&b.getComputedStyle(a,null).getPropertyValue(d)||a.currentStyle&&a.currentStyle[e]}function aj(c,d,e){var b=1,a;return d?b=1*(c.getAttribute(d)||e):(a=i(c,'font-size'))&&(b=a.indexOf('px')>-1&&a.replace('px','')*1||a.indexOf('pt')>-1&&a.replace('pt','')*1.25||a*3.3),b}function u(a){return a.target&&n(a.target.id)?a.target.id:a.srcElement.parentNode.id}function K(a,c){var b,d,e=parseInt(i(c,'width'))/c.width,f=parseInt(i(c,'height'))/c.height;return n(a.offsetX)?b={x:a.offsetX,y:a.offsetY}:(d=X(c.id),n(a.changedTouches)&&(a=a.changedTouches[0]),a.pageX&&(b={x:a.pageX-d.x,y:a.pageY-d.y})),b&&e&&f&&(b.x/=e,b.y/=f),b}function an(c){var d=c.target||c.fromElement.parentNode,b=a.tc[d.id];b&&(b.mx=b.my=-1,b.UnFreeze(),b.EndDrag())}function ad(e){var g,c=a,b,d,f=u(e);for(g in c.tc)b=c.tc[g],b.tttimer&&(clearTimeout(b.tttimer),b.tttimer=null);f&&c.tc[f]&&(b=c.tc[f],(d=K(e,b.canvas))&&(b.mx=d.x,b.my=d.y,b.Drag(e,d)),b.drawn=0)}function ap(b){var e=a,f=c.addEventListener?0:1,d=u(b);d&&b.button==f&&e.tc[d]&&e.tc[d].BeginDrag(b)}function aq(b){var f=a,g=c.addEventListener?0:1,e=u(b),d;e&&b.button==g&&f.tc[e]&&(d=f.tc[e],ad(b),!d.EndDrag()&&!d.touchState&&d.Clicked(b))}function ar(c){var e=u(c),b=e&&a.tc[e],d;b&&c.changedTouches&&(c.touches.length==1&&b.touchState==0?(b.touchState=1,b.BeginDrag(c),(d=K(c,b.canvas))&&(b.mx=d.x,b.my=d.y,b.drawn=0)):c.targetTouches.length==2&&b.pinchZoom?(b.touchState=3,b.EndDrag(),b.BeginPinch(c)):(b.EndDrag(),b.EndPinch(),b.touchState=0))}function ac(c){var d=u(c),b=d&&a.tc[d];if(b&&c.changedTouches){switch(b.touchState){case 1:b.Draw(),b.Clicked();break;break;case 2:b.EndDrag();break;case 3:b.EndPinch()}b.touchState=0}}function au(c){var f,e=a,b,d,g=u(c);for(f in e.tc)b=e.tc[f],b.tttimer&&(clearTimeout(b.tttimer),b.tttimer=null);if(b=g&&e.tc[g],b&&c.changedTouches&&b.touchState){switch(b.touchState){case 1:case 2:(d=K(c,b.canvas))&&(b.mx=d.x,b.my=d.y,b.Drag(c,d)&&(b.touchState=2));break;case 3:b.Pinch(c)}b.drawn=0}}function ab(b){var d=a,c=u(b);c&&d.tc[c]&&(b.cancelBubble=!0,b.returnValue=!1,b.preventDefault&&b.preventDefault(),d.tc[c].Wheel((b.wheelDelta||b.detail)>0))}function aw(d){var c,b=a;clearTimeout(b.scrollTimer);for(c in b.tc)b.tc[c].Pause();b.scrollTimer=setTimeout(function(){var b,c=a;for(b in c.tc)c.tc[b].Resume()},b.scrollPause)}function al(){Z(q())}function Z(b){var c=a.tc,d;a.NextFrame(a.interval),b=b||q();for(d in c)c[d].Draw(b)}function az(){requestAnimationFrame(Z)}function aA(a){setTimeout(al,a)}function X(f){var g=c.getElementById(f),b=g.getBoundingClientRect(),a=c.documentElement,d=c.body,e=window,h=e.pageXOffset||a.scrollLeft,i=e.pageYOffset||a.scrollTop,j=a.clientLeft||d.clientLeft,k=a.clientTop||d.clientTop;return{x:b.left+h-j,y:b.top+i-k}}function aI(a,b,d,e){var c=a.radius*a.z1/(a.z1+a.z2+b.z);return{x:b.x*c*d,y:b.y*c*e,z:b.z,w:(a.z1-b.z)/a.z2}}function V(a){this.e=a,this.br=0,this.line=[],this.text=[],this.original=a.innerText||a.textContent}F=V.prototype,F.Empty=function(){for(var a=0;ah?(d.push(this.line.join(' ')),this.line=[a[b]]):this.line.push(a[b]);d.push(this.line.join(' '))}return this.text=d};function _(a,b){this.ts=null,this.tc=a,this.tag=b,this.x=this.y=this.w=this.h=this.sc=1,this.z=0,this.pulse=1,this.pulsate=a.pulsateTo<1,this.colour=a.outlineColour,this.adash=~~a.outlineDash,this.agap=~~a.outlineDashSpace||this.adash,this.aspeed=a.outlineDashSpeed*1,this.colour=='tag'?this.colour=i(b.a,'color'):this.colour=='tagbg'&&(this.colour=i(b.a,'background-color')),this.Draw=this.pulsate?this.DrawPulsate:this.DrawSimple,this.radius=a.outlineRadius|0,this.SetMethod(a.outlineMethod,a.altImage)}f=_.prototype,f.SetMethod=function(a,d){var b={block:['PreDraw','DrawBlock'],colour:['PreDraw','DrawColour'],outline:['PostDraw','DrawOutline'],classic:['LastDraw','DrawOutline'],size:['PreDraw','DrawSize'],none:['LastDraw']},c=b[a]||b.outline;a=='none'?this.Draw=function(){return 1}:this.drawFunc=this[c[1]],this[c[0]]=this.Draw,d&&(this.RealPreDraw=this.PreDraw,this.PreDraw=this.DrawAlt)},f.Update=function(d,e,i,j,a,f,g,h){var b=this.tc.outlineOffset,c=2*b;this.x=a*d+g-b,this.y=a*e+h-b,this.w=a*i+c,this.h=a*j+c,this.sc=a,this.z=f},f.Ants=function(k){if(!this.adash)return;var b=this.adash,c=this.agap,a=this.aspeed,j=b+c,h=0,g=b,f=c,i=0,d=0,e;a&&(d=p(a)*(q()-this.ts)/50,a<0&&(d=864e4-d),a=~~d%j),a?(b>=a?(h=b-a,g=a):(f=j-a,i=c-f),e=[h,f,g,i]):e=[b,c],k.setLineDash(e)},f.DrawOutline=function(a,d,e,b,c,f){var g=h(this.radius,c/2,b/2);a.strokeStyle=f,this.Ants(a),y(a,d,e,b,c,g,!0)},f.DrawSize=function(i,n,m,l,k,j,a,h,g){var f=a.w,e=a.h,c,b,d;return this.pulsate?(a.image?d=(a.image.height+this.tc.outlineIncrease)/a.image.height:d=a.oscale,b=a.fimage||a.image,c=1+(d-1)*(1-this.pulse),a.h*=c,a.w*=c):b=a.oimage,a.alpha=1,a.Draw(i,h,g,b),a.h=e,a.w=f,1},f.DrawColour=function(d,h,i,e,f,g,a,b,c){return a.oimage?(this.pulse<1?(a.alpha=1-w(this.pulse,2),a.Draw(d,b,c,a.fimage),a.alpha=this.pulse):a.alpha=1,a.Draw(d,b,c,a.oimage),1):this[a.image?'DrawColourImage':'DrawColourText'](d,h,i,e,f,g,a,b,c)},f.DrawColourText=function(f,h,i,j,g,e,a,b,c){var d=a.colour;return a.colour=e,a.alpha=1,a.Draw(f,b,c),a.colour=d,1},f.DrawColourImage=function(a,q,p,o,n,m,i,r,l){var f=a.canvas,e=~~g(q,0),d=~~g(p,0),c=h(f.width-e,o)+.5|0,b=h(f.height-d,n)+.5|0,j;return v?(v.width=c,v.height=b):v=k(c,b),!v?this.SetMethod('outline'):(j=v.getContext('2d'),j.drawImage(f,e,d,c,b,0,0,c,b),a.clearRect(e,d,c,b),this.pulsate?i.alpha=1-w(this.pulse,2):i.alpha=1,i.Draw(a,r,l),a.setTransform(1,0,0,1,0,0),a.save(),a.beginPath(),a.rect(e,d,c,b),a.clip(),a.globalCompositeOperation='source-in',a.fillStyle=m,a.fillRect(e,d,c,b),a.restore(),a.globalAlpha=1,a.globalCompositeOperation='destination-over',a.drawImage(v,0,0,c,b,e,d,c,b),a.globalCompositeOperation='source-over',1)},f.DrawAlt=function(b,a,c,d,f,g){var e=this.RealPreDraw(b,a,c,d,f,g);return a.alt&&(a.DrawImage(b,c,d,a.alt),e=1),e},f.DrawBlock=function(a,d,e,b,c,f){var g=h(this.radius,c/2,b/2);a.fillStyle=f,y(a,d,e,b,c,g)},f.DrawSimple=function(a,b,c,d,e,f){var g=this.tc;return a.setTransform(1,0,0,1,0,0),a.strokeStyle=this.colour,a.lineWidth=g.outlineThickness,a.shadowBlur=a.shadowOffsetX=a.shadowOffsetY=0,a.globalAlpha=f?e:1,this.drawFunc(a,this.x,this.y,this.w,this.h,this.colour,b,c,d)},f.DrawPulsate=function(h,d,e,f){var g=q()-this.ts,c=this.tc,b=c.pulsateTo+(1-c.pulsateTo)*(.5+l(2*Math.PI*g/(1e3*c.pulsateTime))/2);return this.pulse=b=a.Smooth(1,b),this.DrawSimple(h,d,e,f,b,1)},f.Active=function(d,a,b){var c=a>=this.x&&b>=this.y&&a<=this.x+this.w&&b<=this.y+this.h;return c?this.ts=this.ts||q():this.ts=null,c},f.PreDraw=f.PostDraw=f.LastDraw=x;function J(a,h,c,b,e,f,g,d,i,j,k,l,m,n){this.tc=a,this.image=null,this.text=h,this.text_original=n,this.line_widths=[],this.title=c.title||null,this.a=c,this.position=new s(b[0],b[1],b[2]),this.x=this.y=this.z=0,this.w=e,this.h=f,this.colour=g||a.textColour,this.bgColour=d||a.bgColour,this.bgRadius=i|0,this.bgOutline=j||this.colour,this.bgOutlineThickness=k|0,this.textFont=l||a.textFont,this.padding=m|0,this.sc=this.alpha=1,this.weighted=!a.weight,this.outline=new _(a,this),this.audio=null}d=J.prototype,d.Init=function(b){var a=this.tc;this.textHeight=a.textHeight,this.HasText()?this.Measure(a.ctxt,a):(this.w=this.iw,this.h=this.ih),this.SetShadowColour=a.shadowAlpha?this.SetShadowColourAlpha:this.SetShadowColourFixed,this.SetDraw(a)},d.Draw=x,d.HasText=function(){return this.text&&this.text[0].length>0},d.EqualTo=function(a){var b=a.getElementsByTagName('img');return this.a.href!=a.href?0:b.length?this.image.src==b[0].src:(a.innerText||a.textContent)==this.text_original},d.SetImage=function(a){this.image=this.fimage=a},d.SetAudio=function(a){this.audio=a,this.audio.load()},d.SetDraw=function(a){this.Draw=this.fimage?a.ie>7?this.DrawImageIE:this.DrawImage:this.DrawText,a.noSelect&&(this.CheckActive=x)},d.MeasureText=function(d){var a,e=this.text.length,b=0,c;for(a=0;a0?c=H(c,this.oimage.width,this.oimage.height):this.oimage=H(this.oimage,c.width,c.height)),c&&(this.fimage=c,l=this.fimage.width/b,j=this.fimage.height/b),this.SetDraw(a),a.txtOpt=!!this.fimage),this.h=j,this.w=l},d.SetFont=function(a,b,c,d){this.textFont=a,this.colour=b,this.bgColour=c,this.bgOutline=d,this.Measure(this.tc.ctxt,this.tc)},d.SetWeight=function(c){var b=this.tc,e=b.weightMode.split(/[, ]/),d,a,f=c.length;if(!this.HasText())return;this.weighted=!0;for(a=0;a0&&a.weightSizeMax>a.weightSizeMin?this.textHeight=a.weightSize*(a.weightSizeMin+(a.weightSizeMax-a.weightSizeMin)*c):this.textHeight=g(1,b*a.weightSize))},d.SetShadowColourFixed=function(a,b,c){a.shadowColor=b},d.SetShadowColourAlpha=function(a,b,c){a.shadowColor=aE(b,c)},d.DrawText=function(a,h,i){var e=this.tc,g=this.x,f=this.y,c=this.sc,b,d;a.globalAlpha=this.alpha,a.fillStyle=this.colour,e.shadow&&this.SetShadowColour(a,e.shadow,this.alpha),a.font=this.font,g+=h/c,f+=i/c-this.h/2;for(b=0;b{this.stopped?this.audio.pause():this.playing=1}),1}};function a(f,o,k){var d,i,b=c.getElementById(f),l=['id','class','innerHTML'];if(!b)throw 0;if(n(window.G_vmlCanvasManager)&&(b=window.G_vmlCanvasManager.initElement(b),this.ie=parseFloat(navigator.appVersion.split('MSIE')[1])),b&&(!b.getContext||!b.getContext('2d').fillText)){i=c.createElement('DIV');for(d=0;d0?a.scrollPause=~~this.scrollPause:this.scrollPause=0,this.minTags>0&&this.repeatTags<1&&(d=this.GetTags().length)&&(this.repeatTags=af(this.minTags/d)-1),this.transform=m.Identity(),this.startTime=this.time=q(),this.mx=this.my=-1,this.centreImage&&av(this),this.Animate=this.dragControl?this.AnimateDrag:this.AnimatePosition,this.animTiming=typeof a[this.animTiming]=='function'?a[this.animTiming]:a.Smooth,this.shadowBlur||this.shadowOffset[0]||this.shadowOffset[1]?(this.ctxt.shadowColor=this.shadow,this.shadow=this.ctxt.shadowColor,this.shadowAlpha=aD()):delete this.shadow,this.activeAudio===!1?e='off':this.activeAudio&&this.LoadAudio(),this.Load(),o&&this.hideTags&&function(b){a.loaded?b.HideTags():t('load',function(){b.HideTags()},window)}(this),this.yaw=this.initial?this.initial[0]*this.maxSpeed:0,this.pitch=this.initial?this.initial[1]*this.maxSpeed:0,this.tooltip?(this.ctitle=b.title,b.title='',this.tooltip=='native'?this.Tooltip=this.TooltipNative:(this.Tooltip=this.TooltipDiv,this.ttdiv||(this.ttdiv=c.createElement('div'),this.ttdiv.className=this.tooltipClass,this.ttdiv.style.position='absolute',this.ttdiv.style.zIndex=b.style.zIndex+1,t('mouseover',function(a){a.target.style.display='none'},this.ttdiv),c.body.appendChild(this.ttdiv)))):this.Tooltip=this.TooltipNone,!this.noMouse&&!j[f]){j[f]=[['mousemove',ad],['mouseout',an],['mouseup',aq],['touchstart',ar],['touchend',ac],['touchcancel',ac],['touchmove',au]],this.dragControl&&(j[f].push(['mousedown',ap]),j[f].push(['selectstart',x])),this.wheelZoom&&(j[f].push(['mousewheel',ab]),j[f].push(['DOMMouseScroll',ab])),this.scrollPause&&j[f].push(['scroll',aw,window]);for(d=0;dthis.max_weight[a])&&(this.max_weight[a]=c),(!this.min_weight[a]||cthis.min_weight[a]&&(g=1);if(g)for(b=0;b=d&&this.my>=e)return!0},b.ToggleAudio=function(){var a=this.audioOff||e&&e.state==='suspended';a||this.currentAudio&&this.currentAudio.StopAudio(),this.audioOff=!a},b.Draw=function(s){if(this.paused)return;var l=this.canvas,i=l.width,j=l.height,q=0,p=(s-this.time)*a.interval/1e3,h=i/2+this.offsetX,g=j/2+this.offsetY,d=this.ctxt,b,f,c,o=-1,e=this.taglist,k=e.length,t=this.active&&this.active.tag,m='',u=this.frontSelect,r=this.centreFunc==x,n;if(this.time=s,this.frozen&&this.drawn)return this.Animate(i,j,p);n=this.AnimateFixed(),d.setTransform(1,0,0,1,0,0);for(c=0;c=0&&this.my>=0&&this.taglist[c].CheckActive(d,h,g),f&&f.sc>q&&(!u||f.z<=0)&&(b=f,o=c,b.tag=this.taglist[c],q=f.sc);this.active=b}this.txtOpt||this.shadow&&this.SetShadow(d),d.clearRect(0,0,i,j);for(c=0;c=this.fadeIn?(this.fadeIn=0,this.fixedAlpha=1):this.fixedAlpha=b/this.fadeIn),this.fixedAnim)&&(this.fixedAnim.transform||(this.fixedAnim.transform=this.transform),a=this.fixedAnim,b=q()-a.t0,c=a.angle,d,e=this.animTiming(a.t,b),this.transform=a.transform,b>=a.t?(this.fixedCallbackTag=a.tag,this.fixedCallback=a.cb,this.fixedAnim=this.yaw=this.pitch=0):c*=e,d=m.Rotation(c,a.axis),this.transform=this.transform.mul(d),this.fixedAnim!=0)},b.AnimatePosition=function(g,h,f){var a=this,d=a.mx,e=a.my,b,c;!a.frozen&&d>=0&&e>=0&&db&&(a.yaw=c>a.z0?a.yaw*a.decel:0),!a.ly&&d>b&&(a.pitch=d>a.z0?a.pitch*a.decel:0)},b.Zoom=function(a){this.z2=this.z1*(1/a),this.drawn=0},b.Clicked=function(b){if(this.CheckAudioIcon()){this.ToggleAudio();return}var a=this.active;try{a&&a.tag&&(this.clickToFront===!1||this.clickToFront===null?a.tag.Clicked(b):this.TagToFront(a.tag,this.clickToFront,function(){a.tag.Clicked(b)},!0))}catch(a){}},b.Wheel=function(a){var b=this.zoom+this.zoomStep*(a?1:-1);this.zoom=h(this.zoomMax,g(this.zoomMin,b)),this.Zoom(this.zoom)},b.BeginDrag=function(a){this.down=K(a,this.canvas),a.cancelBubble=!0,a.returnValue=!1,a.preventDefault&&a.preventDefault()},b.Drag=function(e,a){if(this.dragControl&&this.down){var d=this.dragThreshold*this.dragThreshold,b=a.x-this.down.x,c=a.y-this.down.y;(this.dragging||b*b+c*c>d)&&(this.dx=b,this.dy=c,this.dragging=1,this.down=a)}return this.dragging},b.EndDrag=function(){var a=this.dragging;return this.dragging=this.down=null,a};function ah(a){var b=a.targetTouches[0],c=a.targetTouches[1];return E(w(c.pageX-b.pageX,2)+w(c.pageY-b.pageY,2))}b.BeginPinch=function(a){this.pinched=[ah(a),this.zoom],a.preventDefault&&a.preventDefault()},b.Pinch=function(d){var b,c,a=this.pinched;if(!a)return;c=ah(d),b=a[1]*c/a[0],this.zoom=h(this.zoomMax,g(this.zoomMin,b)),this.Zoom(this.zoom)},b.EndPinch=function(a){this.pinched=null},b.Pause=function(){this.paused=!0},b.Resume=function(){this.paused=!1},b.SetSpeed=function(a){this.initial=a,this.yaw=a[0]*this.maxSpeed,this.pitch=a[1]*this.maxSpeed},b.FindTag=function(a){if(!n(a))return null;if(n(a.index)&&(a=a.index),!B(a))return this.taglist[a];var c,d,b;n(a.id)?(c='id',d=a.id):n(a.text)&&(c='innerText',d=a.text);for(b=0;b { import { c as api, d as store, i as i18n, aD as notePage, bN as ImgWithBlurhash, bY as getStaticImageUrl, _ as _export_sfc } from './app-!~{001}~.js'; import { M as MkContainer } from './MkContainer-!~{03M}~.js'; import { b as defineComponent, a as ref, e as onMounted, z as resolveComponent, g as openBlock, h as createBlock, i as withCtx, K as createTextVNode, E as toDisplayString, u as unref, l as createBaseVNode, q as normalizeClass, B as createCommentVNode, k as createElementBlock, F as Fragment, C as renderList, A as createVNode } from './vue-!~{002}~.js'; -import './photoswipe-!~{003}~.js'; const _hoisted_1 = /* @__PURE__ */ createBaseVNode("i", { class: "ti ti-photo" }, null, -1); const _sfc_main = /* @__PURE__ */ defineComponent({ @@ -179,7 +178,6 @@ export { index_photos as default }; import { c as api, d as store, i as i18n, aD as notePage, bN as ImgWithBlurhash, bY as getStaticImageUrl, _ as _export_sfc } from './app-!~{001}~.js'; import { M as MkContainer } from './MkContainer-!~{03M}~.js'; import { b as defineComponent, a as ref, e as onMounted, z as resolveComponent, g as openBlock, h as createBlock, i as withCtx, K as createTextVNode, E as toDisplayString, u as unref, l as createBaseVNode, q as normalizeClass, B as createCommentVNode, k as createElementBlock, F as Fragment, C as renderList, A as createVNode } from './vue-!~{002}~.js'; -import './photoswipe-!~{003}~.js'; const _hoisted_1 = /* @__PURE__ */ createBaseVNode("i", { class: "ti ti-photo" }, null, -1); const index_photos = /* @__PURE__ */ defineComponent({ diff --git a/packages/frontend/package.json b/packages/frontend/package.json index 40e14b0431..1b547cf51c 100644 --- a/packages/frontend/package.json +++ b/packages/frontend/package.json @@ -6,11 +6,13 @@ "watch": "vite", "build": "tsx build.ts", "storybook-dev": "nodemon --verbose --watch src --ext \"mdx,ts,vue\" --ignore \"*.stories.ts\" --exec \"pnpm build-storybook-pre && pnpm exec storybook dev -p 6006 --ci\"", - "build-storybook-pre": "(tsgo -p .storybook || echo done.) && node .storybook/generate.js && node .storybook/preload-locale.js && node .storybook/preload-theme.js", + "build-storybook-pre": "(tsc -p .storybook || echo done.) && node .storybook/generate.js && node .storybook/preload-locale.js && node .storybook/preload-theme.js", "build-storybook": "pnpm build-storybook-pre && storybook build --webpack-stats-json storybook-static", "chromatic": "chromatic", - "test": "vitest --run --globals", - "test-and-coverage": "vitest --run --coverage --globals", + "test": "vitest --run --globals --config vitest.config.unit.ts", + "test-and-coverage": "vitest --run --coverage --globals --config vitest.config.unit.ts", + "test:e2e": "playwright test --config playwright.config.ts", + "test:browser": "vitest --run --globals --config vitest.config.browser.ts", "typecheck": "vue-tsc --noEmit", "eslint": "eslint --quiet \"src/**/*.{ts,vue}\"", "lint": "pnpm typecheck && pnpm eslint" @@ -20,7 +22,8 @@ "@mcaptcha/core-glue": "0.1.0-alpha-5", "@misskey-dev/browser-image-resizer": "2024.1.0", "@misskey-dev/emoji-data": "17.0.3", - "@sentry/vue": "10.59.0", + "@misskey-dev/tagcanvas-es": "0.1.2", + "@sentry/vue": "10.65.0", "@simplewebauthn/browser": "13.3.0", "@syuilo/aiscript": "1.2.1", "@syuilo/aiscript-0-19-0": "npm:@syuilo/aiscript@^0.19.0", @@ -32,55 +35,53 @@ "canvas-confetti": "1.9.4", "chart.js": "4.5.1", "chartjs-adapter-date-fns": "3.0.0", - "chartjs-chart-matrix": "3.0.4", + "chartjs-chart-matrix": "3.0.5", "chartjs-plugin-gradient": "0.6.1", "chartjs-plugin-zoom": "2.2.0", - "chromatic": "17.5.0", "compare-versions": "6.1.1", "cropperjs": "2.1.1", "date-fns": "4.4.0", "eventemitter3": "5.0.4", - "execa": "9.6.1", "exifreader": "4.41.0", "frontend-shared": "workspace:*", "hls.js": "1.6.16", "i18n": "workspace:*", "icons-subsetter": "workspace:*", - "idb-keyval": "6.2.5", + "idb-keyval": "6.3.0", "insert-text-at-cursor": "0.3.0", "ios-haptics": "0.1.5", "is-file-animated": "1.0.2", "json5": "2.2.3", "matter-js": "0.20.0", - "mediabunny": "1.49.0", + "mediabunny": "1.50.8", "mfm-js": "0.26.0", "misskey-bubble-game": "workspace:*", "misskey-js": "workspace:*", "misskey-reversi": "workspace:*", "misskey-world": "workspace:*", "frontend-misskey-world-engine": "workspace:*", - "photoswipe": "5.4.4", "punycode.js": "2.3.1", "qr-code-styling": "1.9.2", "qr-scanner": "1.4.2", - "sanitize-html": "2.17.5", - "shiki": "4.2.0", + "sanitize-html": "2.17.6", + "seedrandom": "3.0.5", + "shiki": "4.3.1", "textarea-caret": "3.1.0", - "three": "0.184.0", "throttle-debounce": "5.0.2", "tinycolor2": "1.6.0", "v-code-diff": "1.13.1", - "vue": "3.5.38", + "vue": "3.5.39", "wanakana": "5.3.1" }, "devDependencies": { "@misskey-dev/emoji-assets": "17.0.3", "@misskey-dev/summaly": "5.5.1", + "@playwright/test": "1.61.1", "@rollup/plugin-json": "6.1.0", "@rollup/pluginutils": "5.4.0", "@storybook/addon-essentials": "8.6.18", "@storybook/addon-interactions": "8.6.18", - "@storybook/addon-links": "10.4.6", + "@storybook/addon-links": "10.5.0", "@storybook/addon-mdx-gfm": "8.6.18", "@storybook/addon-storysource": "8.6.18", "@storybook/blocks": "8.6.18", @@ -91,8 +92,8 @@ "@storybook/test": "8.6.18", "@storybook/theming": "8.6.18", "@storybook/types": "8.6.18", - "@storybook/vue3": "10.4.6", - "@storybook/vue3-vite": "10.4.6", + "@storybook/vue3": "10.5.0", + "@storybook/vue3-vite": "10.5.0", "@tabler/icons-webfont": "3.35.0", "@testing-library/vue": "8.1.0", "@types/canvas-confetti": "1.9.0", @@ -100,46 +101,50 @@ "@types/insert-text-at-cursor": "0.3.2", "@types/matter-js": "0.20.2", "@types/micromatch": "4.0.10", - "@types/node": "26.0.0", + "@types/node": "26.1.1", "@types/punycode.js": "npm:@types/punycode@2.1.4", "@types/sanitize-html": "2.16.1", "@types/seedrandom": "3.0.8", "@types/textarea-caret": "3.0.4", "@types/throttle-debounce": "5.0.2", "@types/tinycolor2": "1.4.6", - "@typescript-eslint/eslint-plugin": "8.61.1", - "@typescript-eslint/parser": "8.61.1", - "@vitest/coverage-v8": "4.1.9", - "@vue/compiler-core": "3.5.38", + "@typescript-eslint/eslint-plugin": "8.63.0", + "@typescript-eslint/parser": "8.63.0", + "@vitest/browser-playwright": "4.1.10", + "@vitest/coverage-v8": "4.1.10", + "@vue/compiler-core": "3.5.39", "astring": "1.9.0", + "chromatic": "18.0.1", "eslint-plugin-import": "2.32.0", "eslint-plugin-vue": "10.9.2", + "execa": "9.6.1", "happy-dom": "20.10.6", "intersection-observer": "0.12.2", "lightningcss": "1.32.0", "micromatch": "4.0.8", "minimatch": "10.2.5", - "msw": "2.14.6", + "msw": "2.15.0", "msw-storybook-addon": "2.0.7", "nodemon": "3.1.14", "oxc-walker": "1.0.0", - "prettier": "3.8.4", + "playwright": "1.61.1", + "prettier": "3.9.5", "react": "19.2.7", "react-dom": "19.2.7", - "rolldown": "1.1.2", + "rolldown": "1.1.5", "rollup-plugin-visualizer": "7.0.1", "sass-embedded": "1.100.0", - "seedrandom": "3.0.5", - "storybook": "10.4.6", + "storybook": "10.5.0", "storybook-addon-misskey-theme": "github:misskey-dev/storybook-addon-misskey-theme", - "tsx": "4.22.4", - "vite": "8.1.0", + "tsx": "4.23.0", + "typescript": "6.0.2", + "vite": "8.1.4", "vite-plugin-glsl": "1.6.0", "vite-plugin-turbosnap": "1.0.3", - "vitest": "4.1.9", + "vitest": "4.1.10", "vitest-fetch-mock": "0.4.5", - "vue-component-type-helpers": "3.3.5", + "vue-component-type-helpers": "3.3.7", "vue-eslint-parser": "10.4.1", - "vue-tsc": "3.3.5" + "vue-tsc": "3.3.7" } } diff --git a/packages/frontend/playwright.config.ts b/packages/frontend/playwright.config.ts new file mode 100644 index 0000000000..2511a929c3 --- /dev/null +++ b/packages/frontend/playwright.config.ts @@ -0,0 +1,32 @@ +/* + * SPDX-FileCopyrightText: syuilo and misskey-project + * SPDX-License-Identifier: AGPL-3.0-only + */ + +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + testDir: './test/e2e', + reporter: 'list', + fullyParallel: false, + workers: 1, + timeout: 60_000, + expect: { + timeout: 10_000, + }, + outputDir: './test/e2e/artifacts', + use: { + locale: 'en-US', + baseURL: 'http://localhost:61812', + headless: true, + screenshot: 'only-on-failure', + trace: 'retain-on-failure', + video: 'off', + }, + projects: [{ + name: 'chromium', + use: { + browserName: 'chromium', + }, + }], +}); diff --git a/packages/frontend/src/boot/common.ts b/packages/frontend/src/boot/common.ts index fa60ec4b58..4c5e601dae 100644 --- a/packages/frontend/src/boot/common.ts +++ b/packages/frontend/src/boot/common.ts @@ -5,7 +5,7 @@ import { watch, version as vueVersion } from 'vue'; import { compareVersions } from 'compare-versions'; -import { version, lang, apiUrl, isSafeMode } from '@@/js/config.js'; +import { version, lang, isSafeMode } from '@@/js/config.js'; import defaultLightTheme from '@@/themes/l-light.json5'; import defaultDarkTheme from '@@/themes/d-green-lime.json5'; import { storeBootloaderErrors } from '@@/js/store-boot-errors'; @@ -30,6 +30,7 @@ import { fetchCustomEmojis } from '@/custom-emojis.js'; import { prefer } from '@/preferences.js'; import { $i } from '@/i.js'; import { launchPlugins } from '@/plugin.js'; +import { initTelemetry } from '@/telemetry.js'; export async function common(createVue: () => Promise>) { console.info(`Misskey v${version}`); @@ -286,40 +287,7 @@ export async function common(createVue: () => Promise>) { return root; })(); - if (instance.sentryForFrontend) { - const Sentry = await import('@sentry/vue'); - Sentry.init({ - app, - integrations: [ - ...(instance.sentryForFrontend.vueIntegration !== undefined ? [ - Sentry.vueIntegration(instance.sentryForFrontend.vueIntegration ?? undefined), - ] : []), - ...(instance.sentryForFrontend.browserTracingIntegration !== undefined ? [ - Sentry.browserTracingIntegration(instance.sentryForFrontend.browserTracingIntegration ?? undefined), - ] : []), - ...(instance.sentryForFrontend.replayIntegration !== undefined ? [ - Sentry.replayIntegration(instance.sentryForFrontend.replayIntegration ?? undefined), - ] : []), - ], - - // Set tracesSampleRate to 1.0 to capture 100% - tracesSampleRate: 1.0, - - // Set `tracePropagationTargets` to control for which URLs distributed tracing should be enabled - ...(instance.sentryForFrontend.browserTracingIntegration !== undefined ? { - tracePropagationTargets: [apiUrl], - } : {}), - - // Capture Replay for 10% of all sessions, - // plus for 100% of sessions with an error - ...(instance.sentryForFrontend.replayIntegration !== undefined ? { - replaysSessionSampleRate: 0.1, - replaysOnErrorSampleRate: 1.0, - } : {}), - - ...instance.sentryForFrontend.options, - }); - } + await initTelemetry(instance, app); try { await launchPlugins(); diff --git a/packages/frontend/src/components/MkBlurhash.vue b/packages/frontend/src/components/MkBlurhash.vue new file mode 100644 index 0000000000..5000fe111b --- /dev/null +++ b/packages/frontend/src/components/MkBlurhash.vue @@ -0,0 +1,183 @@ + + + + + + + diff --git a/packages/frontend/src/components/MkCaptcha.vue b/packages/frontend/src/components/MkCaptcha.vue index 2fa1135398..1c9de19c25 100644 --- a/packages/frontend/src/components/MkCaptcha.vue +++ b/packages/frontend/src/components/MkCaptcha.vue @@ -21,8 +21,8 @@ SPDX-License-Identifier: AGPL-3.0-only
    Type "ai-chan-kawaii" to pass captcha
    - - + +
    diff --git a/packages/frontend/src/components/MkChart.vue b/packages/frontend/src/components/MkChart.vue index e418e729ca..2d96cb7572 100644 --- a/packages/frontend/src/components/MkChart.vue +++ b/packages/frontend/src/components/MkChart.vue @@ -46,7 +46,7 @@ export type ChartSrc = diff --git a/packages/frontend/src/components/MkImgWithBlurhash.vue b/packages/frontend/src/components/MkImgWithBlurhash.vue index a61836e101..0262e03f3b 100644 --- a/packages/frontend/src/components/MkImgWithBlurhash.vue +++ b/packages/frontend/src/components/MkImgWithBlurhash.vue @@ -14,18 +14,15 @@ SPDX-License-Identifier: AGPL-3.0-only :enterToClass="prefer.s.animation && props.transition?.enterToClass || undefined" :leaveFromClass="prefer.s.animation && props.transition?.leaveFromClass || undefined" > - + :blurhash="hash ?? null" + :height="imgHeight ?? undefined" + :width="imgWidth ?? undefined" + :onlyAvgColor="props.onlyAvgColor" + :show="hide" + /> - - diff --git a/packages/frontend/src/components/MkInput.vue b/packages/frontend/src/components/MkInput.vue index 1622e57d64..3c0565e8a2 100644 --- a/packages/frontend/src/components/MkInput.vue +++ b/packages/frontend/src/components/MkInput.vue @@ -54,7 +54,6 @@ type ModelValueType = diff --git a/packages/frontend/src/components/MkLightbox.item.vue b/packages/frontend/src/components/MkLightbox.item.vue new file mode 100644 index 0000000000..91751f16b5 --- /dev/null +++ b/packages/frontend/src/components/MkLightbox.item.vue @@ -0,0 +1,1147 @@ + + + + + + + + + diff --git a/packages/frontend/src/components/MkLightbox.vue b/packages/frontend/src/components/MkLightbox.vue new file mode 100644 index 0000000000..ae40c7a612 --- /dev/null +++ b/packages/frontend/src/components/MkLightbox.vue @@ -0,0 +1,319 @@ + + + + + + + diff --git a/packages/frontend/src/components/MkMediaAudio.vue b/packages/frontend/src/components/MkMediaAudio.vue index c178b923ae..b565ed8bce 100644 --- a/packages/frontend/src/components/MkMediaAudio.vue +++ b/packages/frontend/src/components/MkMediaAudio.vue @@ -49,8 +49,8 @@ SPDX-License-Identifier: AGPL-3.0-only tabindex="-1" @click.stop="togglePlayPause" > - - + +
    @@ -100,6 +100,7 @@ import { hms } from '@/filters/hms.js'; import MkMediaRange from '@/components/MkMediaRange.vue'; import { $i, iAmModerator } from '@/i.js'; import { prefer } from '@/preferences.js'; +import { getFileMenu } from '@/utility/get-file-menu.js'; import { canRevealFile, shouldHideFileByDefault } from '@/utility/sensitive-file.js'; const props = defineProps<{ @@ -208,56 +209,11 @@ function showMenu(ev: MouseEvent) { { type: 'divider', }, - { - text: i18n.ts.hide, - icon: 'ti ti-eye-off', - action: () => { - hide.value = true; - }, - }, ]; - if (iAmModerator) { - menu.push({ - text: props.audio.isSensitive ? i18n.ts.unmarkAsSensitive : i18n.ts.markAsSensitive, - icon: props.audio.isSensitive ? 'ti ti-eye' : 'ti ti-eye-exclamation', - danger: true, - action: () => toggleSensitive(props.audio), - }); - } - - const details: MenuItem[] = []; - if ($i?.id === props.audio.userId) { - details.push({ - type: 'link', - text: i18n.ts._fileViewer.title, - icon: 'ti ti-info-circle', - to: `/my/drive/file/${props.audio.id}`, - }); - } - - if (iAmModerator) { - details.push({ - type: 'link', - text: i18n.ts.moderation, - icon: 'ti ti-photo-exclamation', - to: `/admin/file/${props.audio.id}`, - }); - } - - if (details.length > 0) { - menu.push({ type: 'divider' }, ...details); - } - - if (prefer.s.devMode) { - menu.push({ type: 'divider' }, { - icon: 'ti ti-hash', - text: i18n.ts.copyFileId, - action: () => { - copyToClipboard(props.audio.id); - }, - }); - } + menu.push(...getFileMenu(props.audio, (newState) => { + hide.value = newState; + })); menuShowing.value = true; os.popupMenu(menu, ev.currentTarget ?? ev.target, { @@ -268,20 +224,6 @@ function showMenu(ev: MouseEvent) { }); } -async function toggleSensitive(file: Misskey.entities.DriveFile) { - const { canceled } = await os.confirm({ - type: 'warning', - text: file.isSensitive ? i18n.ts.unmarkAsSensitiveConfirm : i18n.ts.markAsSensitiveConfirm, - }); - - if (canceled) return; - - os.apiWithDialog('drive/files/update', { - fileId: file.id, - isSensitive: !file.isSensitive, - }); -} - // MediaControl: Common State const oncePlayed = ref(false); const isReady = ref(false); diff --git a/packages/frontend/src/components/MkMediaImage.vue b/packages/frontend/src/components/MkMediaImage.vue index 4236bd943a..85f6939833 100644 --- a/packages/frontend/src/components/MkMediaImage.vue +++ b/packages/frontend/src/components/MkMediaImage.vue @@ -4,7 +4,7 @@ SPDX-License-Identifier: AGPL-3.0-only --> @@ -68,16 +70,14 @@ SPDX-License-Identifier: AGPL-3.0-only @@ -234,23 +174,6 @@ function onContextmenu(ev: PointerEvent) { cursor: pointer; } -.hide { - display: block; - position: absolute; - background-color: rgba(0, 0, 0, 0.3); - -webkit-backdrop-filter: var(--MI-blur, blur(15px)); - backdrop-filter: var(--MI-blur, blur(15px)); - border-radius: 0 0 0 9px; - color: #fff; - font-size: 12px; - opacity: .5; - padding: 5px 8px; - text-align: center; - cursor: pointer; - top: 0; - right: 0; -} - .hiddenTextWrapper { display: table-cell; text-align: center; @@ -281,16 +204,25 @@ html[data-color-scheme=light] .visible { background-color: rgba(0, 0, 0, 0.3); -webkit-backdrop-filter: var(--MI-blur, blur(15px)); backdrop-filter: var(--MI-blur, blur(15px)); - border-radius: 9px 0 0 0; color: #fff; font-size: 0.8em; width: 28px; height: 28px; text-align: center; +} + +.menuBottom { + border-radius: 8px 0 8px 0; bottom: 0; right: 0; } +.menuTop { + border-radius: 0 8px 0 8px; + top: 0; + right: 0; +} + .imageContainer { display: block; overflow: hidden; diff --git a/packages/frontend/src/components/MkMediaList.vue b/packages/frontend/src/components/MkMediaList.vue index 9090e74bb6..bab49e5375 100644 --- a/packages/frontend/src/components/MkMediaList.vue +++ b/packages/frontend/src/components/MkMediaList.vue @@ -20,8 +20,23 @@ SPDX-License-Identifier: AGPL-3.0-only ]" >
    @@ -29,18 +44,16 @@ SPDX-License-Identifier: AGPL-3.0-only diff --git a/packages/frontend/src/components/MkMention.vue b/packages/frontend/src/components/MkMention.vue index f2cf33eb65..016257ba9f 100644 --- a/packages/frontend/src/components/MkMention.vue +++ b/packages/frontend/src/components/MkMention.vue @@ -33,7 +33,7 @@ const canonical = props.host === localHost ? `@${props.username}` : `@${props.us const url = `/${canonical}`; const isMe = $i && ( - `@${props.username}@${toUnicode(props.host)}` === `@${$i.username}@${toUnicode(localHost)}`.toLowerCase() + `@${props.username}@${toUnicode(props.host)}`.toLowerCase() === `@${$i.username}@${toUnicode(localHost)}`.toLowerCase() ); const avatarUrl = computed(() => prefer.s.disableShowingAnimatedImages || prefer.s.dataSaver.avatar diff --git a/packages/frontend/src/components/MkModal.vue b/packages/frontend/src/components/MkModal.vue index 92174d8ef7..d397531680 100644 --- a/packages/frontend/src/components/MkModal.vue +++ b/packages/frontend/src/components/MkModal.vue @@ -33,7 +33,7 @@ SPDX-License-Identifier: AGPL-3.0-only :duration="transitionDuration" appear @afterLeave="onClosed" @enter="emit('opening')" @afterEnter="onOpened" >
    -
    +
    diff --git a/packages/frontend/src/components/MkModalWindow.vue b/packages/frontend/src/components/MkModalWindow.vue index b8d9da0a13..bb1e63862e 100644 --- a/packages/frontend/src/components/MkModalWindow.vue +++ b/packages/frontend/src/components/MkModalWindow.vue @@ -7,7 +7,7 @@ SPDX-License-Identifier: AGPL-3.0-only
    - + diff --git a/packages/frontend/src/components/MkNote.vue b/packages/frontend/src/components/MkNote.vue index 1cb562fb62..4017924fa4 100644 --- a/packages/frontend/src/components/MkNote.vue +++ b/packages/frontend/src/components/MkNote.vue @@ -144,7 +144,7 @@ SPDX-License-Identifier: AGPL-3.0-only -