Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .codex/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# SQLBot 开发用 MCP

项目级 MCP 布局,提供 TAPD、Playwright 和 DBHub 启动器。
这里服务于 Codex 开发任务,与 SQLBot 产品对外提供的 MCP 接口无关。

## 配置与启用

[Codex 官方 MCP 文档](https://developers.openai.com/codex/mcp)支持受信任项目使用 `.codex/config.toml`。
本仓库三个服务配置为 `enabled = true`。首次使用前需完成本地配置;暂不使用的服务应设为 `enabled = false`,避免缺少配置时启动失败。

1. 在本机安装需要的 MCP 服务,记录其入口文件位置;启动器不会自动安装依赖。
2. 将相应目录的 `.env.example` 复制为同目录 `.env.local`,填写自己的入口、解释器和项目配置。
3. 在 `.codex/config.toml` 中确认需要的服务为 `enabled = true`,其余设为 `false`;从 SQLBot 仓库根启动新的 Codex 会话,并检查 MCP 连接状态。

配置中的启动脚本路径相对于仓库根。不要从子目录启动这些相对路径命令。
`.env.local` 使用 `KEY="value"` 格式,值必须是 JSON 字符串;不支持 shell 展开、单引号或行尾注释。
未知字段、重复字段和缺失必填值会导致启动失败,错误消息不回显配置值。
本地配置、浏览器资料及输出已被 Git 忽略;建议将 `.env.local` 权限设为 `600`。
不要复制其他项目的凭据、项目 ID 或浏览器会话。

## 服务约定

- **TAPD**:填写 SQLBot 对应的 `TAPD_WORKSPACE_ID`、个人用户名、令牌和入口。配置保留参考项目的 Node/Python 两种实现的读取工具白名单,不开放写入工具。Python `mcp-server-tapd` 调用仍须显式传本项目 `workspace_id`;启动器不会自动注入每次调用。不要使用会另行加载其他个人凭据的包装入口。
- **Playwright**:使用支持 Chrome、`--user-data-dir` 和 `--output-dir` 参数的 `@playwright/mcp` 入口,并准备本机 Chrome。浏览器资料位于 `playwright_mcp/browser-profile/`,截图等产物位于 `playwright_mcp/output/`。`TEST_BASE_URL` 是本机示例,需按实际测试环境调整;`TEST_*` 仅供执行测试时按需使用,启动器不会自动登录。
- **DBHub**:沿用参考项目针对 `@bytebase/dbhub@1.3.1` 的配置格式,入口须支持 `--config`。示例 DSN 已改为 PostgreSQL/sqlbot,仅用于展示格式。DSN 通过环境变量传递,`dbhub.toml` 为 `execute_sql` 设置 `readonly = true` 和最多 1000 行;数据库账号本身也必须只读。只在任务授权的测试环境使用,不用生产或共享业务库做配置验证。

## 运行时

外层启动器需要 PATH 中的 Python 3.9+。`mcp_runtime.py` 统一处理配置和入口解析:

- `MCP_RUNTIME="auto"` 根据 `.js/.cjs/.mjs`、`.py` 或 Node/Python shebang 识别运行时,也可明确指定 `node` 或 `python`。
- MCP 入口须为已存在的绝对路径;`NODE_BIN` / `PYTHON_BIN` 可为绝对路径或 PATH 命令名。桌面应用 PATH 可能与终端不同,推荐在本地配置中填写解释器绝对路径。
- Python MCP 使用其安装环境的解释器。Python TAPD 将 `TAPD_API_TOKEN` 映射为 `TAPD_ACCESS_TOKEN`,将用户名映射为 `CURRENT_USER_NICK`,并使用 stdio。
- 运行时识别不保证不同 MCP 实现的命令行参数和工具 API 兼容。Playwright、DBHub 模板面向上述 Node 实现。

## 验证

从仓库根运行:

```bash
python3 -B backend/tests/test_codex_mcp_runtime.py
```

离线测试覆盖配置校验、解释器选择和启动参数/环境变量传递,不连接外部服务。
配置解析或离线测试通过不代表真实 MCP 握手、工具调用或数据库只读权限已经验证。
22 changes: 22 additions & 0 deletions .codex/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# 填写对应 .env.local 后,按需将 enabled 改为 true。
[mcp_servers.tapd]
command = "python3"
args = [".codex/tapd_mcp/launch.py"]
enabled = true
startup_timeout_sec = 20
tool_timeout_sec = 60
enabled_tools = ["tapd_get_current_user", "tapd_parse_url", "tapd_get_story", "tapd_list_stories", "tapd_get_task", "tapd_list_tasks", "tapd_get_bug", "tapd_list_bugs", "tapd_get_iteration", "tapd_list_iterations", "tapd_list_comments", "tapd_get_story_related_bugs", "tapd_get_story_changes", "tapd_list_workspace_users", "get_stories_or_tasks", "get_story_or_task_count", "get_bug", "get_bug_count", "get_comments", "get_iterations", "get_related_bugs", "get_workspace_users", "get_entity_custom_fields", "get_entity_attachments", "get_image", "get_workflows_status_map", "get_workitem_types"]

[mcp_servers.playwright]
command = "python3"
args = [".codex/playwright_mcp/launch.py"]
enabled = true
startup_timeout_sec = 30
tool_timeout_sec = 120

[mcp_servers.dbhub]
command = "python3"
args = [".codex/dbhub_mcp/launch.py"]
enabled = true
startup_timeout_sec = 30
tool_timeout_sec = 120
8 changes: 8 additions & 0 deletions .codex/dbhub_mcp/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
DBHUB_DSN="postgres://readonly_user:password@127.0.0.1:5432/sqlbot"
DBHUB_MCP_ENTRY="/absolute/path/to/@bytebase/dbhub/dist/index.js"
NODE_BIN="node"

# auto 根据扩展名或 shebang 判断;也可指定 node / python
MCP_RUNTIME="auto"
# Python 入口请使用安装该 MCP 的虚拟环境解释器
PYTHON_BIN="python3"
13 changes: 13 additions & 0 deletions .codex/dbhub_mcp/dbhub.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[[sources]]
id = "sqlbot"
dsn = "${DSN}"

[[tools]]
name = "execute_sql"
source = "sqlbot"
readonly = true
max_rows = 1000

[[tools]]
name = "search_objects"
source = "sqlbot"
19 changes: 19 additions & 0 deletions .codex/dbhub_mcp/launch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
"""Launch DBHub using the existing read-only configuration."""
import os
import sys
from pathlib import Path

sys.dont_write_bytecode = True
base = Path(__file__).resolve().parent
sys.path.insert(0, str(base.parent))
from mcp_runtime import execute, load_config, resolve_command

try:
local = load_config(base, {'DBHUB_DSN', 'DBHUB_MCP_ENTRY'}, {'DBHUB_DSN', 'DBHUB_MCP_ENTRY'})
_, command = resolve_command(local, 'DBHUB_MCP_ENTRY')
env = os.environ.copy()
env['DSN'] = local['DBHUB_DSN']
command.extend(['--config', str(base / 'dbhub.toml')])
execute(command, env)
except (OSError, ValueError, KeyError, TypeError):
sys.exit('DBHub MCP 启动失败;请检查 .codex/dbhub_mcp/.env.local 的字段、入口及解释器')
62 changes: 62 additions & 0 deletions .codex/mcp_runtime.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
"""Shared local MCP configuration and runtime resolution (Python 3.9+)."""
import json
import os
import shutil
from pathlib import Path

RUNTIME_KEYS = {'MCP_RUNTIME', 'NODE_BIN', 'PYTHON_BIN'}


def load_config(base, allowed, required):
local = {}
for number, raw in enumerate((base / '.env.local').read_text().splitlines(), 1):
line = raw.strip()
if not line or line.startswith('#'):
continue
key, sep, value = line.partition('=')
key = key.strip()
if not sep or key not in allowed | RUNTIME_KEYS or key in local:
raise ValueError(f'配置第 {number} 行存在未知或重复字段')
try:
parsed = json.loads(value.strip())
except ValueError:
raise ValueError(f'配置第 {number} 行必须使用 JSON 字符串') from None
if not isinstance(parsed, str):
raise ValueError(f'配置第 {number} 行必须使用字符串')
local[key] = parsed
for key in required:
if not local.get(key):
raise ValueError(f'缺少配置字段 {key}')
return local


def resolve_command(local, entry_key):
entry = Path(local[entry_key]).expanduser()
if not entry.is_absolute() or not entry.is_file():
raise ValueError(f'{entry_key} 必须是存在的文件绝对路径')
runtime = local.get('MCP_RUNTIME') or 'auto'
if runtime == 'auto':
if entry.suffix in {'.js', '.cjs', '.mjs'}:
runtime = 'node'
elif entry.suffix == '.py':
runtime = 'python'
else:
with entry.open('rb') as stream:
shebang = stream.readline(512)
if shebang.startswith(b'#!') and b'python' in shebang:
runtime = 'python'
elif shebang.startswith(b'#!') and b'node' in shebang:
runtime = 'node'
else:
raise ValueError('无法判断入口类型,请明确设置 MCP_RUNTIME')
if runtime not in {'node', 'python'}:
raise ValueError('MCP_RUNTIME 仅支持 auto、node、python')
key, fallback = ('NODE_BIN', 'node') if runtime == 'node' else ('PYTHON_BIN', 'python3')
binary = shutil.which(str(Path(local.get(key) or fallback).expanduser()))
if not binary:
raise ValueError(f'{key} 不是可执行文件,请指定实际解释器路径')
return runtime, [binary, str(entry)]


def execute(command, env):
os.execvpe(command[0], command, env)
10 changes: 10 additions & 0 deletions .codex/playwright_mcp/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
PLAYWRIGHT_MCP_ENTRY="/absolute/path/to/@playwright/mcp/cli.js"
NODE_BIN="node"
TEST_BASE_URL="http://127.0.0.1:8000"
TEST_USERNAME=""
TEST_PASSWORD=""

# auto 根据扩展名或 shebang 判断;也可指定 node / python
MCP_RUNTIME="auto"
# Python 入口请使用安装该 MCP 的虚拟环境解释器
PYTHON_BIN="python3"
22 changes: 22 additions & 0 deletions .codex/playwright_mcp/launch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
"""Launch Playwright MCP with project-local browser storage."""
import os
import sys
from pathlib import Path

sys.dont_write_bytecode = True
base = Path(__file__).resolve().parent
sys.path.insert(0, str(base.parent))
from mcp_runtime import execute, load_config, resolve_command

try:
local = load_config(base, {'PLAYWRIGHT_MCP_ENTRY', 'TEST_BASE_URL', 'TEST_USERNAME', 'TEST_PASSWORD'},
{'PLAYWRIGHT_MCP_ENTRY'})
_, command = resolve_command(local, 'PLAYWRIGHT_MCP_ENTRY')
profile = base / 'browser-profile'
output = base / 'output'
profile.mkdir(exist_ok=True, mode=0o700)
output.mkdir(exist_ok=True, mode=0o700)
command.extend(['--browser', 'chrome', '--user-data-dir', str(profile), '--output-dir', str(output)])
execute(command, os.environ.copy())
except (OSError, ValueError, KeyError, TypeError):
sys.exit('Playwright MCP 启动失败;请检查 .codex/playwright_mcp/.env.local 的字段、入口及解释器')
15 changes: 15 additions & 0 deletions .codex/tapd_mcp/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
TAPD_WORKSPACE_ID=""
TAPD_CURRENT_USER=""
TAPD_API_TOKEN=""
TAPD_MCP_ENTRY="/absolute/path/to/tapd-mcp/dist/index.js"
NODE_BIN="node"

# auto 根据扩展名或 shebang 判断;也可指定 node / python
MCP_RUNTIME="auto"
# Python 入口请使用安装该 MCP 的虚拟环境解释器
PYTHON_BIN="python3"

# Python mcp-server-tapd:入口填写虚拟环境 bin/mcp-server-tapd
# PYTHON_BIN 填写同一虚拟环境 bin/python,令牌仍填写 TAPD_API_TOKEN
TAPD_API_BASE_URL="https://api.tapd.cn"
TAPD_BASE_URL="https://www.tapd.cn"
34 changes: 34 additions & 0 deletions .codex/tapd_mcp/launch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
"""Launch Node or Python TAPD MCP using only project-local credentials."""
import os
import sys
from pathlib import Path

sys.dont_write_bytecode = True
base = Path(__file__).resolve().parent
sys.path.insert(0, str(base.parent))
from mcp_runtime import execute, load_config, resolve_command

try:
keys = {'TAPD_WORKSPACE_ID', 'TAPD_CURRENT_USER', 'TAPD_API_TOKEN',
'TAPD_MCP_ENTRY', 'TAPD_API_BASE_URL', 'TAPD_BASE_URL'}
local = load_config(base, keys, {'TAPD_WORKSPACE_ID', 'TAPD_CURRENT_USER',
'TAPD_API_TOKEN', 'TAPD_MCP_ENTRY'})
runtime, command = resolve_command(local, 'TAPD_MCP_ENTRY')
env = os.environ.copy()
for key in ('TAPD_API_USER', 'TAPD_API_PASSWORD', 'TAPD_ACCESS_TOKEN',
'TAPD_API_BASE_URL', 'TAPD_BASE_URL', 'CURRENT_USER_NICK'):
env.pop(key, None)
env.update({key: local[key] for key in ('TAPD_WORKSPACE_ID', 'TAPD_CURRENT_USER', 'TAPD_API_TOKEN')})
if runtime == 'python':
env['TAPD_ACCESS_TOKEN'] = local['TAPD_API_TOKEN']
env['CURRENT_USER_NICK'] = local['TAPD_CURRENT_USER']
env['TAPD_API_BASE_URL'] = local.get('TAPD_API_BASE_URL') or 'https://api.tapd.cn'
env['TAPD_BASE_URL'] = local.get('TAPD_BASE_URL') or 'https://www.tapd.cn'
command.extend(['--mode', 'stdio'])
else:
for key in ('TAPD_API_BASE_URL', 'TAPD_BASE_URL'):
if local.get(key):
env[key] = local[key]
execute(command, env)
except (OSError, ValueError, KeyError, TypeError):
sys.exit('TAPD MCP 启动失败;请检查 .codex/tapd_mcp/.env.local 的字段、入口及解释器')
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -199,3 +199,8 @@ AGENTS.local.env
frontend/.npmrc
backend/logs/
/logs/

# Codex MCP 本地凭据、浏览器会话和输出
/.codex/**/.env.local
/.codex/playwright_mcp/browser-profile/
/.codex/playwright_mcp/output/
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ SQLBot 让业务用户用自然语言提问,基于已配置的数据源生成
| `frontend/` | Vue 3、TypeScript strict、Vite、Pinia、Vue Router 和 Element Plus。请求在 `src/api/`,共享实体在 `src/entity/`,状态在 `src/stores/`,路由在 `src/router/`,UI 在 `src/views/` 和 `src/components/`。 |
| `g2-ssr/` | Node.js 图表渲染服务,使用 `@antv/g2-ssr`;图表实现位于 `charts/`,目录内有独立 `AGENTS.md`。 |
| `installer/` | 离线安装、卸载、配置模板和服务控制脚本。 |
| `.codex/` | Codex 开发用 MCP 配置、启动器与本地配置模板;使用方式见 `.codex/README.md`,不属于 SQLBot 产品 MCP。 |
| `Dockerfile*`、`docker-compose.yaml`、`start.sh` | 容器组装和运行时进程启动。 |

商业扩展源码维护在独立仓库 [dataease/sqlbot-xpack](https://github.com/dataease/sqlbot-xpack)。它不是 submodule,也不是本仓库的固定子目录。本地 `sqlbot-xpack/` checkout 会被主仓库忽略,永远不应出现在本仓库 PR 中。
Expand All @@ -39,6 +40,7 @@ SQLBot 让业务用户用自然语言提问,基于已配置的数据源生成
| 修改后端业务代码 | `docs/agents/backend.md` |
| 修改前端代码 | `docs/agents/frontend.md` |
| 修改或新增测试、执行验证 | `docs/agents/testing.md` |
| 修改 Codex 开发用 MCP 配置或启动器 | `.codex/README.md` |
| 修改用户可见文案或新增语言 | `docs/agents/i18n.md` |
| 修改认证、授权、SQL 执行、上传/下载、嵌入协议或前端渲染安全 | `docs/agents/security.md` |
| 修改 SQLModel 模型或 Alembic 迁移 | `docs/agents/migrations.md` |
Expand Down
111 changes: 111 additions & 0 deletions backend/tests/test_codex_mcp_runtime.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
"""Offline launcher regression tests; no real credentials or services."""
import os
from pathlib import Path
import runpy
import sys
import tempfile
import unittest
from unittest.mock import patch

sys.dont_write_bytecode = True
ROOT = Path(__file__).resolve().parents[2] / '.codex'
sys.path.insert(0, str(ROOT))
from mcp_runtime import load_config, resolve_command


class RuntimeTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.base = Path(self.temp.name)

def entry(self, name, content=''):
p = self.base / name
p.write_text(content)
return str(p)

def test_runtime_detection(self):
for name, content, expected in [('index.js', '', 'node'), ('index.mjs', '', 'node'),
('index.cjs', '', 'node'), ('main.py', '', 'python'),
('mcp-server', '#!/venv/bin/python\n', 'python')]:
with self.subTest(name=name):
cfg = {'ENTRY': self.entry(name, content), 'NODE_BIN': sys.executable, 'PYTHON_BIN': sys.executable}
runtime, command = resolve_command(cfg, 'ENTRY')
self.assertEqual(runtime, expected)
self.assertEqual(command[0], sys.executable)

def test_explicit_runtime_and_missing_binary(self):
cfg = {'ENTRY': self.entry('custom'), 'MCP_RUNTIME': 'python', 'PYTHON_BIN': sys.executable}
self.assertEqual(resolve_command(cfg, 'ENTRY')[0], 'python')
cfg['PYTHON_BIN'] = '/does-not-exist/python'
with self.assertRaises(ValueError):
resolve_command(cfg, 'ENTRY')

def test_bad_configs_do_not_echo_secrets(self):
for value in ['TOKEN="secret"\nTOKEN="secret"', 'TOKEN=secret', 'TOKEN=123', 'UNKNOWN="secret"']:
(self.base / '.env.local').write_text(value)
with self.assertRaises(ValueError) as error:
load_config(self.base, {'TOKEN'}, {'TOKEN'})
self.assertNotIn('secret', str(error.exception))

def test_launcher_contracts(self):
for service, runtime in [('tapd', 'node'), ('tapd', 'python'), ('playwright', 'node'), ('dbhub', 'node')]:
with self.subTest(service=service, runtime=runtime):
config = {'TAPD_WORKSPACE_ID': '123', 'TAPD_CURRENT_USER': 'tester',
'TAPD_API_TOKEN': 'test-token', 'DBHUB_DSN': 'test-dsn'}
command = [sys.executable, '/fake/entry']
with patch('mcp_runtime.load_config', return_value=config), \
patch('mcp_runtime.resolve_command', return_value=(runtime, command)), \
patch('mcp_runtime.execute') as execute, \
patch.object(Path, 'mkdir'), \
patch.dict(os.environ, {'TAPD_API_USER': 'old', 'TAPD_API_PASSWORD': 'old', 'TAPD_ACCESS_TOKEN': 'old'}):
runpy.run_path(str(ROOT / (service + '_mcp') / 'launch.py'), run_name='__main__')
args, env = execute.call_args.args
self.assertNotIn('test-token', args)
self.assertNotIn('test-dsn', args)
if service == 'tapd':
self.assertNotIn('TAPD_API_USER', env)
self.assertNotIn('TAPD_API_PASSWORD', env)
self.assertEqual(env['TAPD_API_TOKEN'], 'test-token')
if runtime == 'python':
self.assertEqual(env['TAPD_ACCESS_TOKEN'], 'test-token')
self.assertEqual(env['CURRENT_USER_NICK'], 'tester')
self.assertEqual(args[-2:], ['--mode', 'stdio'])
else:
self.assertNotIn('TAPD_ACCESS_TOKEN', env)
elif service == 'dbhub':
self.assertEqual(env['DSN'], 'test-dsn')
self.assertIn('--config', args)
else:
self.assertIn('--user-data-dir', args)
self.assertIn('--output-dir', args)

def test_missing_required_configuration(self):
for content in ['', 'TOKEN=""']:
(self.base / '.env.local').write_text(content)
with self.assertRaises(ValueError):
load_config(self.base, {'TOKEN'}, {'TOKEN'})

def test_invalid_runtime_and_entry(self):
for config in [
{'ENTRY': 'relative.py'},
{'ENTRY': str(self.base / 'missing.py')},
{'ENTRY': self.entry('unknown')},
{'ENTRY': self.entry('valid.py'), 'MCP_RUNTIME': 'shell'},
]:
with self.subTest(config=config), self.assertRaises(ValueError):
resolve_command(config, 'ENTRY')

def test_launcher_errors_hide_secrets_and_do_not_execute(self):
for service in ['tapd', 'playwright', 'dbhub']:
with self.subTest(service=service):
with patch('mcp_runtime.load_config', side_effect=ValueError('secret-token')), \
patch('mcp_runtime.execute') as execute, \
self.assertRaises(SystemExit) as error:
runpy.run_path(str(ROOT / (service + '_mcp') / 'launch.py'), run_name='__main__')
self.assertNotIn('secret-token', str(error.exception))
execute.assert_not_called()


if __name__ == '__main__':
unittest.main()
Loading
Loading