{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6cf84fba",
   "metadata": {},
   "source": [
    "# 第 9 章配套 · Wooldridge 教材数据 + Python 复现实战\n",
    "\n",
    "> **定位**：《AI 辅助金融研究学生自学手册》第 9 章「推断统计与实证策略」的 Python fallback 配套\n",
    "> **数据源**：`wooldridge` 包（111 个教材数据集，`pip install wooldridge` 一装全有）\n",
    "> **工具链**：statsmodels（OLS/时间序列）+ linearmodels（面板/IV）\n",
    "> **所有代码已在 macOS + Python 3.12 实测通过**\n",
    "\n",
    "**为什么用这套数据**：不用找数据、不用清洗，`woo.data()` 一行取数，横截面/面板/时间序列/IV 四类\n",
    "实证设计全部配齐，且每个回归都能在教材正文找到对应例题和标准答案——**复现错了立刻能发现**。\n",
    "\n",
    "**两个取数坑（先看再跑）**：\n",
    "1. API 只有 `woo.data(名字)`，网上教程写的 `woo.data_sets()` 不存在\n",
    "2. 数据集名**全小写、无下划线**：是 `wagepan` 不是 `wage_pan`；首次调用联网下载，之后离线可用"
   ]
  },
  {
   "cell_type": "code",
   "id": "8e1b051c",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# 环境准备(装过就跳过):pip install wooldridge linearmodels\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "import wooldridge as woo\n",
    "print(woo.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "228aa382",
   "metadata": {},
   "source": [
    "## 1. 四类实证设计对应的教材数据集\n",
    "\n",
    "| 数据集 | 规模 | 教材章节 | 实证设计 |\n",
    "|---|---|---|---|\n",
    "| `wage1` | 526×24 | 第 1-7 章 | 横截面 OLS（对数工资方程） |\n",
    "| `wagepan` | 4360×44 | 第 13-14 章 | 面板数据（FE/RE） |\n",
    "| `card` | 3010×34 | 第 15 章 | IV/2SLS（教育回报） |\n",
    "| `jtrain` | 471×30 | 第 13-14 章 | 面板/DID 练习（培训效应） |\n",
    "| `intqrt` | 124×23 | 第 10-12 章 | 时间序列（利率期限结构） |\n",
    "| `hprice1` | 88×10 | 第 5-8 章 | 横截面（hedonic 房价） |\n",
    "\n",
    "数据集官方说明在包内：`description/` 目录（每个数据集一个 txt，含变量标签与出处）。"
   ]
  },
  {
   "cell_type": "code",
   "id": "2877df57",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "df = woo.data('wage1')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46f18792",
   "metadata": {},
   "source": [
    "## 2. 横截面 OLS：对数工资方程（教材例 2.4 / 3.1）\n",
    "\n",
    "工资经济学最经典的起点：教育年限每多一年，工资大约上涨百分之几？\n",
    "教材标准答案：`educ` 系数 ≈ **0.092**（控制 exper、tenure 后）。"
   ]
  },
  {
   "cell_type": "code",
   "id": "cb6470c1",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "df['lwage'] = np.log(df['wage'])\n",
    "X = sm.add_constant(df[['educ', 'exper', 'tenure']])\n",
    "m_ols = sm.OLS(df['lwage'], X).fit()                    # 常规标准误\n",
    "m_rob = sm.OLS(df['lwage'], X).fit(cov_type='HC1')       # 异方差稳健(≈Stata robust)\n",
    "print(f\"educ = {m_ols.params['educ']:.4f}  常规SE = {m_ols.bse['educ']:.4f}  \"\n",
    "      f\"稳健SE = {m_rob.bse['educ']:.4f}  n = {int(m_ols.nobs)}\")\n",
    "m_rob.summary().tables[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d3ec380",
   "metadata": {},
   "source": [
    "**结果解读**：educ ≈ 0.092——多读一年书，小时工资约高 9.2%，与教材一致。\n",
    "稳健 SE 略大于常规 SE（异方差存在的证据），这就是 Stata 里 `, robust` 的 Python 写法。\n",
    "\n",
    "## 3. 面板固定效应：工资面板（教材例 14.1，wagepan）\n",
    "\n",
    "**先讲一个必踩的坑**：`educ`（教育年限）在个体内**不随时间变化**，会被个体固定效应完全吸收——\n",
    "Stata 的 `xtreg, fe` 自动 drop 并提示 omitted，linearmodels 直接报 `AbsorbingEffectError`。\n",
    "这是 FE 模型的核心教学点：**固定效应只能估计时变变量的系数**。所以例 14.1 只用时变变量。"
   ]
  },
  {
   "cell_type": "code",
   "id": "cdf7bfed",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from linearmodels.panel import PanelOLS\n",
    "wp = woo.data('wagepan')\n",
    "wp['exper2'] = wp['exper'] ** 2          # wagepan 不带平方项,自己构造\n",
    "wp = wp.set_index(['nr', 'year'])        # 双重索引:个体 nr + 时间 year\n",
    "fe = PanelOLS(wp['lwage'],\n",
    "              sm.add_constant(wp[['exper', 'exper2', 'married', 'union']]),\n",
    "              entity_effects=True).fit()  # ≈ xtreg lwage X, i(nr) fe\n",
    "print(f\"exper = {fe.params['exper']:.4f} (t = {fe.tstats['exper']:.2f})  \"\n",
    "      f\"R2(within) = {fe.rsquared_within:.3f}  n = {fe.nobs}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9dd03fb",
   "metadata": {},
   "source": [
    "**结果解读**：工龄回报 ≈ 0.117/年（教材例 14.1），within-R² ≈ 0.18。\n",
    "对比：把 `entity_effects=True` 去掉就是混合 OLS；换成 `RandomEffects` 就是 RE，\n",
    "再用 `compare_fe_re` 做 Hausman 检验——第 9 章「面板数据策略」小节的完整练习线。\n",
    "\n",
    "## 4. IV/2SLS：教育回报的内生性（教材第 15 章，card）\n",
    "\n",
    "能力与教育正相关、能力不可观测 → OLS 高估教育回报。Card (1995) 的经典工具变量：\n",
    "**1966 年住址附近是否有 4 年制大学（`nearc4`）**——影响教育成本（相关性），\n",
    "不直接影响个人工资（排他性，可争）。\n",
    "\n",
    "**工具变量两大检查**（第 9 章对应小节）：\n",
    "1. 相关性：第一阶段 F 统计量 > 10（弱工具变量检验）\n",
    "2. 排他性：不可检验，只能理论论证——这正是「用 AI 前先看边界」的典型场景：\n",
    "   AI 可以帮你跑 2SLS，但**排他性论证必须你自己写**"
   ]
  },
  {
   "cell_type": "code",
   "id": "c343590f",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from linearmodels.iv import IV2SLS\n",
    "cd = woo.data('card')\n",
    "cd['lwage'] = np.log(cd['wage'])\n",
    "cd['exper2'] = cd['exper'] ** 2\n",
    "# ① OLS 对照\n",
    "ols_c = sm.OLS(cd['lwage'],\n",
    "               sm.add_constant(cd[['educ', 'exper', 'exper2', 'black', 'smsa']])).fit()\n",
    "# ② 2SLS:nearc4 作 educ 的工具 ≈ ivregress 2sls lwage (educ = nearc4) X, robust\n",
    "iv = IV2SLS(dependent=cd['lwage'],\n",
    "            exog=sm.add_constant(cd[['exper', 'exper2', 'black', 'smsa']]),\n",
    "            endog=cd[['educ']],\n",
    "            instruments=cd[['nearc4']]).fit(cov_type='robust')\n",
    "print(f\"OLS: educ = {ols_c.params['educ']:.4f}\")\n",
    "print(f\"IV : educ = {iv.params['educ']:.4f} (t = {iv.tstats['educ']:.2f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24649f1e",
   "metadata": {},
   "source": [
    "**结果解读**：OLS ≈ 0.076，IV ≈ 0.177——IV 明显大于 OLS，与教材例题方向一致。\n",
    "两种解释（论文里都要讨论）：能力偏误使 OLS 低估？还是工具变量排他性被违反\n",
    "（nearc4 通过家庭背景影响工资）使 IV 高估？——**这就是 IV 论文的讨论章为什么必须写**。\n",
    "\n",
    "## 5. 时间序列：利率期限结构（intqrt）\n",
    "\n",
    "124 个季度的美国国债数据，23 个变量把水平值/差分/滞后全预制好。两个经典检验：\n",
    "\n",
    "| 检验 | 回归 | 原假设（市场有效/预期理论） | 实测结果 |\n",
    "|---|---|---|---|\n",
    "| 预期理论 | `hy6 ~ spr63` | 利差对未来 3 个月持有收益无预测力 | b=1.13 (t=3.97)，可预测 |\n",
    "| 有效性 | `chy6 ~ chy6_1` | 收益变化无自相关 | b=-0.42 (t=-5.01)，负自相关 |"
   ]
  },
  {
   "cell_type": "code",
   "id": "795bd486",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "ts = woo.data('intqrt')\n",
    "# ① 预期理论:利差 spr63 预测持有收益 hy6\n",
    "d1 = ts.dropna(subset=['hy6', 'spr63'])\n",
    "m1 = sm.OLS(d1['hy6'], sm.add_constant(d1[['spr63']])).fit()\n",
    "print(f\"hy6 ~ spr63 : b = {m1.params['spr63']:.3f} \"\n",
    "      f\"(t = {m1.tvalues['spr63']:.2f}, R2 = {m1.rsquared:.3f})\")\n",
    "# ② 有效性:持有收益变化的一阶自回归(变量 chy6_1 已是预制滞后项)\n",
    "d2 = ts.dropna(subset=['chy6', 'chy6_1'])\n",
    "m2 = sm.OLS(d2['chy6'], sm.add_constant(d2[['chy6_1']])).fit()\n",
    "print(f\"chy6 ~ chy6_1 : b = {m2.params['chy6_1']:.3f} \"\n",
    "      f\"(t = {m2.tvalues['chy6_1']:.2f}, R2 = {m2.rsquared:.3f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59ced944",
   "metadata": {},
   "source": [
    "## 6. Stata ↔ Python 命令对照表\n",
    "\n",
    "| 任务 | Stata | Python |\n",
    "|---|---|---|\n",
    "| 取教材数据 | `use wage1` | `woo.data('wage1')` |\n",
    "| 横截面 OLS | `reg y x, robust` | `sm.OLS(y, X).fit(cov_type='HC1')` |\n",
    "| 面板 FE | `xtreg y x, i(id) fe` | `PanelOLS(..., entity_effects=True).fit()` |\n",
    "| IV/2SLS | `ivregress 2sls y (x = z), robust` | `IV2SLS(...).fit(cov_type='robust')` |\n",
    "| AR(1) | `reg d.y L.d.y` | `sm.OLS` + 预制滞后列（`_1` 后缀） |\n",
    "| 稳健 SE | `, vce(robust)` | `cov_type='HC1'` / `'robust'` |\n",
    "\n",
    "**工作流建议（第 9 章主线）**：stata-mcp 跑主回归，Python fallback 做交叉验证——\n",
    "两个软件结果不一致时，先查数据（样本、缺失、权重），再查命令（SE 类型、自由度修正）。\n",
    "\n",
    "## 7. AI 诚信栏（对应手册第 9 章）\n",
    "\n",
    "- ✅ **允许**：用 AI 生成以上代码框架、解释回归输出、写 Stata/Python 对照\n",
    "- ❌ **不允许**：让 AI 「编」一个结果汇报；跑都没跑就把系数抄进论文；复现结果与教材不一致时隐瞒差异\n",
    "- ⚠️ **必须做**：论文中汇报的工具是 statsmodels/linearmodels 还是 Stata；复现与教材的差异要说明原因（样本版本、自由度修正、设定差异）\n",
    "\n",
    "> 教材数据的好处就在这里：**每个回归都有标准答案**。答案对不上，不是教材错了，是你的代码错了——这是 AI 时代最便宜的自查机制。\n",
    "\n",
    "---\n",
    "*配套《AI 辅助金融研究——学生自学手册》第 9 章 · 2026 秋季*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12.0"
  },
  "title": "第9章配套-Wooldridge教材数据Python复现实战"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}