{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 命名实体识别 (NER)\n",
    "\n",
    "本笔记本来自 [AI for Beginners Curriculum](http://aka.ms/ai-beginners)。\n",
    "\n",
    "在这个示例中，我们将学习如何在 [命名实体识别的标注语料库](https://www.kaggle.com/datasets/abhinavwalia95/entity-annotated-corpus) 数据集上训练 NER 模型。在开始之前，请将 [ner_dataset.csv](https://www.kaggle.com/datasets/abhinavwalia95/entity-annotated-corpus?resource=download&select=ner_dataset.csv) 文件下载到当前目录中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from tensorflow import keras\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据集\n",
    "\n",
    "我们将从将数据集读取到一个数据框开始。如果你想了解更多关于使用 Pandas 的内容，可以访问我们的[数据科学入门](http://aka.ms/datascience-beginners)中的[数据处理课程](https://github.com/microsoft/Data-Science-For-Beginners/tree/main/2-Working-With-Data/07-python)。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sentence #</th>\n",
       "      <th>Word</th>\n",
       "      <th>POS</th>\n",
       "      <th>Tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>Thousands</td>\n",
       "      <td>NNS</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>of</td>\n",
       "      <td>IN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>demonstrators</td>\n",
       "      <td>NNS</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>have</td>\n",
       "      <td>VBP</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>marched</td>\n",
       "      <td>VBN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Sentence #           Word  POS Tag\n",
       "0  Sentence: 1      Thousands  NNS   O\n",
       "1          NaN             of   IN   O\n",
       "2          NaN  demonstrators  NNS   O\n",
       "3          NaN           have  VBP   O\n",
       "4          NaN        marched  VBN   O"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('ner_dataset.csv',encoding='unicode-escape')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们获取唯一标签并创建查找字典，以便我们可以将标签转换为类别编号：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['O', 'B-geo', 'B-gpe', 'B-per', 'I-geo', 'B-org', 'I-org', 'B-tim',\n",
       "       'B-art', 'I-art', 'I-per', 'I-gpe', 'I-tim', 'B-nat', 'B-eve',\n",
       "       'I-eve', 'I-nat'], dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags = df.Tag.unique()\n",
    "tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'O'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2tag = dict(enumerate(tags))\n",
    "tag2id = { v : k for k,v in id2tag.items() }\n",
    "\n",
    "id2tag[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们需要对词汇做同样的处理。为简单起见，我们将在不考虑词频的情况下创建词汇表；在实际应用中，您可能需要使用 Keras 向量化工具，并限制单词的数量。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = set(df['Word'].apply(lambda x: x.lower()))\n",
    "id2word = { i+1 : v for i,v in enumerate(vocab) }\n",
    "id2word[0] = '<UNK>'\n",
    "vocab.add('<UNK>')\n",
    "word2id = { v : k for k,v in id2word.items() }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们需要创建一个句子数据集用于训练。让我们遍历原始数据集，并将所有单独的句子分离为 `X`（单词列表）和 `Y`（标记列表）：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "X,Y = [],[]\n",
    "s,t = [],[]\n",
    "for i,row in df[['Sentence #','Word','Tag']].iterrows():\n",
    "    if pd.isna(row['Sentence #']):\n",
    "        s.append(row['Word'])\n",
    "        t.append(row['Tag'])\n",
    "    else:\n",
    "        if len(s)>0:\n",
    "            X.append(s)\n",
    "            Y.append(t)\n",
    "        s,t = [row['Word']],[row['Tag']]\n",
    "X.append(s)\n",
    "Y.append(t)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([10386,\n",
       "  23515,\n",
       "  4134,\n",
       "  29620,\n",
       "  7954,\n",
       "  13583,\n",
       "  21193,\n",
       "  12222,\n",
       "  27322,\n",
       "  18258,\n",
       "  5815,\n",
       "  15880,\n",
       "  5355,\n",
       "  25242,\n",
       "  31327,\n",
       "  18258,\n",
       "  27067,\n",
       "  23515,\n",
       "  26444,\n",
       "  14412,\n",
       "  358,\n",
       "  26551,\n",
       "  5011,\n",
       "  30558],\n",
       " [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def vectorize(seq):\n",
    "    return [word2id[x.lower()] for x in seq]\n",
    "\n",
    "def tagify(seq):\n",
    "    return [tag2id[x] for x in seq]\n",
    "\n",
    "Xv = list(map(vectorize,X))\n",
    "Yv = list(map(tagify,Y))\n",
    "\n",
    "Xv[0], Yv[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了简单起见，我们将所有句子用0标记填充到最大长度。在现实生活中，我们可能希望使用更聪明的策略，仅在一个小批量内填充序列。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_data = keras.preprocessing.sequence.pad_sequences(Xv,padding='post')\n",
    "Y_data = keras.preprocessing.sequence.pad_sequences(Yv,padding='post')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义标记分类网络\n",
    "\n",
    "我们将使用两层双向 LSTM 网络进行标记分类。为了对最后一层 LSTM 的每个输出应用密集分类器，我们将使用 `TimeDistributed` 构造，它会在每一步将相同的密集层复制到 LSTM 的每个输出：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_4 (Embedding)     (None, 104, 300)          9545400   \n",
      "                                                                 \n",
      " bidirectional_6 (Bidirectio  (None, 104, 200)         320800    \n",
      " nal)                                                            \n",
      "                                                                 \n",
      " bidirectional_7 (Bidirectio  (None, 104, 200)         240800    \n",
      " nal)                                                            \n",
      "                                                                 \n",
      " time_distributed_3 (TimeDis  (None, 104, 17)          3417      \n",
      " tributed)                                                       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 10,110,417\n",
      "Trainable params: 10,110,417\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "maxlen = X_data.shape[1]\n",
    "vocab_size = len(vocab)\n",
    "num_tags = len(tags)\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Embedding(vocab_size, 300, input_length=maxlen),\n",
    "    keras.layers.Bidirectional(keras.layers.LSTM(units=100, activation='tanh', return_sequences=True)),\n",
    "    keras.layers.Bidirectional(keras.layers.LSTM(units=100, activation='tanh', return_sequences=True)),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(num_tags, activation='softmax'))\n",
    "])\n",
    "model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "请注意，这里我们为数据集明确指定了 `maxlen`——如果我们希望网络能够处理可变长度的序列，那么在定义网络时需要更巧妙一些。\n",
    "\n",
    "现在让我们开始训练模型。为了加快速度，我们只训练一个周期，但你可以尝试训练更长时间。此外，你可能希望将数据集的一部分分离出来作为训练数据集，以观察验证准确率。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1499/1499 [==============================] - 740s 488ms/step - loss: 0.0667 - acc: 0.9841\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x16f0bb2a310>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_data,Y_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试结果\n",
    "\n",
    "现在让我们看看实体识别模型在一个示例句子上的表现：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "sent = 'John Smith went to Paris to attend a conference in cancer development institute'\n",
    "words = sent.lower().split()\n",
    "v = keras.preprocessing.sequence.pad_sequences([[word2id[x] for x in words]],padding='post',maxlen=maxlen)\n",
    "res = model(v)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "john -> B-per\n",
      "smith -> I-per\n",
      "went -> O\n",
      "to -> O\n",
      "paris -> B-geo\n",
      "to -> O\n",
      "attend -> O\n",
      "a -> O\n",
      "conference -> O\n",
      "in -> O\n",
      "cancer -> B-org\n",
      "development -> I-org\n",
      "institute -> I-org\n"
     ]
    }
   ],
   "source": [
    "r = np.argmax(res.numpy(),axis=1)\n",
    "for i,w in zip(r,words):\n",
    "    print(f\"{w} -> {id2tag[i]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 关键点\n",
    "\n",
    "即使是简单的 LSTM 模型在命名实体识别（NER）任务中也能表现出不错的效果。然而，如果想要获得更好的结果，可以考虑使用大型预训练语言模型，例如 BERT。使用 Huggingface Transformers 库训练 BERT 进行 NER 的方法可以在[这里](https://huggingface.co/course/chapter7/2?fw=pt)找到。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n---\n\n**免责声明**：  \n本文档使用AI翻译服务 [Co-op Translator](https://github.com/Azure/co-op-translator) 进行翻译。尽管我们努力确保翻译的准确性，但请注意，自动翻译可能包含错误或不准确之处。应以原始语言的文档作为权威来源。对于关键信息，建议使用专业人工翻译。我们对因使用此翻译而引起的任何误解或误读不承担责任。\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 ('py38')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "orig_nbformat": 4,
  "coopTranslator": {
   "original_hash": "254d25052dcca4ef84f59a05f2935bdc",
   "translation_date": "2025-08-31T10:45:16+00:00",
   "source_file": "lessons/5-NLP/19-NER/NER-TF.ipynb",
   "language_code": "zh"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}