<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Mrli&#39;s Blog</title>
  
  <subtitle>别装作很努力，结局不会陪你演戏。</subtitle>
  <link href="/atom.xml" rel="self"/>
  
  <link href="https://nymrli.top/"/>
  <updated>2022-06-03T02:18:07.863Z</updated>
  <id>https://nymrli.top/</id>
  
  <author>
    <name>Mrli</name>
    
  </author>
  
  <generator uri="http://hexo.io/">Hexo</generator>
  
  <entry>
    <title>爬虫——请求参数逆向</title>
    <link href="https://nymrli.top/2022/06/01/%E7%88%AC%E8%99%AB%E2%80%94%E2%80%94%E8%AF%B7%E6%B1%82%E5%8F%82%E6%95%B0%E9%80%86%E5%90%91/"/>
    <id>https://nymrli.top/2022/06/01/爬虫——请求参数逆向/</id>
    <published>2022-06-01T15:13:46.000Z</published>
    <updated>2022-06-03T02:18:07.863Z</updated>
    
    <content type="html"><![CDATA[<h1 id="登陆参数类"><a class="markdownIt-Anchor" href="#登陆参数类"></a> 登陆参数类</h1><h2 id="新浪博客登陆"><a class="markdownIt-Anchor" href="#新浪博客登陆"></a> <a href="http://blog.sina.com.cn/" target="_blank" rel="noopener">新浪博客登陆</a></h2><p>Input输入:</p><ol><li>Username账号</li><li>password密码</li></ol><p>params请求参数：</p><ol><li><a href="https://login.sina.com.cn/sso/prelogin.php?entry=boke&amp;callback=sinaSSOController.preloginCallBack&amp;su=MTUyMzIxMzEy&amp;rsakt=mod&amp;checkpin=1&amp;client=ssologin.js(v1.4.19)&amp;_=1654090585990" target="_blank" rel="noopener">https://login.sina.com.cn/sso/prelogin.php?entry=boke&amp;callback=sinaSSOController.preloginCallBack&amp;su=MTUyMzIxMzEy&amp;rsakt=mod&amp;checkpin=1&amp;client=ssologin.js(v1.4.19)&amp;_=1654090585990</a><ul><li>entry: boke——固定值</li><li>callback: sinaSSOController.preloginCallBack——固定值</li><li>su: MTUyMzIxMzEy——变：加密</li><li>rsakt: mod——固定值</li><li>checkpin: 1——固定值</li><li>client: ssologin.js(v1.4.19)——固定值</li><li>_: 1654090585990——变：13位时间戳</li></ul></li><li><a href="https://login.sina.com.cn/sso/login.php?client=ssologin.js(v1.4.19)&amp;_=1654090586051" target="_blank" rel="noopener">https://login.sina.com.cn/sso/login.php?client=ssologin.js(v1.4.19)&amp;_=1654090586051</a><ul><li>client: ssologin.js(v1.4.19)——固定值</li><li>_: 1654090586051——13位时间戳</li></ul></li></ol><h3 id="针对su参数的解密流程"><a class="markdownIt-Anchor" href="#针对su参数的解密流程"></a> 针对<code>su</code>参数的解密流程</h3><ol><li><p><code>Ctrl+Shift+F</code>全局搜索: <code>su =</code></p></li><li><p>搜索结果中定位到具体的文件：<code>ssologin.js</code></p></li><li><p>Jsonify后阅读代码，定位具体行数：971</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">request = objMerge(request, me.loginExtraFlag);</span><br><span class="line">request = objMerge(request, me.loginExtraQuery);</span><br><span class="line">request.su = sinaSSOEncoder.base64.encode(urlencode(username));</span><br><span class="line"><span class="keyword">if</span> (me.service) &#123;</span><br><span class="line">    request.service = me.service</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li><li><p>给971行: <code>request.su = sinaSSOEncoder.base64.encode(urlencode(username));</code>加上断点</p></li><li><p>重新输入账号密码后，提交登陆请求</p></li><li><p>捕捉4中打的断点，查看参数运行时的值</p></li><li><p>观察971行具体的加密逻辑：</p></li><li><p>扣出sinaSSOEncoder类的定义：</p></li></ol><h3 id="附js中call调用"><a class="markdownIt-Anchor" href="#附js中call调用"></a> 附——JS中call调用</h3><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> a =  a||&#123;&#125;;</span><br><span class="line">(<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>&#123;</span><br><span class="line">    <span class="keyword">this</span>.say =&#123;</span><br><span class="line">        what: <span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>&#123;</span><br><span class="line">            <span class="built_in">console</span>.log(<span class="string">"what"</span>);</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;).call(a);</span><br><span class="line"></span><br><span class="line"><span class="built_in">console</span>.log(a)</span><br><span class="line"><span class="built_in">console</span>.log(a.say.what())</span><br></pre></td></tr></table></figure><h1 id="请求参数类"><a class="markdownIt-Anchor" href="#请求参数类"></a> 请求参数类</h1>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h1 id=&quot;登陆参数类&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#登陆参数类&quot;&gt;&lt;/a&gt; 登陆参数类&lt;/h1&gt;
&lt;h2 id=&quot;新浪博客登陆&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#新浪博客登陆&quot;&gt;&lt;/a&gt; &lt;
      
    
    </summary>
    
    
      <category term="爬虫" scheme="https://nymrli.top/tags/%E7%88%AC%E8%99%AB/"/>
    
      <category term="Python" scheme="https://nymrli.top/tags/Python/"/>
    
      <category term="JS" scheme="https://nymrli.top/tags/JS/"/>
    
  </entry>
  
  <entry>
    <title>每周一个开源项目——ddt-sharp-shooter</title>
    <link href="https://nymrli.top/2022/05/29/%E6%AF%8F%E5%91%A8%E4%B8%80%E4%B8%AA%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE%E2%80%94%E2%80%94ddt-sharp-shoote/"/>
    <id>https://nymrli.top/2022/05/29/每周一个开源项目——ddt-sharp-shoote/</id>
    <published>2022-05-29T10:04:15.000Z</published>
    <updated>2022-05-30T15:40:16.738Z</updated>
    
    <content type="html"><![CDATA[<h1 id="ddt-sharp-shooter"><a class="markdownIt-Anchor" href="#ddt-sharp-shooter"></a> <a href="https://github.com/boring-plans/ddt-sharp-shooter" target="_blank" rel="noopener">ddt-sharp-shooter</a></h1><blockquote><p>这是一个基于 <a href="https://github.com/moses-palmer/pynput" target="_blank" rel="noopener">Pynput</a> 的 DDT 工具。基本原理在于，得知风力、角度、距离的情况下，参考力度表得出发射力度，而后发射。<br>其中，风力、角度通过 <a href="https://github.com/sml2h3/ddddocr" target="_blank" rel="noopener">ddddocr</a>（An awesome captcha recognition library）识别，屏距通过标记屏距测量框、敌我位置来推算，力度通过按压时长来体现，具体见<a href="https://github.com/boring-plans/ddt-sharp-shooter/tree/master" target="_blank" rel="noopener">这里</a>。</p></blockquote><h2 id="使用到的库"><a class="markdownIt-Anchor" href="#使用到的库"></a> 使用到的库:</h2><p><code>screeninfo、pillow、ddddocr、pynput、py2app</code></p><ul><li>pynput: 控制和监视输入设备；类似的有PyHook3（监视键鼠）、pywin32 （模拟键鼠）</li><li>ddddocr：识别验证码，这边用来识别数字</li><li>py2app: 将Python程序打包成MacOS应用程序</li><li>screeninfo: 获得屏幕显示信息：<code>monitors = screeninfo.get_monitors()</code>、<code>_screen_size = (monitors[0].width, monitors[0].height)</code></li><li>pillow: 进行屏幕截图</li></ul><h2 id="whats-new"><a class="markdownIt-Anchor" href="#whats-new"></a> What’s New:</h2><h3 id="1进程间通信"><a class="markdownIt-Anchor" href="#1进程间通信"></a> 1.进程间通信</h3><blockquote><p>Tkinter界面开启mainloop进程，其中又开辟出一个子线程来侦听其他进程发送的数据消息，然后通过<code>tk.Text</code>控件来展示</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> multiprocessing</span><br><span class="line"><span class="keyword">import</span> threading</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> tkinter</span><br><span class="line"><span class="comment"># 声明全局对象及类型</span></span><br><span class="line">_tk: tkinter.Tk</span><br><span class="line">_text: tkinter.Text</span><br><span class="line">_terminate = <span class="keyword">False</span></span><br><span class="line">_queue: multiprocessing.Queue</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">update_text</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""子线程侦听进程消息"""</span></span><br><span class="line">    <span class="keyword">while</span> <span class="keyword">not</span> _terminate:</span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> _queue.empty():</span><br><span class="line">            text = _queue.get(<span class="keyword">False</span>)</span><br><span class="line">            append_text(text)</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            time.sleep(<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">append_text</span><span class="params">(text)</span>:</span></span><br><span class="line">    _text.config(state=<span class="string">'normal'</span>)</span><br><span class="line">    _text.insert(<span class="string">'end'</span>, <span class="string">f'\n<span class="subst">&#123;text&#125;</span>'</span>)</span><br><span class="line">    _text.see(<span class="string">'end'</span>)</span><br><span class="line">    _text.config(state=<span class="string">'disabled'</span>)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(gui_queue)</span>:</span></span><br><span class="line">    <span class="keyword">global</span> _tk, _text, _queue, _screen_size</span><br><span class="line">    _queue = gui_queue</span><br><span class="line">    <span class="comment"># ...</span></span><br><span class="line">    threading.Thread(target=update_text).start()</span><br><span class="line">    </span><br><span class="line">    _tk.mainloop()</span><br></pre></td></tr></table></figure><h3 id="2py2app创建macos应用"><a class="markdownIt-Anchor" href="#2py2app创建macos应用"></a> 2.py2app创建MacOS应用</h3><p><code>py2app setup</code>，在macos下创建python应用， <code>python setup.py py2app</code></p><h3 id="3ddddocr识别数字并清晰"><a class="markdownIt-Anchor" href="#3ddddocr识别数字并清晰"></a> 3.ddddocr识别数字并清晰</h3><p>对纯数字识别结果进行清洗</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">recognize_digits</span><span class="params">(image: bytes)</span>:</span></span><br><span class="line">    <span class="string">"""进行数字识别"""</span></span><br><span class="line">    ocr = ddddocr.DdddOcr(show_ad=<span class="keyword">False</span>)</span><br><span class="line">    result = ocr.classification(image)</span><br><span class="line">    <span class="keyword">return</span> wash_digits(result)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">wash_digits</span><span class="params">(digits: str)</span>:</span></span><br><span class="line">    <span class="string">"""由于不会出现非数字, 所以对易识别错误的字符进行替换"""</span></span><br><span class="line">    washed = digits \</span><br><span class="line">        .replace(<span class="string">'g'</span>, <span class="string">'9'</span>).replace(<span class="string">'q'</span>, <span class="string">'9'</span>) \</span><br><span class="line">        .replace(<span class="string">'l'</span>, <span class="string">'1'</span>).replace(<span class="string">'i'</span>, <span class="string">'1'</span>) \</span><br><span class="line">        .replace(<span class="string">'z'</span>, <span class="string">'2'</span>) \</span><br><span class="line">        .replace(<span class="string">'o'</span>, <span class="string">'0'</span>)</span><br><span class="line">    <span class="keyword">return</span> re.sub(<span class="string">r'\D'</span>, <span class="string">'0'</span>, washed)</span><br></pre></td></tr></table></figure><h3 id="4pynput处理键鼠事件"><a class="markdownIt-Anchor" href="#4pynput处理键鼠事件"></a> 4.pynput处理键鼠事件</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># km.py</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">Keyboard and mouse input/output,</span></span><br><span class="line"><span class="string">and ScreenGrabbing</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="keyword">from</span> pynput <span class="keyword">import</span> keyboard, mouse</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">on_click</span><span class="params">(x, y, btn_type, down)</span>:</span></span><br><span class="line">    <span class="string">"""鼠标点击回调函数"""</span></span><br><span class="line">    <span class="keyword">if</span> btn_type == mouse.Button.left <span class="keyword">and</span> down:</span><br><span class="line">        _queue.put((int(x), int(y)))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">on_press</span><span class="params">(event)</span>:</span></span><br><span class="line">    <span class="string">"""按键回调函数"""</span></span><br><span class="line">    <span class="keyword">try</span>:</span><br><span class="line">        <span class="comment"># 处理正常的ASCII字符</span></span><br><span class="line">        <span class="keyword">if</span> event.char == <span class="string">'q'</span>:</span><br><span class="line">            <span class="comment"># 设置毒药, 当queue.get()获得None的时候, 消费端应该被kill</span></span><br><span class="line">            _queue.put(<span class="keyword">None</span>)</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            _queue.put(event.char)</span><br><span class="line">    <span class="keyword">except</span> AttributeError:</span><br><span class="line">        <span class="comment"># 处理特殊的按键</span></span><br><span class="line">        <span class="keyword">if</span> event == keyboard.Key.esc:</span><br><span class="line">            _queue.put(<span class="string">'esc'</span>)</span><br><span class="line">        <span class="keyword">elif</span> event == keyboard.Key.enter:</span><br><span class="line">            _queue.put(<span class="string">'enter'</span>)</span><br><span class="line">        <span class="keyword">elif</span> event == keyboard.Key.space:</span><br><span class="line">            _queue.put(<span class="string">' '</span>)</span><br><span class="line">        <span class="keyword">elif</span> event == keyboard.Key.backspace:</span><br><span class="line">            _queue.put(<span class="string">'delete'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">space_press_and_release</span><span class="params">(duration)</span>:</span></span><br><span class="line">    <span class="string">"""Press the key 'space' down for a while, and then release"""</span></span><br><span class="line">    <span class="comment"># @param duration: 持续时间</span></span><br><span class="line">    k = keyboard.Controller()</span><br><span class="line">    k.press(keyboard.Key.space)</span><br><span class="line">    time.sleep(duration)</span><br><span class="line">    k.release(keyboard.Key.space)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">key_press_and_release</span><span class="params">(key)</span>:</span></span><br><span class="line">    <span class="string">"""Press certain key several times down, and then release immediately"""</span></span><br><span class="line">    <span class="comment"># @param key: 按下按键</span></span><br><span class="line">    k = keyboard.Controller()</span><br><span class="line">    k.press(key)</span><br><span class="line">    k.release(key)</span><br><span class="line">    time.sleep(<span class="number">0.37</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(km_queue)</span>:</span></span><br><span class="line">    <span class="keyword">global</span> _queue</span><br><span class="line">    _queue = km_queue</span><br><span class="line"><span class="comment"># 注册键盘事件监听与回调函数</span></span><br><span class="line">    keyboard_listener = keyboard.Listener(on_press=on_press)</span><br><span class="line">    keyboard_listener.start()</span><br><span class="line">    time.sleep(<span class="number">.5</span>)</span><br><span class="line">    <span class="comment"># 注册鼠标事件监听与回调函数</span></span><br><span class="line">    mouse_listener = mouse.Listener(on_click=on_click)</span><br><span class="line">    mouse_listener.start()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    run(<span class="keyword">None</span>)</span><br><span class="line">    <span class="keyword">while</span> <span class="keyword">True</span>:</span><br><span class="line">        time.sleep(<span class="number">1</span>)</span><br></pre></td></tr></table></figure><p>on_press函数只是产生了queue的数据，数据具体是在main.py中的<code>handle_inputs</code>中被消费的</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">km_listen_queue</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="keyword">while</span> <span class="keyword">True</span>:</span><br><span class="line">        inputs = _km_queue.get()</span><br><span class="line">        <span class="keyword">if</span> inputs <span class="keyword">is</span> <span class="keyword">None</span>:  <span class="comment"># poison bill</span></span><br><span class="line">            <span class="comment"># 接收毒药, 毒药的设置可见上文Listener中回调函数是何时put None的</span></span><br><span class="line">            <span class="keyword">if</span> len(_wind_degree_points) &gt; <span class="number">0</span>:</span><br><span class="line">                <span class="keyword">with</span> open(_W_D_POINTS_DUMP_NAME, <span class="string">'wb'</span>) <span class="keyword">as</span> f:</span><br><span class="line">                    pickle.dump(_wind_degree_points, f)</span><br><span class="line">            _gui_process.terminate()</span><br><span class="line">            <span class="keyword">break</span></span><br><span class="line">        <span class="keyword">else</span>:  <span class="comment"># handle inputs</span></span><br><span class="line">            handle_inputs(inputs)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">handle_inputs</span><span class="params">(inputs)</span>:</span></span><br><span class="line">    <span class="string">"""To handle inputs"""</span></span><br><span class="line">    <span class="keyword">global</span> _command_flag, _direct_force_typing, _wind_direction</span><br><span class="line">    inputs_type = type(inputs)</span><br><span class="line">    <span class="comment"># 键盘事件on_press, 返回str</span></span><br><span class="line">    <span class="keyword">if</span> inputs_type == str:</span><br><span class="line">        <span class="comment"># press ESC to cancel</span></span><br><span class="line">        <span class="keyword">if</span> inputs == <span class="string">'esc'</span>:</span><br><span class="line">            reset_inputs()</span><br><span class="line">        <span class="keyword">elif</span> inputs == <span class="string">'-'</span>:</span><br><span class="line">            _wind_direction = <span class="number">-1</span></span><br><span class="line">        <span class="keyword">elif</span> inputs == <span class="string">'='</span>:</span><br><span class="line">            _wind_direction = <span class="number">1</span></span><br><span class="line">        <span class="comment"># press the key 't' twice to enable command mode</span></span><br><span class="line">        <span class="keyword">elif</span> inputs == <span class="string">'t'</span>:</span><br><span class="line">            _command_flag += <span class="number">1</span></span><br><span class="line">            <span class="comment"># 最大t的次数为4, 所以4为一个周期</span></span><br><span class="line">            _command_flag %= <span class="number">4</span></span><br><span class="line">            <span class="keyword">if</span> _command_flag == <span class="number">2</span>:</span><br><span class="line">                _gui_queue.put(<span class="string">"指令输入开启💡"</span>)</span><br><span class="line">            <span class="keyword">elif</span> _command_flag == <span class="number">0</span>:</span><br><span class="line">                _gui_queue.put(<span class="string">"指令输入关闭🔒"</span>)</span><br><span class="line">        <span class="keyword">elif</span> _command_flag == <span class="number">2</span>:</span><br><span class="line">            <span class="comment"># _command_flag == 2此时为命令模式, 输入enter完成设置, reset_inputs</span></span><br><span class="line">            <span class="comment"># press enter to submit command and fire</span></span><br><span class="line">            <span class="keyword">if</span> inputs == <span class="string">'enter'</span>:</span><br><span class="line">                direct_force = analyse_direct_force()</span><br><span class="line">                <span class="keyword">if</span> direct_force &gt; <span class="number">0</span>:</span><br><span class="line">                    fire(force=direct_force)</span><br><span class="line">                <span class="keyword">else</span>:</span><br><span class="line">                    wind, degree, distance = analyse_wind(), analyse_degree(), analyse_distance()</span><br><span class="line">                    fire(wind, degree, distance)</span><br><span class="line">                reset_inputs()</span><br><span class="line">            <span class="comment"># edit command</span></span><br><span class="line">            <span class="keyword">elif</span> inputs == <span class="string">'delete'</span>:</span><br><span class="line">                _direct_force_typing = _direct_force_typing[:<span class="number">-1</span>]</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                _direct_force_typing += inputs</span><br><span class="line">        <span class="comment"># when not in command mode</span></span><br><span class="line">        <span class="comment"># any key except 't' will reset mode flag</span></span><br><span class="line">        <span class="comment"># which means only consecutive 't' input can enable command mode</span></span><br><span class="line">        <span class="keyword">elif</span> _command_flag == <span class="number">1</span>:</span><br><span class="line">            reset_inputs()</span><br><span class="line">    <span class="comment"># 鼠标事件on_click, 返回(x, y)</span></span><br><span class="line">    <span class="keyword">elif</span> inputs_type == tuple:</span><br><span class="line">        <span class="keyword">if</span> _command_flag == <span class="number">2</span>:</span><br><span class="line">            _distance_points.append(inputs)</span><br><span class="line">            _gui_queue.put(<span class="string">f'<span class="subst">&#123;len(_distance_points)&#125;</span> 个点已标记'</span>)</span><br><span class="line">        <span class="comment"># 按三次t之后再点击, 进入设置“角度中心位置、风力中心位置”</span></span><br><span class="line">        <span class="keyword">if</span> _command_flag == <span class="number">3</span>:</span><br><span class="line">            <span class="keyword">if</span> len(_wind_degree_points) == <span class="number">2</span>:</span><br><span class="line">                <span class="comment"># 如果标记过了风力和角度, 则重新标记</span></span><br><span class="line">                _wind_degree_points.clear()</span><br><span class="line">            _wind_degree_points.append(inputs)</span><br><span class="line">            <span class="keyword">if</span> len(_wind_degree_points) == <span class="number">1</span>:</span><br><span class="line">                _gui_queue.put(<span class="string">'角度位置已标记📐️'</span>)</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                _gui_queue.put(<span class="string">'风力位置已标记🌪️'</span>)</span><br></pre></td></tr></table></figure><h3 id="5屏幕截图"><a class="markdownIt-Anchor" href="#5屏幕截图"></a> 5.屏幕截图</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">grab_box</span><span class="params">(box: tuple)</span> -&gt; bytes:</span></span><br><span class="line">    <span class="comment"># 开启内存IO</span></span><br><span class="line">    bytes_io = io.BytesIO()</span><br><span class="line">    <span class="comment"># 截图</span></span><br><span class="line">    image = ImageGrab.grab().resize((_screen_size[<span class="number">0</span>], _screen_size[<span class="number">1</span>])).crop(box)</span><br><span class="line">    <span class="comment"># 写入内存IO</span></span><br><span class="line">    image.save(bytes_io, format=<span class="string">'png'</span>)</span><br><span class="line">    <span class="comment"># 获得bytes值</span></span><br><span class="line">    <span class="keyword">return</span> bytes_io.getvalue()</span><br></pre></td></tr></table></figure><h2 id="执行流程"><a class="markdownIt-Anchor" href="#执行流程"></a> 执行流程</h2><blockquote><p><code>mian.py</code>为程序入口</p></blockquote><ol start="0"><li><p>定义全局变量</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">_W_D_POINTS_DUMP_NAME = <span class="string">'wind_degree_points.list'</span></span><br><span class="line">_PRESS_DURATION_PER_FORCE = <span class="number">4.1</span> / <span class="number">100</span></span><br><span class="line">_screen_size: tuple</span><br><span class="line">_command_flag = <span class="number">0</span></span><br><span class="line">_direct_force_typing = <span class="string">''</span></span><br><span class="line">_wind_direction = <span class="number">-1</span></span><br><span class="line">_distance_unit = <span class="number">0</span>  <span class="comment"># 用于像素与屏距转换</span></span><br><span class="line">_distance_points = []  <span class="comment"># 用于计算 _distance_unit 以及 屏距</span></span><br><span class="line">_wind_degree_points = []  <span class="comment"># 用于配置角度、风力 屏幕截取位置的点</span></span><br><span class="line">_gui_process: multiprocessing.Process</span><br><span class="line"><span class="comment"># GUI是另外的进程, 所以需要使用进程间的队列</span></span><br><span class="line">_gui_queue = multiprocessing.Queue()</span><br><span class="line"><span class="comment"># 相比之下, km是主进程中的, 不涉及进程间数据通信, 所以直接使用普通Queue就行</span></span><br><span class="line">_km_queue = Queue()</span><br></pre></td></tr></table></figure></li><li><p>获得屏幕信息：<code>_screen_size = (monitors[0].width, monitors[0].height)</code></p></li><li><p>开启GUI进程: <code>_gui_process = multiprocessing.Process(target=gui_run, args=(_gui_queue,))</code></p></li><li><p>开启pynput的键鼠侦听Listener线程</p></li><li><p>开启键鼠处理线程</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">threading.Thread(target=km_listen_queue).start()</span><br><span class="line"><span class="comment"># km_listen_queue中接收poison bill(即按下"q")时, 关闭进程</span></span><br><span class="line">threading.Thread(target=gui_check_alive).start()</span><br><span class="line"><span class="comment"># GUI检测心跳线程, 当GUI的状态不是is_alive时, 向queue中投递毒药, 将terminate设置为False. 从而关闭GUI进程中的update_text线程</span></span><br></pre></td></tr></table></figure></li><li><p>在键鼠处理线程中，根据键鼠输入值，进行功能生效</p></li></ol><h2 id="代码风格特点"><a class="markdownIt-Anchor" href="#代码风格特点"></a> 代码风格特点：</h2><p>每个模块（py文件）都有自己的全局变量，通过主程序调用模块函数时进行传引用，而变量全在主程序<code>main.py</code>中定义。</p><h2 id="总结"><a class="markdownIt-Anchor" href="#总结"></a> 总结：</h2><p>涉及了<strong>多线程、多进程、图像识别、屏幕截图识别、键鼠检测、打包应用</strong>等内容，其中对于①多线程和多进程使用、②识别结果后的针对纯数字数据进行清洗；③消息队列的使用、毒药设置；都比较让人有收获。</p><h1 id="视频思路"><a class="markdownIt-Anchor" href="#视频思路"></a> 视频思路</h1><ol><li>讲解程序功能</li><li>介绍使用到的库</li><li>讲解程序文件结构</li><li>讲解每个文件实现</li><li>介绍优点和有价值的</li><li>介绍缺点<ol><li>C式的全局变量风格</li></ol></li></ol><p><a href="https://www.bilibili.com/video/BV1iz411b7Fa/" target="_blank" rel="noopener">【PR教程】2分钟学会制作视频内容导航条</a></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h1 id=&quot;ddt-sharp-shooter&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#ddt-sharp-shooter&quot;&gt;&lt;/a&gt; &lt;a href=&quot;https://github.com/boring-plans/ddt-sharp-sh
      
    
    </summary>
    
    
      <category term="Python" scheme="https://nymrli.top/tags/Python/"/>
    
      <category term="开源项目" scheme="https://nymrli.top/tags/%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE/"/>
    
  </entry>
  
  <entry>
    <title>爬虫App——夜神模拟器xposed+inspeckage</title>
    <link href="https://nymrli.top/2022/05/10/%E7%88%AC%E8%99%ABApp%E2%80%94%E2%80%94xposed-inspeckage/"/>
    <id>https://nymrli.top/2022/05/10/爬虫App——xposed-inspeckage/</id>
    <published>2022-05-10T06:34:49.000Z</published>
    <updated>2022-05-10T12:47:02.087Z</updated>
    
    <content type="html"><![CDATA[<p>头一次接触到App参数加密的，因此需要对加密参数进行反编译，了解到有工具inspeckage可以方便的来找寻加密参数。而其使用的条件为</p><ul><li>手机已ROOT</li><li>在Xposed框架下运行</li></ul><p>于是开始在夜神模拟器上捣鼓安装xposed、inspeckage。</p><h2 id="安装"><a class="markdownIt-Anchor" href="#安装"></a> 安装</h2><h3 id="xposed框架"><a class="markdownIt-Anchor" href="#xposed框架"></a> xposed框架</h3><p>在不少地方都下载了xposed框架，但都不太使用，后来了解到模拟器的xposed是专用的，根据夜神模拟器官网介绍的<a href="https://support.yeshen.com/zh-CN/qt/xp" target="_blank" rel="noopener">夜神模拟器如何安装xposed框架</a>后，决定还是在<strong>模拟器的应用商店</strong>进行安装xposed，下载安装后的名称为&quot;xposed 派大星&quot;。</p><p>下载安装后立马打开还是会显示没安装，需要点击&quot;安装/更新&quot;下的选项进行安装才行。在安装完成后再选择<strong>重启模拟器</strong>。</p><h3 id="inspeckage模块"><a class="markdownIt-Anchor" href="#inspeckage模块"></a> inspeckage模块</h3><blockquote><p>Inspeckage是一个用于提供Android应用程序动态分析的工具。通过对Android API的函数使用hook技术，帮助用户了解应用程序在运行时的行为——crypto菜单栏下会hook捕捉常见的加密，Hash菜单栏下会hook捕捉常见的hash。</p></blockquote><p>原以为需要额外去搜寻下载，后来发现推荐的都是安装好xposed软件后，在其菜单栏的&quot;下载&quot;中搜寻&quot;inspeckage&quot;模块使用（搜索结果为Inspeckage-Andriod package inspector）后安装，安装成功切换到菜单栏的&quot;模块&quot;中，勾选&quot;inspeckage&quot;后<strong>重启模拟器</strong>。</p><p>注：xposed的作用只是为了给inspeckage提供运行环境。</p><h2 id="启动inspeckage"><a class="markdownIt-Anchor" href="#启动inspeckage"></a> 启动inspeckage</h2><blockquote><p>参考：Inspeckage插件的运用</p></blockquote><ol><li><p>打开模拟器的inspeckage程序，<code>choose target</code>：选择目标app程序</p><p><img src="/2022/05/10/爬虫App——xposed-inspeckage/E:%5Chexo%5Csource_posts%5C%E7%88%AC%E8%99%ABApp%E2%80%94%E2%80%94xposed-inspeckage%5C%E6%A8%A1%E6%8B%9F%E5%99%A8inspeckage.jpg" alt="模拟器inspeckage"></p></li><li><p>点击&quot;Lauch APP&quot;打开要捕获程序</p></li><li><p>在宿主机cmd里面输入命令<code>adb forward tcp:8008 tcp:8008</code>，然后在浏览器里输入http://127.0.0.1:8008 就会看到Inspeckage的界面</p></li></ol><p><img src="/2022/05/10/爬虫App——xposed-inspeckage/E:%5Chexo%5Csource_posts%5C%E7%88%AC%E8%99%ABApp%E2%80%94%E2%80%94xposed-inspeckage%5Cinspeckage%E7%BD%91%E9%A1%B5.png" alt="inspeckage网页"></p><p>注：得手动点击ON才能实时抓信息。</p><p>更多介绍见：<a href="https://blog.csdn.net/tom__chen/article/details/78216732" target="_blank" rel="noopener">Inspeckage使用笔记（app安全检测工具）</a></p><h2 id="使用inspeckage"><a class="markdownIt-Anchor" href="#使用inspeckage"></a> 使用inspeckage</h2><p>菜单页说明</p><table><thead><tr><th style="text-align:center">模块名</th><th style="text-align:center">功能</th></tr></thead><tbody><tr><td style="text-align:center">Logcat</td><td style="text-align:center">查看该app的logcat输出</td></tr><tr><td style="text-align:center">Tree View</td><td style="text-align:center">浏览app的数据目录并直接下载文件到本地</td></tr><tr><td style="text-align:center">Package Information</td><td style="text-align:center">应用基本信息（组件信息、权限信息、共享库信息）</td></tr><tr><td style="text-align:center">Shared Preferences</td><td style="text-align:center">LOG：app XML文件读写记录；Files：具体XML写入内容</td></tr><tr><td style="text-align:center">Serialization</td><td style="text-align:center">反序列化记录</td></tr><tr><td style="text-align:center">Crypto</td><td style="text-align:center">常见加解密记录，AES、DES、RSA（KEY、IV值）</td></tr><tr><td style="text-align:center">Hash</td><td style="text-align:center">常见的哈希算法记录</td></tr><tr><td style="text-align:center">SQLite</td><td style="text-align:center">SQLite数据库操作记录</td></tr><tr><td style="text-align:center">HTTP</td><td style="text-align:center">HTTP网络请求记录</td></tr><tr><td style="text-align:center">File System</td><td style="text-align:center">文件读写记录</td></tr><tr><td style="text-align:center">Misc.</td><td style="text-align:center">调用Clipboard,URL.Parse()记录</td></tr><tr><td style="text-align:center">WebView</td><td style="text-align:center">调用webview内容</td></tr><tr><td style="text-align:center">IPC</td><td style="text-align:center">进程之间通信记录</td></tr><tr><td style="text-align:center">+Hooks</td><td style="text-align:center">运行过程中用户自定义Hook记录</td></tr></tbody></table><p>注：inspeckage主要是用来hook加密的，对hash检测的效果可能不是很好——AXHC</p><p><strong>使用说明</strong>：</p><p>如果是要参看加密前的数据（解密数据），可以直接查看Crypto菜单栏下的内容，搜索你所需要找的加密参数就可以得到原文和使用的加密、Hash类型，直接秒杀！</p><p>比如:<code>Algorithm( MD5 ) [eexssdsd : zxcxzc]</code>，说明使用的算法为MD5， <code>[plaintext : encrypted]</code></p><p><strong>原理分析</strong>：那么这个效果是怎么做到的呢？其实很简单，就是直接将Java标准库中常见的被用于生成加密参数的方法给Hook了，监听它们的输入参数和返回值，这样就能直接得到加密、Hash前的原文、密钥、IV等内容了，怎么样？是不是很简单？——相对于反编译app，自己再看代码实现简单、快捷多了。</p><p><strong>实战</strong>：</p><ul><li>解析sign参数的明文：<a href="https://blog.csdn.net/qq_38832624/article/details/100897605" target="_blank" rel="noopener">https://blog.csdn.net/qq_38832624/article/details/100897605</a></li><li><a href="https://blog.csdn.net/fatesunlove/article/details/105146882" target="_blank" rel="noopener">inspeckage使用实战两例</a></li><li><a href="https://blog.csdn.net/z434890/article/details/100528704" target="_blank" rel="noopener">app反编译实战三：工具inspeckage的使用</a>——识货MD5参数解析</li><li>Jadx反编译：<a href="https://cloud.tencent.com/developer/article/1447630?from=article.detail.1548593" target="_blank" rel="noopener">https://cloud.tencent.com/developer/article/1447630?from=article.detail.1548593</a></li><li>视频教程：<ul><li><a href="https://space.bilibili.com/175052335" target="_blank" rel="noopener">APP逆向解密 安卓HOOK</a>——恰恰小视频，复合加密: MD5(password, “saltA” + phoneNum + “saltB”)</li><li><a href="https://www.bilibili.com/video/BV1Qa4y1J7bc?spm_id_from=333.337.search-card.all.click" target="_blank" rel="noopener">安卓逆向 HOOK APP加密字段sign解密</a>——众赢时空</li></ul></li></ul><h1 id="附录"><a class="markdownIt-Anchor" href="#附录"></a> 附录</h1><h2 id="mudule-disabled"><a class="markdownIt-Anchor" href="#mudule-disabled"></a> Mudule disabled</h2><p><strong>Module disabled</strong>表明<a href="http://lib.csdn.net/base/softwaretest" target="_blank" rel="noopener">测试</a>环境中没有安装<em><strong>Xposed框架*</strong>，用户可以在没有Xposed的情况下运行</em><em>Inspeckage</em>*，但是绝大部分功能都要依赖Xposed框架，所以建议在测试环境中安装该框架。</p><p>如果安装Xposed后，并且inspeckage就是通过对应的Xposed软件下载的，那么检查&quot;模块&quot;中是否将&quot;inspeckage&quot;勾选，然后重启。</p><h2 id="手机目录"><a class="markdownIt-Anchor" href="#手机目录"></a> 手机目录</h2><p>MT管理器能看到</p><ul><li><code>/data/data/</code> 下是手机系统文件和你安装软件的数据文件，不建议随便删除，或造成死机或开不了机。需要root</li><li><code>/etc/data</code></li></ul><h2 id="反编译"><a class="markdownIt-Anchor" href="#反编译"></a> 反编译</h2><blockquote><p>如果程序被加固(加壳，腾讯乐固、360加固)，则需要去壳后反编译，比如脱壳工具FDex2，脱壳后得到多个dex文件，将其压缩成压缩包，再使用反编译如jadx导入分析。</p></blockquote>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;头一次接触到App参数加密的，因此需要对加密参数进行反编译，了解到有工具inspeckage可以方便的来找寻加密参数。而其使用的条件为&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;手机已ROOT&lt;/li&gt;
&lt;li&gt;在Xposed框架下运行&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;于是开始在夜神模拟器上捣
      
    
    </summary>
    
    
      <category term="爬虫" scheme="https://nymrli.top/tags/%E7%88%AC%E8%99%AB/"/>
    
  </entry>
  
  <entry>
    <title>toy reimplementation of an event loop in Python[翻译]</title>
    <link href="https://nymrli.top/2022/05/08/toy-reimplementation-of-an-event-loop-in-Python-%E7%BF%BB%E8%AF%91/"/>
    <id>https://nymrli.top/2022/05/08/toy-reimplementation-of-an-event-loop-in-Python-翻译/</id>
    <published>2022-05-08T07:34:46.000Z</published>
    <updated>2022-05-09T14:04:56.146Z</updated>
    
    <content type="html"><![CDATA[<h1 id="事件循环的一个小故事"><a class="markdownIt-Anchor" href="#事件循环的一个小故事"></a> 事件循环的一个小故事</h1><blockquote><p>Translated from: <a href="https://github.com/AndreLouisCaron/a-tale-of-event-loops" target="_blank" rel="noopener">https://github.com/AndreLouisCaron/a-tale-of-event-loops</a></p></blockquote><p>我最近被伯克利大学的研究员Nathaniel J. Smith关于 <a href="https://docs.python.org/3/library/asyncio.html" target="_blank" rel="noopener">asyncio</a> 的惊人见解所困。他的文章《关于后async/await世界中异步API设计的一些想法》( <a href="https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/" target="_blank" rel="noopener">Some thoughts on asynchronous API design in a post-async/await world</a>) 使得我 在最近大量使用asyncio后，对它有了一些（略）复杂的感受。</p><p>虽然这或多或少是他文章的重点，但他认为 <a href="https://github.com/dabeaz/curio" target="_blank" rel="noopener">curio</a> 的实现比asyncio简单得多，因为它预先假定你有一个实现<a href="https://www.python.org/dev/peps/pep-0492/" target="_blank" rel="noopener">PEP 492</a>（<code>async</code>和<code>await</code>关键字）的Python版本。</p><p>我很感兴趣，看了一下curio的源代码，…由于它获得了很多功能（有希望使它更接近于一个可生产的库），我认为它现在已经拥有足够的功能，主要的本质已经被淡化。这绝对不是一件坏事，但如果是像我一样，想了解Nathaniel在说什么，并学习一些整洁的Python技巧的话，阅读curio的源代码几乎不是一个好的起点。</p><p>Python核心开发人员Brett Cannon在他的帖子《<a href="http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5" target="_blank" rel="noopener">How the heck does async/await work in Python 3.5?</a>》中提供了一些关于如何使用协程对象（例如，如果你正在实现事件循环）非常好的见解，他的重点是实现细节，至少对我来说，要摸清coroutine还是缺少一小块。</p><p>TL; DR: 这是我试图抓住curio所基于的核心基本原理的尝试。希望它能成为熟悉curio源代码的一个好的垫脚石。</p><p>让我们开始吧 😃</p><h2 id="协程的本质"><a class="markdownIt-Anchor" href="#协程的本质"></a> 协程的本质</h2><p>首先，让我们来看看当调用一个协程时会发生什么。</p><p><strong>提示</strong>：你得到一个 “coroutine对象”，它有一个<code>.send()</code>和一个<code>.throw()</code>方法，就像生成器对象那样。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>coro = hello(<span class="string">'world'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>type(coro)</span><br><span class="line">&lt;<span class="class"><span class="keyword">class</span> '<span class="title">coroutine</span>'&gt;</span></span><br><span class="line"><span class="class">&gt;&gt;&gt; <span class="title">type</span><span class="params">(coro.send)</span></span></span><br><span class="line"><span class="class">&lt;<span class="title">class</span> '<span class="title">builtin_function_or_method</span>'&gt;</span></span><br><span class="line"><span class="class">&gt;&gt;&gt; <span class="title">type</span><span class="params">(coro.throw)</span></span></span><br><span class="line"><span class="class">&lt;<span class="title">class</span> '<span class="title">builtin_function_or_method</span>'&gt;</span></span><br><span class="line"><span class="class">&gt;&gt;&gt;</span></span><br><span class="line"><span class="class">...</span></span><br></pre></td></tr></table></figure><p>当然，你通常不会使用这些方法，因为它们隐藏在<code>await coro</code>和<code>asyncio.get_event_loop().run_until_complete()</code>后面，但既然我们想研究它的工作原理… 😃</p><p>请注意，上面的代码从来没有打印过我们的 <code>&quot;Hello, world!&quot;</code>信息。这是因为我们从未在coroutine函数中实际执行过语句——我们只是创建了coroutine对象。事实上，如果你在解释器中实际运行这段代码，你会得到一个警告，说明我们的hello coroutine从未完成。</p><p>如果你想执行coroutine函数，你需要以某种方式安排它。为了做到这一点，我们将调用<code>.send()</code>方法。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>task = hello(<span class="string">'world'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>    task.send(<span class="keyword">None</span>)</span><br><span class="line"><span class="meta">... </span><span class="keyword">except</span> StopIteration:</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">pass</span></span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><p>就像生成器对象一样，一个协程对象的<code>.send()</code>方法在协程结束时引发<code>StopIteration</code>事件。</p><p>我们将在后面看到，我们可以使用.send()方法在第二次、第三次或第N次调用<code>.send()</code>方法重新开始时将信息传递给coroutine。</p><p>如果该循环程序没有正常返回，而是出现了异常，那么该异常将通过<code>.send()</code>传播回来。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="class"><span class="keyword">class</span> <span class="title">HelloException</span><span class="params">(Exception)</span>:</span></span><br><span class="line"><span class="meta">... </span>   <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name)</span>:</span></span><br><span class="line"><span class="meta">... </span>       self._name = name</span><br><span class="line"><span class="meta">... </span>   <span class="function"><span class="keyword">def</span> <span class="title">__str__</span><span class="params">(self)</span>:</span></span><br><span class="line"><span class="meta">... </span>       <span class="keyword">return</span> <span class="string">'Hello, %s!'</span> % (self._name,)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">raise</span> HelloException(name)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>task = hello(<span class="string">'world'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>    task.send(<span class="keyword">None</span>)</span><br><span class="line"><span class="meta">... </span><span class="keyword">except</span> Exception <span class="keyword">as</span> error:</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: exception will be propagated here!</span></span><br><span class="line"><span class="meta">... </span>    print(error)</span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><p>我还提到了一个<code>.throw()</code>方法。和<code>.send()</code>一样，它恢复了协程，但它不是传递一个值，而是在协程中的暂停点引发一个异常。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="class"><span class="keyword">class</span> <span class="title">Hello</span><span class="params">(Exception)</span>:</span></span><br><span class="line"><span class="meta">... </span>   <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name)</span>:</span></span><br><span class="line"><span class="meta">... </span>       self._name = name</span><br><span class="line"><span class="meta">... </span>   <span class="function"><span class="keyword">def</span> <span class="title">__str__</span><span class="params">(self)</span>:</span></span><br><span class="line"><span class="meta">... </span>       <span class="keyword">return</span> <span class="string">'Hello, %s!'</span> % (self._name,)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">pass</span></span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>task = hello()</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: inject exception.</span></span><br><span class="line"><span class="meta">... </span>    task.throw(Hello(<span class="string">'world'</span>))</span><br><span class="line"><span class="meta">... </span><span class="keyword">except</span> Exception <span class="keyword">as</span> error:</span><br><span class="line"><span class="meta">... </span>    print(error)</span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><p>在这一点上，你应该对这样一个事实感到满意：coroutine 对象与生成器对象非常非常相似，后者从 Python 2.2 (<a href="https://www.python.org/dev/peps/pep-0255/" target="_blank" rel="noopener">PEP 255</a>) 开始就存在，并且从 Python 2.5 (<a href="https://www.python.org/dev/peps/pep-0342/" target="_blank" rel="noopener">PEP 342</a>) 开始就有 <code>.send()</code>和 <code>.throw()</code>方法。</p><h2 id="与事件循环的对话"><a class="markdownIt-Anchor" href="#与事件循环的对话"></a> 与事件循环的对话</h2><p>如果你仔细观察（或尝试），你会发现，与生成器函数相反，coroutine函数不能使用<code>yield</code>表达式。这就提出了（而不是 <a href="http://grammarist.com/usage/begging-the-question-usage/" target="_blank" rel="noopener">begs</a>）一个问题：到底怎样才能让coroutine函数将控制权交还给调用<code>.send()</code>的代码？</p><p>答案是在一个<em>awaitalbe</em>对象上使用<code>await</code>。对于一个对象来说，它必须实现特殊的<code>__await__()</code>方法，以返回一个iterable对象。在实践中，这有点尴尬，所以在标准库中有一个<code>@types.coroutine</code>装饰器，允许你以一种类似<code>@contextlib.contextmanager</code>的风格来创建awaitable对象。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: this is an awaitable object!</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">nice</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span></span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: this makes ``.send()`` return!</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">await</span> nice()</span><br><span class="line"><span class="meta">... </span>    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>task = hello(<span class="string">'world'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: call send twice!</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>task.send(<span class="keyword">None</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>    task.send(<span class="keyword">None</span>)</span><br><span class="line"><span class="meta">... </span><span class="keyword">except</span> StopIteration:</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">pass</span></span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><p>当然，我们的<code>nice()</code>对象是非常无用的。别着急，我们很快就会做一些更有用的事情。</p><h2 id="looping"><a class="markdownIt-Anchor" href="#looping"></a> Looping</h2><p>我们前面的例子正好调用<code>.send()</code>两次，因为它知道<code>hello()</code>只产生一次控制。当我们不知道时（常见的情况），我们需要把它放在一个循环中。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">nice</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span></span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: yield many times!</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">5</span>):</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">await</span> nice()</span><br><span class="line"><span class="meta">... </span>    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>task = hello(<span class="string">'world'</span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: loop!</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">while</span> <span class="keyword">True</span>:</span><br><span class="line"><span class="meta">... </span>        task.send(<span class="keyword">None</span>)</span><br><span class="line"><span class="meta">... </span><span class="keyword">except</span> StopIteration:</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">pass</span></span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><p>我们逐渐开始得到了一些最简单的实现，其类似于<code>asyncio.get_event_loop().run_until_complete()</code>。所以我们让它在语法上更加相似。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">nice</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span></span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">5</span>):</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">await</span> nice()</span><br><span class="line"><span class="meta">... </span>    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: now a reusable function!</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">while</span> <span class="keyword">True</span>:</span><br><span class="line"><span class="meta">... </span>            task.send(<span class="keyword">None</span>)</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">except</span> StopIteration:</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">pass</span></span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: call it as a function!</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>run_until_complete(hello(<span class="string">'world'</span>))</span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><h2 id="生成子任务"><a class="markdownIt-Anchor" href="#生成子任务"></a> 生成子任务</h2><p>现在我们已经有了一个可以完成单个任务的事件循环，我们可能想开始做一些有用的事情。我们期望做许多不同的事情，但由于这是关于并发的，让我们从允许创建子任务开始。</p><p>我们在这里需要做的主要事情是引入一个新的原语<code>spawn()</code>，用于安排新的子任务。一旦任务被安排好，我们要把控制权返回给父任务，这样它就可以继续前进了。</p><p><strong>注意</strong>：这个例子是故意不完整的。我们以后会看到如何join任务。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> inspect <span class="keyword">import</span> iscoroutine</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: awaitable object that sends a request to launch a child task!</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">spawn</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span> task</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">await</span> nice()</span><br><span class="line"><span class="meta">... </span>    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: parent task!</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>     <span class="comment"># NEW: create a child task!</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">await</span> spawn(hello(<span class="string">'world'</span>))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: schedule the "root" task.</span></span><br><span class="line"><span class="meta">... </span>    tasks = [(task, <span class="keyword">None</span>)]</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">while</span> tasks:</span><br><span class="line"><span class="meta">... </span>        <span class="comment"># NEW: round-robin between a set tasks (we may now have more</span></span><br><span class="line"><span class="meta">... </span>        <span class="comment">#      than one and we'd like to be as "fair" as possible).</span></span><br><span class="line"><span class="meta">... </span>        queue, tasks = tasks, []</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">for</span> task, data <span class="keyword">in</span> queue:</span><br><span class="line"><span class="meta">... </span>            <span class="comment"># NEW: resume the task *once*.</span></span><br><span class="line"><span class="meta">... </span>            <span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>                data = task.send(data)</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">except</span> StopIteration:</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">pass</span></span><br><span class="line"><span class="meta">... </span>            <span class="keyword">except</span> Exception <span class="keyword">as</span> error:</span><br><span class="line"><span class="meta">... </span>                <span class="comment"># NEW: prevent crashed task from ending the loop.</span></span><br><span class="line"><span class="meta">... </span>                print(error)</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                <span class="comment"># NEW: schedule the child task.</span></span><br><span class="line"><span class="meta">... </span>                <span class="keyword">if</span> iscoroutine(data):</span><br><span class="line"><span class="meta">... </span>                    tasks.append((data, <span class="keyword">None</span>))</span><br><span class="line"><span class="meta">... </span>                <span class="comment"># NEW: reschedule the parent task.</span></span><br><span class="line"><span class="meta">... </span>                tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>run_until_complete(main())</span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><p><strong>哇</strong>! 这比我们之前版本的<code>run_until_complete()</code>要复杂得多。这些都是怎么来的？</p><p>好吧…现在我们可以运行多个任务了，我们需要担心的事情包括:</p><ol><li>等待所有的子任务完成（递归），尽管任何任务都有错误</li><li>在任务之间交替进行，让所有的任务同时完成</li></ol><p>注意，我们现在有一个嵌套循环。</p><ul><li>外循环负责检查我们是否完成了任务</li><li>内循环负责处理一次调度器的 “tick”（倒计时完成）。</li></ul><p>还有其他方法可以做到这一点，而且我们为什么要这样做可能不是很明显，但这很重要，因为事件循环中还缺少两个关键部分：定时器和I/O。当我们以后增加对这些的支持时，我们也需要以一种 &quot;公平 &quot;的方式来安排内部检查。外循环为我们提供了一个检查计时器和轮询I/O操作状态的方便位置。</p><ol><li>检查定时器，恢复已经过了延迟期的睡眠任务。</li><li>检查I/O操作，安排那些待定I/O操作已经完成的任务。</li><li>执行一次调度器的&quot;tick&quot;，以恢复我们刚刚调度的所有任务。</li></ol><p>简而言之，这就是基于coroutine的调度器循环的要点。</p><p><strong>然而</strong>，在我们进入更复杂的定时器和I/O之前…还记得我在前面提到这个例子是故意不完整的吗？我们知道如何生成新的子任务，但我们还不知道如何等待它们完成。这是一个很好的机会来学习如何扩展，由我们的awaitable对象发送事件循环 &quot;请求&quot;的词汇表。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> collections <span class="keyword">import</span> defaultdict</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">nice</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span></span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">spawn</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: recover the child task handle to pass it back to the parent.</span></span><br><span class="line"><span class="meta">... </span>    child = <span class="keyword">yield</span> (<span class="string">'spawn'</span>, task)</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">return</span> child</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: awaitable object that sends a request to be notified when a</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment">#      concurrent task completes.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">join</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span> (<span class="string">'join'</span>, task)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">await</span> nice()</span><br><span class="line"><span class="meta">... </span>    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: recover the child task handle.</span></span><br><span class="line"><span class="meta">... </span>    child = <span class="keyword">await</span> spawn(hello(<span class="string">'world'</span>))</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: wait for the child task to complete.</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">await</span> join(child)</span><br><span class="line"><span class="meta">... </span>    print(<span class="string">'(after join)'</span>)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    tasks = [(task, <span class="keyword">None</span>)]</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: keep track of tasks to resume when a task completes.</span></span><br><span class="line"><span class="meta">... </span>    watch = defaultdict(list)</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">while</span> tasks:</span><br><span class="line"><span class="meta">... </span>        queue, tasks = tasks, []</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">for</span> task, data <span class="keyword">in</span> queue:</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>                data = task.send(data)</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">except</span> StopIteration:</span><br><span class="line"><span class="meta">... </span>                <span class="comment"># NEW: wait up tasks waiting on this one.</span></span><br><span class="line"><span class="meta">... </span>                tasks.extend((t, <span class="keyword">None</span>) <span class="keyword">for</span> t <span class="keyword">in</span> watch.pop(task, []))</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                <span class="comment"># NEW: dispatch request sent by awaitable object since</span></span><br><span class="line"><span class="meta">... </span>                <span class="comment">#      we now have 3 different types of requests.</span></span><br><span class="line"><span class="meta">... </span>                <span class="keyword">if</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'spawn'</span>:</span><br><span class="line"><span class="meta">... </span>                    tasks.append((data[<span class="number">1</span>], <span class="keyword">None</span>))</span><br><span class="line"><span class="meta">... </span>                    tasks.append((task, data[<span class="number">1</span>]))</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">elif</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'join'</span>:</span><br><span class="line"><span class="meta">... </span>                    watch[data[<span class="number">1</span>]].append(task)</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                    tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>run_until_complete(main())</span><br><span class="line">Hello, world!</span><br><span class="line">(after join)</span><br></pre></td></tr></table></figure><p>由于实际原因，我们可能希望有某种Task包装器用于coroutine对象。这对于暴露cancle的API和处理一些竞争情况是很方便的，比如子任务在父任务试图<code>join()</code>它之前就结束了（你能发现这个bug吗？）</p><p>将子任务的返回值作为<code>await join()</code> 的结果传回，并传播使子任务崩溃的异常，这些都是留给读者的练习。</p><h2 id="sleeping-timers"><a class="markdownIt-Anchor" href="#sleeping-timers"></a> Sleeping &amp; timers</h2><p>现在我们已经控制了任务调度，我们可以开始处理一些更高级的东西，比如定时器和I/O。I/O是最终的目标，但它牵涉到很多新的东西，所以我们先看看定时器。</p><p>如果你需要休眠Sleep，你不能只调用<code>time.sleep()</code>，因为你会阻塞所有的任务，而不仅仅是你想暂停的那个。</p><p>你现在可能已经发现了这个模式。我们将添加两样东西。</p><ol><li>一种新的请求类型</li><li>一段基于<code>task.send()</code>返回值的调度代码。</li></ol><p>我们还将添加一些bookKeeping以跟踪那些被暂停的任务。请记住，tasks是一个预定在下一个tick中运行的coroutine列表，但沉睡的任务在准备再次运行之前可能会跳过一个或多个tick。</p><p>请记住，休眠的任务不太可能按照先进先出的顺序重新安排，所以我们需要一些更进化的东西。维护计时器，最实用的方法（直到你允许取消它们）是使用一个优先级队列，感谢标准库的 <a href="https://docs.python.org/3.5/library/heapq.html" target="_blank" rel="noopener">heapq</a> 模块使之超级简单。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> heapq <span class="keyword">import</span> heappop, heappush</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> time <span class="keyword">import</span> sleep <span class="keyword">as</span> _sleep</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> timeit <span class="keyword">import</span> default_timer</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: we need to keep track of elapsed time.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>clock = default_timer</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: request that the event loop reschedule us "later".</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">sleep</span><span class="params">(seconds)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span> (<span class="string">'sleep'</span>, seconds)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: verify elapsed time matches our request.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    ref = clock()</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">await</span> sleep(<span class="number">3.0</span>)</span><br><span class="line"><span class="meta">... </span>    now = clock()</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">assert</span> (now - ref) &gt;= <span class="number">3.0</span></span><br><span class="line"><span class="meta">... </span>    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    tasks = [(task, <span class="keyword">None</span>)]</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: keep track of tasks that are sleeping.</span></span><br><span class="line"><span class="meta">... </span>    timers = []</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: watch out, all tasks might be suspended at the same time.</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">while</span> tasks <span class="keyword">or</span> timers:</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>        <span class="comment"># NEW: if we have nothing to do for now, don't spin.</span></span><br><span class="line"><span class="meta">... </span>        <span class="keyword">if</span> <span class="keyword">not</span> tasks:</span><br><span class="line"><span class="meta">... </span>            _sleep(max(<span class="number">0.0</span>, timers[<span class="number">0</span>][<span class="number">0</span>] - clock()))</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>        <span class="comment"># NEW: schedule tasks when their timer has elapsed.</span></span><br><span class="line"><span class="meta">... </span>        <span class="keyword">while</span> timers <span class="keyword">and</span> timers[<span class="number">0</span>][<span class="number">0</span>] &lt; clock():</span><br><span class="line"><span class="meta">... </span>            _, task = heappop(timers)</span><br><span class="line"><span class="meta">... </span>            tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>        queue, tasks = tasks, []</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">for</span> task, data <span class="keyword">in</span> queue:</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>                data = task.send(data)</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">except</span> StopIteration:</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">pass</span></span><br><span class="line"><span class="meta">... </span>            <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                <span class="comment"># NEW: set a timer and don't reschedule right away.</span></span><br><span class="line"><span class="meta">... </span>                <span class="keyword">if</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'sleep'</span>:</span><br><span class="line"><span class="meta">... </span>                    heappush(timers, (clock() + data[<span class="number">1</span>], task))</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                    tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>run_until_complete(hello(<span class="string">'world'</span>))</span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><p>哇，我们真的掌握了这个窍门! 也许这个异步的东西毕竟没有那么难？</p><p>让我们看看我们能为I/O做什么!</p><h2 id="处理io"><a class="markdownIt-Anchor" href="#处理io"></a> 处理I/O</h2><p>现在我们已经经历了所有其他的事情，是时候进行终极对决了：I/O。</p><p>可伸缩的 I/O 通常使用native C的APIs 来实现 I/O 的复用。通常，这是 I/O 库中最困难的部分，但值得庆幸的是，Python 的 <a href="https://docs.python.org/3/library/selectors.html" target="_blank" rel="noopener">selectors</a> 模块使其非常容易实现。</p><p>至于到目前为止我们添加的所有其他操作，我们将在事件循环中添加一些新的I/O请求和相应的请求处理程序。另外，像计时器一样，我们需要在每个调度器的开始阶段做一些内部检查。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> selectors <span class="keyword">import</span> (</span><br><span class="line"><span class="meta">... </span>    DefaultSelector,</span><br><span class="line"><span class="meta">... </span>    EVENT_READ,</span><br><span class="line"><span class="meta">... </span>    EVENT_WRITE,</span><br><span class="line"><span class="meta">... </span>)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> socket <span class="keyword">import</span> socketpair <span class="keyword">as</span> _socketpair</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: request that the event loop tell us when we can read.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">recv</span><span class="params">(stream, size)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span> (EVENT_READ, stream)</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">return</span> stream.recv(size)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: request that the event loop tell us when we can write.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">send</span><span class="params">(stream, data)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">while</span> data:</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">yield</span> (EVENT_WRITE, stream)</span><br><span class="line"><span class="meta">... </span>        size = stream.send(data)</span><br><span class="line"><span class="meta">... </span>        data = data[size:]</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: connect sockets, make sure they never, ever block the loop.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">socketpair</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>    lhs, rhs = _socketpair()</span><br><span class="line"><span class="meta">... </span>    lhs.setblocking(<span class="keyword">False</span>)</span><br><span class="line"><span class="meta">... </span>    rhs.setblocking(<span class="keyword">False</span>)</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">return</span> lhs, rhs</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: send a message through the socket pair.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    lhs, rhs = <span class="keyword">await</span> socketpair()</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">await</span> send(lhs, <span class="string">'Hello, world!'</span>.encode(<span class="string">'utf-8'</span>))</span><br><span class="line"><span class="meta">... </span>    data = <span class="keyword">await</span> recv(rhs, <span class="number">1024</span>)</span><br><span class="line"><span class="meta">... </span>    print(data.decode(<span class="string">'utf-8'</span>))</span><br><span class="line"><span class="meta">... </span>    lhs.close()</span><br><span class="line"><span class="meta">... </span>    rhs.close()</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    tasks = [(task, <span class="keyword">None</span>)]</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: prepare for I/O multiplexing.</span></span><br><span class="line"><span class="meta">... </span>    selector = DefaultSelector()</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: watch out, all tasks might be suspended at the same time.</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">while</span> tasks <span class="keyword">or</span> selector.get_map():</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>        <span class="comment"># NEW: poll I/O operation status and resume tasks when ready.</span></span><br><span class="line"><span class="meta">... </span>        timeout = <span class="number">0.0</span> <span class="keyword">if</span> tasks <span class="keyword">else</span> <span class="keyword">None</span></span><br><span class="line"><span class="meta">... </span>        <span class="keyword">for</span> key, events <span class="keyword">in</span> selector.select(timeout):</span><br><span class="line"><span class="meta">... </span>            tasks.append((key.data, <span class="keyword">None</span>))</span><br><span class="line"><span class="meta">... </span>            selector.unregister(key.fileobj)</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>        queue, tasks = tasks, []</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">for</span> task, data <span class="keyword">in</span> queue:</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>                data = task.send(data)</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">except</span> StopIteration:</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">pass</span></span><br><span class="line"><span class="meta">... </span>            <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                <span class="comment"># NEW: register for I/O and suspend the task.</span></span><br><span class="line"><span class="meta">... </span>                <span class="keyword">if</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == EVENT_READ:</span><br><span class="line"><span class="meta">... </span>                    selector.register(data[<span class="number">1</span>], EVENT_READ, task)</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">elif</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == EVENT_WRITE:</span><br><span class="line"><span class="meta">... </span>                    selector.register(data[<span class="number">1</span>], EVENT_WRITE, task)</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                    tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>run_until_complete(hello(<span class="string">'world'</span>))</span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><h2 id="取消任务"><a class="markdownIt-Anchor" href="#取消任务"></a> 取消任务</h2><p>最后，但也是最重要的一块拼图是——取消任务，这就是我们利用coroutine对象的<code>.throw()</code>方法的地方。</p><p>由于取消任务有一个竞争条件（它可能与尝试取消任务同时完成），所以需要跟踪所有正在运行的任务，以了解它们的 “状态”。</p><p>否则，它就是任务生成和加入的简单扩展。</p><p><strong>注意</strong>：这个实现是故意不完整的。它没有正确处理取消一个已经安排在下一个tick中运行的任务的可能性。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> inspect <span class="keyword">import</span> iscoroutine</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">spawn</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    task = <span class="keyword">yield</span> (<span class="string">'spawn'</span>, task)</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">return</span> task</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">join</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span> (<span class="string">'join'</span>, task)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: exception to be raised inside tasks when they are cancelled.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="class"><span class="keyword">class</span> <span class="title">CancelledError</span><span class="params">(Exception)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">pass</span></span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: request that CancelledError be raised inside the task.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">cancel</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    cancelled = <span class="keyword">yield</span> (<span class="string">'cancel'</span>, task, CancelledError())</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">assert</span> cancelled <span class="keyword">is</span> <span class="keyword">True</span></span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: pause the task without plans to reschedule it (this is simply to</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment">#      guarantee execution order in this demo).</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>@coroutine</span><br><span class="line"><span class="meta">... </span><span class="function"><span class="keyword">def</span> <span class="title">suspend</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">yield</span> (<span class="string">'suspend'</span>,)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line"><span class="meta">... </span>    <span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">await</span> suspend()</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">except</span> CancelledError:</span><br><span class="line"><span class="meta">... </span>        print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">raise</span></span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="comment"># NEW: spawn a task and then cancel it.</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"><span class="meta">... </span>    child = <span class="keyword">await</span> spawn(hello(<span class="string">'world'</span>))</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">await</span> cancel(child)</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">await</span> join(child)</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line"><span class="meta">... </span>    tasks = [(task, <span class="keyword">None</span>)]</span><br><span class="line"><span class="meta">... </span>    watch = defaultdict(list)</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>    <span class="comment"># NEW: keep track of all tasks in the tree.</span></span><br><span class="line"><span class="meta">... </span>    tree = &#123;task&#125;</span><br><span class="line">...</span><br><span class="line"><span class="meta">... </span>    <span class="keyword">while</span> tasks:</span><br><span class="line"><span class="meta">... </span>        queue, tasks = tasks, []</span><br><span class="line"><span class="meta">... </span>        <span class="keyword">for</span> task, data <span class="keyword">in</span> queue:</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">try</span>:</span><br><span class="line"><span class="meta">... </span>                <span class="comment"># NEW: we may need to pass data or inject an exception.</span></span><br><span class="line"><span class="meta">... </span>                <span class="keyword">if</span> isinstance(data, Exception):</span><br><span class="line"><span class="meta">... </span>                    data = task.throw(data)</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                    data = task.send(data)</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">except</span> (StopIteration, CancelledError):</span><br><span class="line"><span class="meta">... </span>                tasks.extend((t, <span class="keyword">None</span>) <span class="keyword">for</span> t <span class="keyword">in</span> watch.pop(task, []))</span><br><span class="line"><span class="meta">... </span>                <span class="comment"># NEW: update bookkeeping.</span></span><br><span class="line"><span class="meta">... </span>                tree.discard(task)</span><br><span class="line"><span class="meta">... </span>            <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">if</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'spawn'</span>:</span><br><span class="line"><span class="meta">... </span>                    tasks.append((data[<span class="number">1</span>], <span class="keyword">None</span>))</span><br><span class="line"><span class="meta">... </span>                    tasks.append((task, data[<span class="number">1</span>]))</span><br><span class="line"><span class="meta">... </span>                    <span class="comment"># NEW: update bookkeeping.</span></span><br><span class="line"><span class="meta">... </span>                    tree.add(data[<span class="number">1</span>])</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">elif</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'join'</span>:</span><br><span class="line"><span class="meta">... </span>                    watch[data[<span class="number">1</span>]].append(task)</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">elif</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'cancel'</span>:</span><br><span class="line"><span class="meta">... </span>                    <span class="comment"># NEW: schedule to raise the exception in the task.</span></span><br><span class="line"><span class="meta">... </span>                    <span class="keyword">if</span> data[<span class="number">1</span>] <span class="keyword">in</span> tree:</span><br><span class="line"><span class="meta">... </span>                        tasks.append((data[<span class="number">1</span>], data[<span class="number">2</span>]))</span><br><span class="line"><span class="meta">... </span>                        tasks.append((task, <span class="keyword">True</span>))</span><br><span class="line"><span class="meta">... </span>                    <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                        tasks.append((task, <span class="keyword">False</span>))</span><br><span class="line"><span class="meta">... </span>                <span class="keyword">elif</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'suspend'</span>:</span><br><span class="line"><span class="meta">... </span>                    <span class="keyword">pass</span></span><br><span class="line"><span class="meta">... </span>                <span class="keyword">else</span>:</span><br><span class="line"><span class="meta">... </span>                    tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line">&gt;&gt;&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>run_until_complete(main())</span><br><span class="line">Hello, world!</span><br></pre></td></tr></table></figure><h2 id="许可证"><a class="markdownIt-Anchor" href="#许可证"></a> 许可证</h2><p>本文件的版权归Andre Caron <a href="mailto:andre.l.caron@gmail.com" target="_blank" rel="noopener">andre.l.caron@gmail.com</a>所有，并在知识共享 <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode" target="_blank" rel="noopener">CC-BY-SA</a>  许可下向您提供。</p><h1 id="个人收获"><a class="markdownIt-Anchor" href="#个人收获"></a> 个人收获</h1><p>一个异步框架通常主要包括事件循环、<a href="https://www.zhihu.com/search?q=%E4%BA%8B%E4%BB%B6%E9%98%9F%E5%88%97&amp;search_source=Entity&amp;hybrid_search_source=Entity&amp;hybrid_search_extra=%7B%22sourceType%22%3A%22answer%22%2C%22sourceId%22%3A555273313%7D" target="_blank" rel="noopener">事件队列</a>、polling、timer队列，所有的异步框架皆不例外，asyncio也是如此。本文的toy event loop也是如此，在自己实现的时候就是关注这几个部分。</p><p>文中的实验都表明了，协程的<code>send()</code>、<code>throw()</code>、<code>await</code>与<code>send()</code>、<code>throw()</code>、<code>yield</code>使用都类似。</p><h1 id="个人实验"><a class="markdownIt-Anchor" href="#个人实验"></a> 个人实验</h1><p>针对<a href="#%E4%B8%8E%E4%BA%8B%E4%BB%B6%E5%BE%AA%E7%8E%AF%E7%9A%84%E5%AF%B9%E8%AF%9D">与事件循环的对话</a>的实验代码，我又增加了一些内容看效果：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: this is an awaitable object!</span></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">nice</span><span class="params">()</span>:</span></span><br><span class="line">    print(<span class="string">'before yield'</span>)</span><br><span class="line">    <span class="keyword">yield</span></span><br><span class="line">    print(<span class="string">"after yield"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line">    <span class="comment"># NEW: this makes ``.send()`` return!</span></span><br><span class="line">    <span class="keyword">await</span> nice()</span><br><span class="line">    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">task = hello(<span class="string">'world'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: call send twice!</span></span><br><span class="line">task.send(<span class="keyword">None</span>)</span><br><span class="line">print(<span class="string">"after first sned"</span>)</span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line">    task.send(<span class="keyword">None</span>)</span><br><span class="line"><span class="keyword">except</span> StopIteration:</span><br><span class="line">    <span class="keyword">pass</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">before yield</span></span><br><span class="line"><span class="string">after first sned</span></span><br><span class="line"><span class="string">after yield</span></span><br><span class="line"><span class="string">Hello, world!</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure><p>通过输出我们可以发现， <code>task.send()</code>会在<code>nice()</code>的<code>yield</code>处停下来。<code>@coroutine</code>的注解，让<code>nice()</code>函数成为了awaitalbe对象，从而可以<code>await nice()</code>调用。</p><p>针对<a href="#%E7%94%9F%E6%88%90%E5%AD%90%E4%BB%BB%E5%8A%A1">生成子任务</a>，发现main协程结束的比hello协程早。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> inspect <span class="keyword">import</span> iscoroutine</span><br><span class="line"><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">nice</span><span class="params">()</span>:</span></span><br><span class="line">    print(<span class="string">"before nice"</span>)</span><br><span class="line">    <span class="keyword">yield</span></span><br><span class="line">    print(<span class="string">"after nice"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: awaitable object that sends a request to launch a child task!</span></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">spawn</span><span class="params">(task)</span>:</span></span><br><span class="line">    print(<span class="string">"before spawn"</span>)</span><br><span class="line">    <span class="keyword">yield</span> task</span><br><span class="line">    print(<span class="string">"after spawn"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line">    print(<span class="string">"hello: before await"</span>)</span><br><span class="line">    <span class="keyword">await</span> nice()</span><br><span class="line">    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: parent task!</span></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line">    print(<span class="string">"main: before"</span>)</span><br><span class="line">    <span class="comment"># NEW: create a child task!</span></span><br><span class="line">    <span class="keyword">await</span> spawn(hello(<span class="string">'world'</span>))</span><br><span class="line">    print(<span class="string">"main: after"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line">    <span class="comment"># NEW: schedule the "root" task.</span></span><br><span class="line">    tasks = [(task, <span class="keyword">None</span>)]</span><br><span class="line">    <span class="keyword">while</span> tasks:</span><br><span class="line">        <span class="comment"># NEW: round-robin between a set tasks (we may now have more than one and we'd like to be as "fair" as possible).</span></span><br><span class="line">        <span class="comment"># 新：一组任务之间的循环（我们现在可能有多个任务，我们希望尽可能“公平”）。</span></span><br><span class="line">        queue, tasks = tasks, []</span><br><span class="line">        <span class="keyword">for</span> task, data <span class="keyword">in</span> queue:</span><br><span class="line">            <span class="comment"># NEW: resume the task *once*.</span></span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                data = task.send(data)</span><br><span class="line">            <span class="keyword">except</span> StopIteration:</span><br><span class="line">                <span class="keyword">pass</span></span><br><span class="line">            <span class="keyword">except</span> Exception <span class="keyword">as</span> error:</span><br><span class="line">                <span class="comment"># NEW: prevent crashed task from ending the loop.</span></span><br><span class="line">                print(error)</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                <span class="comment"># NEW: schedule the child task.</span></span><br><span class="line">                <span class="keyword">if</span> iscoroutine(data):</span><br><span class="line">                    tasks.append((data, <span class="keyword">None</span>))</span><br><span class="line">                <span class="comment"># NEW: reschedule the parent task.</span></span><br><span class="line">                tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">run_until_complete(main())</span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">main: before</span></span><br><span class="line"><span class="string">before spawn</span></span><br><span class="line"><span class="string">hello: before await</span></span><br><span class="line"><span class="string">before nice</span></span><br><span class="line"><span class="string">after spawn</span></span><br><span class="line"><span class="string">main: after</span></span><br><span class="line"><span class="string">after nice</span></span><br><span class="line"><span class="string">Hello, world!</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure><p>等待子任务join</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> collections <span class="keyword">import</span> defaultdict</span><br><span class="line"><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">nice</span><span class="params">()</span>:</span></span><br><span class="line">    print(<span class="string">"nice: before yield"</span>)</span><br><span class="line">    <span class="keyword">yield</span></span><br><span class="line">    print(<span class="string">"nice: after yield"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">spawn</span><span class="params">(task)</span>:</span></span><br><span class="line">    <span class="comment"># NEW: recover the child task handle to pass it back to the parent.</span></span><br><span class="line">    print(<span class="string">"spawn: before yield"</span>)</span><br><span class="line">    child = <span class="keyword">yield</span> (<span class="string">'spawn'</span>, task)</span><br><span class="line">    print(<span class="string">"spawn: after yield"</span>)</span><br><span class="line">    <span class="keyword">return</span> child</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: awaitable object that sends a request to be notified when a</span></span><br><span class="line"><span class="comment">#      concurrent task completes.</span></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">join</span><span class="params">(task)</span>:</span></span><br><span class="line">    print(<span class="string">"join: before yield"</span>)</span><br><span class="line">    <span class="keyword">yield</span> (<span class="string">'join'</span>, task)</span><br><span class="line">    print(<span class="string">"join: after yield"</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line">    print(<span class="string">"hello: before nice()"</span>)</span><br><span class="line">    <span class="keyword">await</span> nice()</span><br><span class="line">    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="comment"># NEW: recover the child task handle.</span></span><br><span class="line">    child = <span class="keyword">await</span> spawn(hello(<span class="string">'world'</span>))</span><br><span class="line">    print(<span class="string">"main: after spawn and before join()"</span>)</span><br><span class="line">    <span class="comment"># NEW: wait for the child task to complete.</span></span><br><span class="line">    <span class="comment"># 等待子协程完成</span></span><br><span class="line">    <span class="keyword">await</span> join(child)</span><br><span class="line">    print(<span class="string">'(after join)'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line">    tasks = [(task, <span class="keyword">None</span>)]</span><br><span class="line">    <span class="comment"># NEW: keep track of tasks to resume when a task completes.</span></span><br><span class="line">    watch = defaultdict(list)</span><br><span class="line">    <span class="keyword">while</span> tasks:</span><br><span class="line">        queue, tasks = tasks, []</span><br><span class="line">        <span class="keyword">for</span> task, data <span class="keyword">in</span> queue:</span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                data = task.send(data)</span><br><span class="line">            <span class="keyword">except</span> StopIteration:</span><br><span class="line">                <span class="comment"># NEW: wait up tasks waiting on this one.</span></span><br><span class="line">                <span class="comment"># 子协程退出时, 唤醒被join阻塞的父协程</span></span><br><span class="line">                <span class="comment"># 对watch中的task进行pop, 如果没有则返回[]</span></span><br><span class="line">                tasks.extend((t, <span class="keyword">None</span>) <span class="keyword">for</span> t <span class="keyword">in</span> watch.pop(task, []))</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                <span class="comment"># NEW: dispatch request sent by awaitable object since we now have 3 different types of requests.</span></span><br><span class="line">                <span class="comment"># NEW: 由等待对象发送的调度请求，因为我们现在有 3 种不同类型的请求</span></span><br><span class="line">                <span class="comment"># 1. 如果是会产生新协程的操作</span></span><br><span class="line">                <span class="keyword">if</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'spawn'</span>:</span><br><span class="line">                    <span class="comment"># 将新协程加入</span></span><br><span class="line">                    tasks.append((data[<span class="number">1</span>], <span class="keyword">None</span>))</span><br><span class="line">                    <span class="comment"># 将父协程和新协程写入, 在task.send(data)中可以使得父协程中赋值child=yield epx为子协程值</span></span><br><span class="line">                    tasks.append((task, data[<span class="number">1</span>]))</span><br><span class="line">                <span class="keyword">elif</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'join'</span>:</span><br><span class="line">                    <span class="comment"># 如果父协程执行的快, 那么把子协程与父协程的绑定关系记录下来</span></span><br><span class="line">                    watch[data[<span class="number">1</span>]].append(task)</span><br><span class="line">                <span class="keyword">else</span>:</span><br><span class="line">                    <span class="comment"># 正常协程执行结果</span></span><br><span class="line">                    tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">run_until_complete(main())</span><br><span class="line"></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">spawn: before yield</span></span><br><span class="line"><span class="string">hello: before nice()</span></span><br><span class="line"><span class="string">nice: before yield</span></span><br><span class="line"><span class="string">spawn: after yield</span></span><br><span class="line"><span class="string">main: after spawn and before join()</span></span><br><span class="line"><span class="string">--- 此时main执行完成, join等待nice()被执行</span></span><br><span class="line"><span class="string">join: before yield</span></span><br><span class="line"><span class="string">nice: after yield</span></span><br><span class="line"><span class="string">--- nice执行完成</span></span><br><span class="line"><span class="string">Hello, world!</span></span><br><span class="line"><span class="string">--- hello执行完成</span></span><br><span class="line"><span class="string">join: after yield</span></span><br><span class="line"><span class="string">--- join结束</span></span><br><span class="line"><span class="string">(after join)</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure><p>多了两样东西：</p><ul><li>新的请求类型:<ul><li><code>spawn</code></li><li><code>join</code></li><li><code>push</code></li></ul></li><li>基于<code>task.send()</code>返回值的调度代码。</li></ul><p>针对[Sleeping &amp; timers](#Sleeping &amp; timers)</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> heapq <span class="keyword">import</span> heappop, heappush</span><br><span class="line"><span class="keyword">from</span> time <span class="keyword">import</span> sleep <span class="keyword">as</span> _sleep</span><br><span class="line"><span class="keyword">from</span> timeit <span class="keyword">import</span> default_timer</span><br><span class="line"><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: we need to keep track of elapsed time.</span></span><br><span class="line">clock = default_timer</span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: request that the event loop reschedule us "later".</span></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">sleep</span><span class="params">(seconds)</span>:</span></span><br><span class="line">    <span class="keyword">yield</span> (<span class="string">'sleep'</span>, seconds)</span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: verify elapsed time matches our request.</span></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line">    ref = clock()</span><br><span class="line">    <span class="keyword">await</span> sleep(<span class="number">3.0</span>)</span><br><span class="line">    now = clock()</span><br><span class="line">    <span class="keyword">assert</span> (now - ref) &gt;= <span class="number">3.0</span></span><br><span class="line">    print(<span class="string">'Hello, %s!'</span> % (name,))</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line">    tasks = [(task, <span class="keyword">None</span>)]</span><br><span class="line"></span><br><span class="line">    <span class="comment"># NEW: keep track of tasks that are sleeping.</span></span><br><span class="line">    timers = []</span><br><span class="line"></span><br><span class="line">    <span class="comment"># NEW: watch out, all tasks might be suspended at the same time.</span></span><br><span class="line">    <span class="keyword">while</span> tasks <span class="keyword">or</span> timers:</span><br><span class="line"></span><br><span class="line">        <span class="comment"># NEW: if we have nothing to do for now, don't spin.</span></span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> tasks:</span><br><span class="line">            _sleep(max(<span class="number">0.0</span>, timers[<span class="number">0</span>][<span class="number">0</span>] - clock()))</span><br><span class="line"></span><br><span class="line">        <span class="comment"># NEW: schedule tasks when their timer has elapsed.</span></span><br><span class="line">        <span class="comment"># 如果times队列中有任务, 且最近的时间已经到了, 则取出任务</span></span><br><span class="line">        <span class="keyword">while</span> timers <span class="keyword">and</span> timers[<span class="number">0</span>][<span class="number">0</span>] &lt; clock():</span><br><span class="line">            _, task = heappop(timers)</span><br><span class="line">            tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line"></span><br><span class="line">        queue, tasks = tasks, []</span><br><span class="line">        <span class="keyword">for</span> task, data <span class="keyword">in</span> queue:</span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                data = task.send(data)</span><br><span class="line">            <span class="keyword">except</span> StopIteration:</span><br><span class="line">                <span class="keyword">pass</span></span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                <span class="comment"># NEW: set a timer and don't reschedule right away.</span></span><br><span class="line">                <span class="keyword">if</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == <span class="string">'sleep'</span>:</span><br><span class="line">                    <span class="comment"># 往堆里面添加任务, 默认以时间最近的在上(最小堆)</span></span><br><span class="line">                    heappush(timers, (clock() + data[<span class="number">1</span>], task))</span><br><span class="line">                <span class="keyword">else</span>:</span><br><span class="line">                    <span class="comment"># 递归添加下一次要做的协程任务</span></span><br><span class="line">                    tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line"></span><br><span class="line">run_until_complete(hello(<span class="string">'world'</span>))</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> selectors <span class="keyword">import</span> (</span><br><span class="line">    DefaultSelector,</span><br><span class="line">    EVENT_READ,</span><br><span class="line">    EVENT_WRITE,</span><br><span class="line">)</span><br><span class="line"><span class="keyword">from</span> socket <span class="keyword">import</span> socketpair <span class="keyword">as</span> _socketpair</span><br><span class="line"><span class="keyword">from</span> types <span class="keyword">import</span> coroutine</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: request that the event loop tell us when we can read.</span></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">recv</span><span class="params">(stream, size)</span>:</span></span><br><span class="line">    <span class="keyword">yield</span> (EVENT_READ, stream)</span><br><span class="line">    <span class="keyword">return</span> stream.recv(size)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: request that the event loop tell us when we can write.</span></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">send</span><span class="params">(stream, data)</span>:</span></span><br><span class="line">    <span class="keyword">while</span> data:</span><br><span class="line">        <span class="keyword">yield</span> (EVENT_WRITE, stream)</span><br><span class="line">        size = stream.send(data)</span><br><span class="line">        data = data[size:]</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: connect sockets, make sure they never, ever block the loop.</span></span><br><span class="line"><span class="meta">@coroutine</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">socketpair</span><span class="params">()</span>:</span></span><br><span class="line">    lhs, rhs = _socketpair()</span><br><span class="line">    lhs.setblocking(<span class="keyword">False</span>)</span><br><span class="line">    rhs.setblocking(<span class="keyword">False</span>)</span><br><span class="line">    <span class="keyword">yield</span></span><br><span class="line">    <span class="keyword">return</span> lhs, rhs</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># NEW: send a message through the socket pair.</span></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">(name)</span>:</span></span><br><span class="line">    lhs, rhs = <span class="keyword">await</span> socketpair()</span><br><span class="line">    <span class="keyword">await</span> send(lhs, <span class="string">'Hello, world!'</span>.encode(<span class="string">'utf-8'</span>))</span><br><span class="line">    data = <span class="keyword">await</span> recv(rhs, <span class="number">1024</span>)</span><br><span class="line">    print(data.decode(<span class="string">'utf-8'</span>))</span><br><span class="line">    lhs.close()</span><br><span class="line">    rhs.close()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run_until_complete</span><span class="params">(task)</span>:</span></span><br><span class="line">    tasks = [(task, <span class="keyword">None</span>)]</span><br><span class="line"></span><br><span class="line">    <span class="comment"># NEW: prepare for I/O multiplexing.</span></span><br><span class="line">    selector = DefaultSelector()</span><br><span class="line"></span><br><span class="line">    <span class="comment"># NEW: watch out, all tasks might be suspended at the same time.</span></span><br><span class="line">    <span class="keyword">while</span> tasks <span class="keyword">or</span> selector.get_map():</span><br><span class="line"></span><br><span class="line">        <span class="comment"># NEW: poll I/O operation status and resume tasks when ready.</span></span><br><span class="line">        timeout = <span class="number">0.0</span> <span class="keyword">if</span> tasks <span class="keyword">else</span> <span class="keyword">None</span></span><br><span class="line">        <span class="keyword">for</span> key, events <span class="keyword">in</span> selector.select(timeout):</span><br><span class="line">            tasks.append((key.data, <span class="keyword">None</span>))</span><br><span class="line">            selector.unregister(key.fileobj)</span><br><span class="line"></span><br><span class="line">        queue, tasks = tasks, []</span><br><span class="line">        <span class="keyword">for</span> task, data <span class="keyword">in</span> queue:</span><br><span class="line">            <span class="keyword">try</span>:</span><br><span class="line">                data = task.send(data)</span><br><span class="line">            <span class="keyword">except</span> StopIteration:</span><br><span class="line">                <span class="keyword">pass</span></span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                <span class="comment"># NEW: register for I/O and suspend the task.</span></span><br><span class="line">                <span class="keyword">if</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == EVENT_READ:</span><br><span class="line">                    selector.register(data[<span class="number">1</span>], EVENT_READ, task)</span><br><span class="line">                <span class="keyword">elif</span> data <span class="keyword">and</span> data[<span class="number">0</span>] == EVENT_WRITE:</span><br><span class="line">                    selector.register(data[<span class="number">1</span>], EVENT_WRITE, task)</span><br><span class="line">                <span class="keyword">else</span>:</span><br><span class="line">                    tasks.append((task, <span class="keyword">None</span>))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">run_until_complete(hello(<span class="string">'world'</span>))</span><br></pre></td></tr></table></figure><p>Q: socketpair是什么？</p><ul><li><code>socket.socketpair()</code>函数仅返回两个已经连接的套接字对象，参数和socket.socket()里的参数一样的用法。</li><li><code>socket.socketpair()</code>可以理解为 创建了两个socket, 比喻为一个server的 socket，一个client的socket，这两个socket是已经connected连接状态</li><li>socket.socketpair()`是全双工模式，也就是每个socket都能收发，比喻为\server.send—&gt;client.recv,和 client.send—&gt;server.recv</li><li>socket.socketpair()`默认是创建unix套接字</li></ul><p>总结：</p><ul><li>函数加上<code>async def</code>变成了<code>coroutine object</code></li><li>函数加上<code>@types.coroutine</code>变成了<code>awaitable</code>的function</li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h1 id=&quot;事件循环的一个小故事&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#事件循环的一个小故事&quot;&gt;&lt;/a&gt; 事件循环的一个小故事&lt;/h1&gt;
&lt;blockquote&gt;
&lt;p&gt;Translated from: &lt;a href=&quot;https://g
      
    
    </summary>
    
    
      <category term="Python" scheme="https://nymrli.top/tags/Python/"/>
    
      <category term="翻译" scheme="https://nymrli.top/tags/%E7%BF%BB%E8%AF%91/"/>
    
      <category term="并发" scheme="https://nymrli.top/tags/%E5%B9%B6%E5%8F%91/"/>
    
  </entry>
  
  <entry>
    <title>selenium不行的工作,pyppeteer上</title>
    <link href="https://nymrli.top/2022/05/05/selenium%E4%B8%8D%E8%A1%8C%E7%9A%84%E5%B7%A5%E4%BD%9C-pyppeteer%E4%B8%8A/"/>
    <id>https://nymrli.top/2022/05/05/selenium不行的工作-pyppeteer上/</id>
    <published>2022-05-05T12:31:39.000Z</published>
    <updated>2022-05-07T08:33:12.577Z</updated>
    
    <content type="html"><![CDATA[<h2 id="验证码问题"><a class="markdownIt-Anchor" href="#验证码问题"></a> 验证码问题</h2><p>该网站对频率检测十分敏感，很快就会出现&quot;按住&quot;验证码，模拟按住倒不难，但是按完以后有时并不能解决验证…</p><p>但是意外发现，在“无痕模式”下进行重新登录后，就可以解决验证…但是通过requests进行模拟登录后cookies还是无效。</p><p><strong>解决方案</strong>：</p><blockquote><p>既然模拟不行，那么就手动，因此想用selenium来模拟点击登陆。但是因为俄罗斯的问题，无法登陆。</p></blockquote><p><img src="/2022/05/05/selenium不行的工作-pyppeteer上/selenium%E4%B8%8D%E8%A1%8C%E7%9A%84%E5%B7%A5%E4%BD%9C-pyppeteer%E4%B8%8A%5C%E8%A2%AB%E5%B0%81.jpg" alt="被封"></p><p>于是只好更换selenium为pyppeteer。之前就听说了pyppeteer是可以代替selenium的模拟神器，相比Selenium 不太方便的地方：比如速度太慢、对版本配置要求严苛，最麻烦是经常要更新对应的驱动，还有些网页是可以检测到是否使用了Selenium ，pyppeteer都有提升，这次也借机会学习一下。</p><blockquote><p>安装：<code>pip install pyppeteer</code>, 需要Python3.6以上（因为用了async）</p></blockquote><h3 id="pyppeteer"><a class="markdownIt-Anchor" href="#pyppeteer"></a> pyppeteer</h3><p>最简单demo：打开网页，并截图</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"><span class="keyword">from</span> pyppeteer <span class="keyword">import</span> launch</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="comment"># 启动浏览器</span></span><br><span class="line">    browser = <span class="keyword">await</span> launch(headless=<span class="keyword">True</span>)</span><br><span class="line">    <span class="comment"># 打开并跳转页面</span></span><br><span class="line">    page = <span class="keyword">await</span> browser.newPage()</span><br><span class="line">    <span class="keyword">await</span> page.goto(<span class="string">'https://example.com'</span>)</span><br><span class="line">    <span class="comment"># 截图</span></span><br><span class="line">    <span class="keyword">await</span> page.screenshot(&#123;<span class="string">'path'</span>: <span class="string">'example.png'</span>&#125;)</span><br><span class="line">    <span class="keyword">await</span> browser.close()</span><br><span class="line"></span><br><span class="line">asyncio.get_event_loop().run_until_complete(main())</span><br></pre></td></tr></table></figure><p>执行JS，相当于requests-html中的render</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">dimensions = <span class="keyword">await</span> page.evaluate(<span class="string">'''() =&gt; &#123;</span></span><br><span class="line"><span class="string">    return &#123;</span></span><br><span class="line"><span class="string">        width: document.documentElement.clientWidth,</span></span><br><span class="line"><span class="string">        height: document.documentElement.clientHeight,</span></span><br><span class="line"><span class="string">        deviceScaleFactor: window.devicePixelRatio,</span></span><br><span class="line"><span class="string">    &#125;</span></span><br><span class="line"><span class="string">&#125;'''</span>)</span><br><span class="line"></span><br><span class="line">print(dimensions)</span><br><span class="line"><span class="comment"># &gt;&gt;&gt; &#123;'width': 800, 'height': 600, 'deviceScaleFactor': 1&#125;</span></span><br><span class="line"><span class="keyword">await</span> browser.close()</span><br></pre></td></tr></table></figure><p><strong>元素选择器</strong>：</p><p><code>Page.querySelector()</code>/<code>Page.querySelectorAll()</code>/<code>Page.xpath()</code>缩写为<code>Page.J()</code>, <code>Page.JJ()</code>, and <code>Page.Jx()</code>.</p><p>xpath进阶：</p><ul><li>starts-with 顾名思义，匹配一个属性开始位置的关键字</li><li>contains 匹配一个属性值中包含的<a href="https://so.csdn.net/so/search?q=%E5%AD%97%E7%AC%A6%E4%B8%B2&amp;spm=1001.2101.3001.7020" target="_blank" rel="noopener">字符串</a></li><li><code>text()</code> 匹配的是显示文本信息： <code>//a[contains(text(), &quot;百度搜索&quot;)]</code></li></ul><p><a href="https://blog.csdn.net/weixin_48615832/article/details/107319268" target="_blank" rel="noopener">xpath的contains用法</a></p><p><strong>元素操作：点击、输入</strong></p><ul><li><code>pyppeteer.input.Mouse</code><ul><li><code>click(x: float*, y: float, options: dict = None*, ***kwargs)</code><ul><li><code>button</code> (str): <code>left</code>, <code>right</code>, or <code>middle</code>, defaults to <code>left</code>：<ul><li>右键点击：<code>await page.click('a#cccccc',{&quot;button&quot;: &quot;right&quot;})</code></li></ul></li><li><code>clickCount</code> (int): defaults to 1.</li><li><code>delay</code> (int|float): Time to wait between <code>mousedown</code> and <code>mouseup</code> in milliseconds. Defaults to 0.</li></ul></li><li><code>down</code>(options: dict = None, *<em>kwargs</em>) `<ul><li><code>button</code> (str): <code>left</code>, <code>right</code>, or <code>middle</code>, defaults to <code>left</code>.</li><li><code>clickCount</code> (int): defaults to 1.</li></ul></li><li><code>move(x: float, y: float, options: dict = None, **kwargs*)</code></li><li><code>p(options: dict = None, **kwargs*)</code></li></ul></li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 使用xpath</span></span><br><span class="line"><span class="keyword">await</span> page.waitForXPath(<span class="string">'//input[contains(@id, "mat-input")]'</span>)</span><br><span class="line"><span class="keyword">await</span> asyncio.sleep(<span class="number">10</span>)</span><br><span class="line">inputs = <span class="keyword">await</span> page.xpath(<span class="string">'//input[contains(@id, "mat-input")]'</span>)</span><br><span class="line">print(len(inputs))</span><br><span class="line"><span class="keyword">await</span> inputs[<span class="number">1</span>].type(<span class="string">'janetgao@indiana.edu'</span>)</span><br><span class="line"><span class="keyword">await</span> inputs[<span class="number">2</span>].type(<span class="string">'KAProject2021'</span>)</span><br><span class="line"><span class="comment"># 使用selector</span></span><br><span class="line"><span class="keyword">await</span> page.waitForSelector(<span class="string">'#mat-input-1'</span>)</span><br><span class="line"><span class="keyword">await</span> page.type(<span class="string">'#mat-input-1'</span>, <span class="string">'janetgao@indiana.edu'</span>)</span><br><span class="line"><span class="keyword">await</span> page.type(<span class="string">'#mat-input-2'</span>, <span class="string">'KAProject2021'</span>)</span><br></pre></td></tr></table></figure><p><strong>launch</strong></p><ul><li><p><code>headless</code>：无界面模式</p></li><li><p><code>devtools</code> ：启用开发者助手</p></li><li><p><code>userDataDir='./userdata'</code>： 记录些用户信息，比如cookies</p></li><li><p><code>dumpio=True</code> 防止浏览器开多个页面而卡死</p></li><li><p><code>executablePath</code>: 指定自定义的chrome路径</p></li><li><p>参数args设置，可以类比selenium</p><ul><li><p><code>--start-maximized</code>: 页面全屏</p></li><li><p><code>--window-size={width},{height}</code>：设置窗口大小，区别于页面的<code>page.setViewport({'width': 1366, 'height': 768})</code></p></li><li><p><code>--disable-infobars</code>:不显示信息栏  比如 chrome正在受到自动测试软件的控制</p></li><li><p><code>--incognito</code>：百度搜到的chrome.exe无痕命令(selenium是如此，pyppeteer不推荐)</p></li><li><p><code>--user-agent</code>： 比如<code>'--user-agent=Mozilla/5.0'</code></p></li><li><p><code>--proxy-server={proxy}</code>: 浏览器代理 配合某些中间人代理使用</p></li><li><p><code>--no-sandbox</code>,  # 取消沙盒模式 沙盒模式下权限太小</p></li><li><p><code>'--log-level=3'</code></p></li><li><p><code>'--load-extension={}'.format(chrome_extension)</code>、<code>'--disable-extensions-except={}'.format(chrome_extension)</code> # 加载插件</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">chrome_extension_path = <span class="string">"插件所在目录"</span></span><br><span class="line">args = [</span><br><span class="line">    <span class="string">"--load-extension=&#123;&#125;"</span>.format(chrome_extension_path),</span><br><span class="line">    <span class="string">"--disable-extensions-except=&#123;&#125;"</span>.format(chrome_extension_path),</span><br><span class="line">]</span><br></pre></td></tr></table></figure><p><a href="https://www.cnblogs.com/dyfblog/p/12225796.html" target="_blank" rel="noopener">pyppeteer控制Chromium在隐身模式下启用插件 </a></p></li></ul></li><li><p>more: <a href="https://blog.csdn.net/Coding_CLearLove/article/details/110265509" target="_blank" rel="noopener">launch参数说明</a></p></li></ul><p>执行JS：</p><ul><li><p><code>page.evaluate('window.scrollBy(100, document.body.scrollHeight)')</code></p></li><li><p><code>await page.evaluate('alert(&quot;在浏览器执行js脚本!&quot;)')</code></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 相当于拿到了dom对象</span></span><br><span class="line">element = <span class="keyword">await</span> page.J(<span class="string">'#ul&gt;a[name="tj_trtieba"]'</span>)</span><br><span class="line">print(<span class="keyword">await</span> page.evaluate(<span class="string">'el =&gt; el.innerHTML'</span>, element))</span><br></pre></td></tr></table></figure></li><li><p>page.evaluateOnNewDocument(pageFunction[, …args]), 指定的函数在所属的页面被创建并且所属页面的任意script执行之前被调用。常用于修改页面JS环境。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">page = <span class="keyword">await</span> browser.newPage()</span><br><span class="line"><span class="keyword">await</span> page.setViewPort(&#123;<span class="string">'width'</span>: <span class="number">1366</span>, <span class="string">'height'</span>: <span class="number">768</span>&#125;)</span><br><span class="line"><span class="keyword">await</span> page.evaluateOnNewDocument(<span class="string">'''() =&gt; &#123;</span></span><br><span class="line"><span class="string">        Object.defineProperty(navigator, 'webdriver', &#123;get: () =&gt; false &#125;);</span></span><br></pre></td></tr></table></figure><p>附—ElementHandle元素操作：</p><ol><li>获取元素边界框坐标：boundingBox()，返回元素的边界框（相对于主框架）=&gt; x 坐标、 y 坐标、width、height</li><li>元素是否可见：isIntersectingViewport()</li><li>上传文件：uploadFile(*filpaths)</li><li>ElementHandle 类 转 Frame类：contentFrame()，如果句柄未引用iframe，则返回None。</li><li>聚焦该元素：focus()</li><li>与鼠标相关：hover () ，将鼠标悬停到元素上面</li><li>与键盘相关：press (key[, options])，按键，key 表示按键的名称，option可配置:<ul><li>text (string) - 如果指定，则使用此文本生成输入事件</li><li>delay (number) - keydown 和 keyup 之间等待的时间。默认是 0</li><li><code>page.keyboard.down('Shift')</code>：按下shift</li></ul></li></ol><p>more: <a href="https://blog.csdn.net/weizhen11/article/details/102509230" target="_blank" rel="noopener">页面元素操作</a></p></li></ul><p><strong>安全性：</strong></p><ol><li>在使用 Pyppeteer 仍然会遇到无头浏览器检测，这里安利一个第三方库「pyppeteer-stealth」，「puppeteer-extra-plugin-stealth」引用Github上的说明「Applies various evasion techniques to make detection of headless puppeteer harder.」「A plugin for puppeteer-extra to prevent detection.」（Github地址：<a href="https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth" target="_blank" rel="noopener">https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth</a> ）。可见，「pyppeteer-stealth」也是用于防止机器人检测的。用法也很简单，给page加上stealth</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"><span class="keyword">from</span> pyppeteer <span class="keyword">import</span> launch</span><br><span class="line"><span class="keyword">from</span> pyppeteer_stealth <span class="keyword">import</span> stealth</span><br><span class="line"> </span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line">    browser = <span class="keyword">await</span> launch(headless=<span class="keyword">True</span>)</span><br><span class="line">    page = <span class="keyword">await</span> browser.newPage()</span><br><span class="line"> </span><br><span class="line">    <span class="keyword">await</span> stealth(page)  <span class="comment"># &lt;-- Here</span></span><br><span class="line"> </span><br><span class="line">    <span class="keyword">await</span> page.goto(<span class="string">"https://bot.sannysoft.com/"</span>)</span><br><span class="line">    <span class="keyword">await</span> browser.close()</span><br><span class="line">asyncio.get_event_loop().run_until_complete(main())</span><br></pre></td></tr></table></figure><p>这样就可以省去隐藏WebDriver等操作，可谓省时省力省心。</p><ol start="2"><li><code>await page.evaluateOnNewDocument('Object.defineProperty(navigator,&quot;webdriver&quot;,{get:()=&gt;undefined})')</code></li><li><code>stealth.min.js</code><a href="https://cloud.tencent.com/developer/article/1755513" target="_blank" rel="noopener">最完美方案！模拟浏览器如何正确隐藏特征</a></li></ol><p><strong>无痕隐身模式访问</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">start_browser</span><span class="params">()</span>:</span></span><br><span class="line">    browser = <span class="keyword">await</span> launch(**&#123;<span class="string">"headless"</span>: <span class="keyword">False</span>&#125;)</span><br><span class="line">    <span class="comment"># 创建隐身上下文</span></span><br><span class="line">    browser_context = <span class="keyword">await</span> browser.createIncognitoBrowserContext()</span><br><span class="line">    page = <span class="keyword">await</span> browser_context.newPage()</span><br><span class="line"></span><br><span class="line">    <span class="keyword">await</span> page.goto(<span class="string">"http://www.baidu.com"</span>)</span><br><span class="line">    <span class="keyword">await</span> browser.close()</span><br><span class="line">    <span class="keyword">return</span></span><br></pre></td></tr></table></figure><p>较全的API说明：<a href="https://blog.csdn.net/Qwertyuiop2016/article/details/92667875" target="_blank" rel="noopener">pyppeteer的基本使用</a>、<a href="https://blog.csdn.net/weizhen11/article/details/102497647" target="_blank" rel="noopener">页面信息、设置</a></p><h3 id="问题按钮不生效"><a class="markdownIt-Anchor" href="#问题按钮不生效"></a> 问题：按钮不生效</h3><p>网页输入账号和密码之前，按钮状态为<code>disable=true</code>，两个都输入后才能按下，而使用pyppeteer时，发现无论怎么尝试都不能使得其disable变化（用开发者助手修改后，无法生成正确的提交请求）。突然巧合之中，想到了是不是元素没加载成功，于是确实在开发者助手的<code>console</code>中看到有很多不支持的，比如angular、bootstrap……</p><p>于是百度问题，看看是不是什么设置没开，但都没找到。后来想到升级下pyppeteer版本会不会好一点，发现自己的版本为1.0.2，已经是最新的了(<a href="https://github.com/miyakogi/pyppeteer/issues/306" target="_blank" rel="noopener">https://github.com/miyakogi/pyppeteer/issues/306</a>)。由于不通过程序直接通过浏览器进入程序也出现了无法点击的问题。再加上下面有人提出chromium有点难下载，并通过源码找道了默认的配置和设置，于是经过一阵的折腾过后，我发现**“按钮无法点击”的问题就在于chromium版本过低**。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pyppeteer.chromium_downloader</span><br><span class="line">print(<span class="string">'默认版本是：&#123;&#125;'</span>.format(pyppeteer.__chromium_revision__))</span><br><span class="line">print(<span class="string">'可执行文件默认路径：&#123;&#125;'</span>.format(pyppeteer.chromium_downloader.chromiumExecutable.get(<span class="string">'win64'</span>)))</span><br><span class="line">print(<span class="string">'win64平台下载链接为：&#123;&#125;'</span>.format(pyppeteer.chromium_downloader.downloadURLs.get(<span class="string">'win64'</span>)))</span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">默认版本是：588429</span></span><br><span class="line"><span class="string">可执行文件默认路径：C:\Users\mrli\AppData\Local\pyppeteer\pyppeteer\local-chromium\588429\chrome-win32\chrome.exe</span></span><br><span class="line"><span class="string">win64平台下载链接为：https://storage.googleapis.com/chromium-browser-snapshots/Win_x64/588429/chrome-win32.zip</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure><p>登录这个storage的网站可以看到，上面所记录的chromium全都是2013年的版本，必然版本落后了。于是下载最新的chromium覆盖<code>C:\Users\mrli\AppData\Local\pyppeteer\pyppeteer\local-chromium\588429\chrome-win32</code>目录后再运行，按钮不能点击的问题就消失了!~😄</p><p>chromium下载地址：<a href="https://registry.npmmirror.com/binary.html?path=chromium-browser-snapshots/Win_x64/%EF%BC%8C%E6%B3%A8%EF%BC%9Achromium%E8%B7%9Fchromedriver%E4%B8%8D%E4%B8%80%E6%A0%B7%EF%BC%8C%E4%BB%96%E8%B7%9F%E8%87%AA%E5%B7%B1%E5%B7%B2%E7%BB%8F%E5%AE%89%E8%A3%85%E7%9A%84chrome%E5%B9%B6%E6%B2%A1%E6%9C%89%E5%85%B3%E8%81%94%E3%80%82chromium%E8%87%AA%E5%B8%A6%E6%B5%8F%E8%A7%88%E5%99%A8%E5%BC%95%E6%93%8E%E6%A0%B8%E5%BF%83%EF%BC%8C%E6%89%80%E4%BB%A5%E7%9B%B4%E6%8E%A5%E4%B8%8B%E8%BD%BD%E6%9C%80%E6%96%B0%E7%9A%84%E7%89%88%E6%9C%AC%E5%8D%B3%E5%8F%AF%EF%BC%8C%E6%AF%94%E5%A6%82%E6%88%91%E7%9B%B4%E6%8E%A5%E4%B8%8B%E8%BD%BD%E7%9A%84%E5%B0%B1%E6%98%AF999303%E3%80%82" target="_blank" rel="noopener">https://registry.npmmirror.com/binary.html?path=chromium-browser-snapshots/Win_x64/，注：chromium跟chromedriver不一样，他跟自己已经安装的chrome并没有关联。chromium自带浏览器引擎核心，所以直接下载最新的版本即可，比如我直接下载的就是999303。</a></p><p>启发于：<a href="https://cloud.tencent.com/developer/article/1731720" target="_blank" rel="noopener">使用pyppeteer 下载chromium 报错 或速度慢</a>、<a href="https://www.cnblogs.com/Summer-skr--blog/p/12020207.html" target="_blank" rel="noopener">pyppeteer使用时常见的bug及解决办法【转载】 </a></p><ul><li>📖pyppeteer实战案例：<a href="https://blog.csdn.net/weixin_44826986/article/details/124114130" target="_blank" rel="noopener">Pyppeteer的使用——爬取京东</a>——消除指纹、页面滑动事件、获取数据，获取到的是Json数据：<code>(await a[0].getProperty(&quot;textContent&quot;)).jsonValue()</code></li><li>📖官方API： <a href="https://pyppeteer.github.io/pyppeteer/reference.html#pyppeteer.page.Page.xpath%E3%80%81" target="_blank" rel="noopener">https://pyppeteer.github.io/pyppeteer/reference.html#pyppeteer.page.Page.xpath、</a><a href="https://learnku.com/docs/puppeteer/3.1.0/class-mouse/8549" target="_blank" rel="noopener">puppeteer中文手册</a></li><li><a href="https://www.pythonheidong.com/blog/article/1133668/afc4477523a740d64a23/" target="_blank" rel="noopener">setViewport全屏</a></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h2 id=&quot;验证码问题&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#验证码问题&quot;&gt;&lt;/a&gt; 验证码问题&lt;/h2&gt;
&lt;p&gt;该网站对频率检测十分敏感，很快就会出现&amp;quot;按住&amp;quot;验证码，模拟按住倒不难，但是按完以后有时并不能解决验证…&lt;/
      
    
    </summary>
    
    
      <category term="爬虫" scheme="https://nymrli.top/tags/%E7%88%AC%E8%99%AB/"/>
    
      <category term="环境配置" scheme="https://nymrli.top/tags/%E7%8E%AF%E5%A2%83%E9%85%8D%E7%BD%AE/"/>
    
      <category term="Python" scheme="https://nymrli.top/tags/Python/"/>
    
  </entry>
  
  <entry>
    <title>mitmproxy拦截代理——PC版HttpCanary</title>
    <link href="https://nymrli.top/2022/04/29/mitmproxy%E6%8B%A6%E6%88%AA%E4%BB%A3%E7%90%86%E2%80%94%E2%80%94PC%E7%89%88HttpCanary/"/>
    <id>https://nymrli.top/2022/04/29/mitmproxy拦截代理——PC版HttpCanary/</id>
    <published>2022-04-29T12:52:09.000Z</published>
    <updated>2022-05-23T14:13:56.515Z</updated>
    
    <content type="html"><![CDATA[<h2 id="httpcanary"><a class="markdownIt-Anchor" href="#httpcanary"></a> HttpCanary</h2><p>HttpCanary是Android系统下功能最强大的网络分析工具(IOS的是Thro)，支持TCP/UDP/HTTP/HTTPS/<u>WebSocket</u>等多种协议，可以视为Android平台下的Fiddler和Charles。</p><p><strong>★强大的调试功能</strong></p><ul><li>断点指定请求，支持修改数据再提交。</li><li>注入器支持，创建自定义规则修改返回数据包（★静态注入）。</li><li>支持对网络请求重发、编辑重发、批量重发等操作。</li><li>配置黑名单或者白名单，自由指定目标抓包。</li><li>支持数据包屏蔽，比如屏蔽客户端发向服务器的指定数据包。</li></ul><h2 id="mitmproxy"><a class="markdownIt-Anchor" href="#mitmproxy"></a> mitmproxy</h2><p>顾名思义，mitmproxy 就是用于 MITM 的 proxy，MITM 即<a href="https://zh.wikipedia.org/wiki/%E4%B8%AD%E9%97%B4%E4%BA%BA%E6%94%BB%E5%87%BB" target="_blank" rel="noopener">中间人攻击</a>（Man-in-the-middle attack）。用于中间人攻击的<strong>代理</strong>首先会像正常的代理一样转发请求，保障<strong>服务端</strong>与<strong>客户端</strong>的通信，其次，会适时的查、记录其截获的数据，或篡改数据，引发服务端或客户端特定的行为。不同于 fiddler 或 wireshark 等抓包工具，mitmproxy 不仅可以<strong>截获请求</strong>帮助开发者查看、分析，更可以通过自定义脚本进行二次开发。</p><p>注：mitmproxy最大的特点跟httpCanary一样，其拥有修改响应的能力，实现如下功能：：“截获对浏览器对该 url 的请求，将返回内容置空，并将真实的返回内容存到某个数据库，出现异常时发出邮件通知”，这些都</p><p>但 mitmproxy 并不会真的对无辜的人发起中间人攻击，由于 mitmproxy 工作在 HTTP 层，而当前 HTTPS 的普及让客户端拥有了检测并规避中间人攻击的能力，所以要让 mitmproxy 能够正常工作，必须要让<strong>客户端（APP 或浏览器）主动信任 mitmproxy 的 SSL 证书</strong>，或忽略证书异常，★<strong>这也就意味着 APP 或浏览器是属于开发者本人的</strong>★——显而易见，这不是在做黑产，而是在做开发或测试。</p><p><strong>特性</strong></p><ul><li>拦截HTTP和HTTPS请求和响应并即时修改它们</li><li>保存完整的HTTP对话以供以之后重发和分析</li><li>重发HTTP对话的客户端</li><li>重发先前记录的服务的HTTP响应</li><li>反向代理模式将流量转发到指定的服务器</li><li>在macOS和Linux上实现透明代理模式</li><li>使用Python对HTTP流量进行脚本化修改</li><li>实时生成用于拦截的SSL / TLS证书</li></ul><h3 id="安装"><a class="markdownIt-Anchor" href="#安装"></a> 安装</h3><p>“安装 mitmproxy”这句话是有歧义的，既可以指①“安装 mitmproxy 工具”，也可以指②“安装 python 的 mitmproxy 包”，注意后者是包含前者的。即安装 python 的 mitmproxy 包除了会得到 mitmproxy 工具外，还会得到开发定制脚本所需要的包依赖，其安装过程并不复杂。</p><p>如果只是拿 mitmproxy 做一个替代 fiddler 的工具，没有什么定制化的需求（修改响应等功能），那完全只需要“安装 mitmproxy 工具”即可，去 <a href="https://mitmproxy.org/" target="_blank" rel="noopener">mitmproxy 官网</a> 上下载一个 installer 便可开箱即用，不需要提前准备好 python 开发环境。但显然，这不是这里要讨论的，我们需要的是“安装 python 的 mitmproxy 包”。</p><p>安装命令：<code>pip install mitmproxy</code></p><p>完成后，系统将拥有 <code>mitmproxy</code>（windows中不支持）、<code>mitmdump</code>、<code>mitmweb</code> 三个命令，可以拿 <code>mitmdump</code> 测试一下安装是否成功，执行：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mitmdump --version</span><br></pre></td></tr></table></figure><p>应当可以看到类似于这样的输出：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">Mitmproxy: 4.0.1</span><br><span class="line">Python:    3.6.5</span><br><span class="line">OpenSSL:   OpenSSL 1.1.0h  27 Mar 2018</span><br><span class="line">Platform:  Windows-10-10.0.16299-SP0</span><br></pre></td></tr></table></figure><h3 id="运行"><a class="markdownIt-Anchor" href="#运行"></a> 运行</h3><p>要启动 mitmproxy 用 <code>mitmproxy</code>、<code>mitmdump</code>、<code>mitmweb</code> 这三个命令中的任意一个即可，这三个命令功能一致，且都可以加载自定义脚本，唯一的区别是交互界面的不同。</p><ul><li><code>mitmproxy</code> 命令启动后，会提供一个命令行界面，用户可以实时看到发生的请求，并通过命令过滤请求，查看请求数据。</li><li><code>mitmweb</code> 命令启动后，会提供一个 web 界面，用户可以实时看到发生的请求，并通过 GUI 交互来过滤请求，查看请求数据。</li><li><code>mitmdump</code> 命令启动后——没有界面，程序默默运行，所以 mitmdump 无法提供过滤请求、查看数据的功能，只能结合自定义脚本，默默工作。</li></ul><p>由于 <code>mitmproxy</code> 命令的交互操作稍显繁杂且不支持 windows 系统，而我们主要的使用方式又是载入自定义脚本，并不需要交互，所以原则上说只需要 <code>mitmdump</code> 即可，但考虑到有交互界面可以更方便排查错误，所以这里以 <code>mitmweb</code> 命令为例。实际使用中可以根据情况选择任何一个命令。</p><h4 id="mitmweb"><a class="markdownIt-Anchor" href="#mitmweb"></a> mitmweb</h4><p>mitmproxy 绑定了 <code>*:8080</code> 作为代理端口，并提供了一个 web 交互界面在 <code>127.0.0.1:8081</code>。</p><p>现在可以测试一下代理，让 Chrome 以 mitmproxy 为代理并忽略证书错误。为了不影响平时正常使用，我们不去改 Chrome 的配置，而是通过命令行带参数起一个 Chrome。如果你不使用 Chrome 而是其他浏览器，也可以搜一下对应的启动参数是什么，应该不会有什么坑。此外示例仅以 windows 系统为例，因为使用 linux 或 mac 开发的同学应该更熟悉命令行的使用才对，应当能自行推导出在各自环境中对应的操作。</p><p>接下来关闭所有 Chrome 窗口，否则命令行启动时的附加参数将失效。打开 cmd，执行：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 前面那一长串是 Chrome 的的安装路径，应当根据系统实际情况修改，后面两参数设置了代理地址并强制忽略掉证书错误。</span></span><br><span class="line">$ <span class="string">"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"</span> --proxy-server=127.0.0.1:8080 --ignore-certificate-errors</span><br></pre></td></tr></table></figure><p>▲推荐先使用<code>mitmdump</code>：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ mitmdump </span><br><span class="line">Proxy server listening at http://*:8080</span><br><span class="line"><span class="comment"># 然后windows设置 -&gt; 代理 -&gt; 127.0.0.1 8080 -&gt; 保存</span></span><br></pre></td></tr></table></figure><p>但是如果没安装https证书的话，是抓取不了https数据的，因此需要安装证书。</p><h3 id="证书安装"><a class="markdownIt-Anchor" href="#证书安装"></a> 证书安装</h3><h4 id="pc"><a class="markdownIt-Anchor" href="#pc"></a> PC</h4><p>官网介绍<a href="https://docs.mitmproxy.org/stable/concepts-certificates/" target="_blank" rel="noopener">About Certificates</a></p><p>使用<code>mitmdump</code>之后下面的文件就会被创建，位于<code>C:\Users\mrli\.mitmproxy</code></p><table><thead><tr><th style="text-align:left">Filename</th><th style="text-align:left">Contents</th></tr></thead><tbody><tr><td style="text-align:left">mitmproxy-ca.pem</td><td style="text-align:left">The certificate <strong>and the private key</strong> in PEM format.</td></tr><tr><td style="text-align:left">mitmproxy-ca-cert.pem</td><td style="text-align:left">The certificate in PEM format. Use this to distribute on most non-Windows platforms.</td></tr><tr><td style="text-align:left">mitmproxy-ca-cert.p12</td><td style="text-align:left">The certificate in PKCS12 format. <strong>For use on Windows.</strong></td></tr><tr><td style="text-align:left">mitmproxy-ca-cert.cer</td><td style="text-align:left">Same file as .pem, but with an extension expected by some Android devices.</td></tr></tbody></table><p>双击<code>mitmproxy-ca-cert.p12</code>安装，“本地计算机”-&gt;<code>下一页</code>-&gt;<code>下一页</code>-&gt;<code>将所有的证书都放入下列存储</code>(受信任的根证书颁布机构)然后下一页，即可完整证书安装</p><h4 id="手机"><a class="markdownIt-Anchor" href="#手机"></a> 手机</h4><p>WIFI中设置代理，ip需要在电脑上通过<code>ipconfig</code>查看，而port参数跟上面一样，通过<code>mitmdump</code>可以看到侦听的端口具体为多少，默认为8080。</p><p>证书安装：进入网址<code>mitm.it</code>，根据OS选择安装证书</p><p>详情见: <a href="https://www.bilibili.com/video/BV1UC4y1t7EL?p=1" target="_blank" rel="noopener">mitmproxy抓包工具！！！ 从安装到简单使用</a></p><h3 id="脚本"><a class="markdownIt-Anchor" href="#脚本"></a> ★脚本</h3><p>接下来开始开发自定义脚本，这才是 mitmproxy 真正强大的地方（可以在python代码中直接处理数据包）。</p><p>总共有两种写法：</p><ol><li>定义一些接受指定参数的函数</li></ol><blockquote><p>编写一个 py 文件供 mitmproxy 加载，文件中定义了若干函数，这些函数实现了某些 mitmproxy 提供的事件，mitmproxy 会在某个事件发生时调用对应的函数，形如：</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> mitmproxy.http</span><br><span class="line"><span class="keyword">from</span> mitmproxy <span class="keyword">import</span> ctx</span><br><span class="line"></span><br><span class="line">num = <span class="number">0</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">request</span><span class="params">(flow: mitmproxy.http.HTTPFlow)</span>:</span></span><br><span class="line">    <span class="keyword">global</span> num</span><br><span class="line">    num = num + <span class="number">1</span></span><br><span class="line">    ctx.log.info(<span class="string">"We've seen %d flows"</span> % num)</span><br></pre></td></tr></table></figure><p>第二个是，编写一个 py 文件供 mitmproxy 加载，文件定义了变量 addons，addons 是个数组，每个元素是一个类实例，这些类有若干方法，这些方法实现了某些 mitmproxy 提供的事件，mitmproxy 会在某个事件发生时调用对应的方法。这些类，称为一个个 <code>addon</code>，比如一个叫 Counter 的 addon(插件)：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> mitmproxy.http</span><br><span class="line"><span class="keyword">from</span> mitmproxy <span class="keyword">import</span> ctx</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Counter</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        self.num = <span class="number">0</span></span><br><span class="line"><span class="comment"># 重写指定方法</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">request</span><span class="params">(self, flow: mitmproxy.http.HTTPFlow)</span>:</span></span><br><span class="line">        self.num = self.num + <span class="number">1</span></span><br><span class="line">        ctx.log.info(<span class="string">"We've seen %d flows"</span> % self.num)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">addons = [</span><br><span class="line">    Counter()</span><br><span class="line">]</span><br></pre></td></tr></table></figure><p>⭐️这里强烈建议使用第二种套路，直觉上就会感觉第二种套路更为先进，使用会更方便也更容易管理和拓展。况且这也是<a href="https://github.com/mitmproxy/mitmproxy/tree/master/mitmproxy/addons" target="_blank" rel="noopener">官方内置的一些 addon</a> 的实现方式。</p><p>我们将上面第二种套路的示例代码存为 <a href="http://addons.py" target="_blank" rel="noopener">addons.py</a>，再重新启动 mitmproxy：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mitmweb -s addons.py</span><br></pre></td></tr></table></figure><p>当浏览器使用代理进行访问时，就应该能看到控制台里有类似这样的日志：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Web server listening at http://127.0.0.1:8081/</span><br><span class="line">Loading script addons.py</span><br><span class="line">Proxy server listening at http://*:8080</span><br><span class="line">We&apos;ve seen 1 flows</span><br><span class="line">……</span><br></pre></td></tr></table></figure><p>这就说明自定义脚本生效了。</p><p>总结：自定义脚本可以实现拦截请求，从而对请求头、请求、响应头、响应头进行修改的效果——当人操作时，就能在拦截请求进行处理的时候进行数据处理。所以可以mitmproxy+appnium或者selenium实现模拟点击+抓数据包的方式。</p><h3 id="事件"><a class="markdownIt-Anchor" href="#事件"></a> 事件</h3><p>上述当 request 时间发生时，计数器加一，并打印日志。这里对应的是 request 事件，那么mitmproxy中一共有哪些事件呢？</p><p>参考：<a href="https://www.cnblogs.com/grandlulu/p/9525417.html" target="_blank" rel="noopener">使用 mitmproxy + python 做拦截代理</a></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h2 id=&quot;httpcanary&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#httpcanary&quot;&gt;&lt;/a&gt; HttpCanary&lt;/h2&gt;
&lt;p&gt;HttpCanary是Android系统下功能最强大的网络分析工具(IOS的是Thro)，支持T
      
    
    </summary>
    
    
      <category term="爬虫" scheme="https://nymrli.top/tags/%E7%88%AC%E8%99%AB/"/>
    
  </entry>
  
  <entry>
    <title>使用Python转义实现前端加密算法-记录笔记</title>
    <link href="https://nymrli.top/2022/04/29/%E4%BD%BF%E7%94%A8Python%E8%BD%AC%E4%B9%89%E5%AE%9E%E7%8E%B0%E5%89%8D%E7%AB%AF%E5%8A%A0%E5%AF%86%E7%AE%97%E6%B3%95-%E8%AE%B0%E5%BD%95%E7%AC%94%E8%AE%B0/"/>
    <id>https://nymrli.top/2022/04/29/使用Python转义实现前端加密算法-记录笔记/</id>
    <published>2022-04-29T01:22:28.000Z</published>
    <updated>2022-04-29T04:36:04.026Z</updated>
    
    <content type="html"><![CDATA[<h2 id="环境安装"><a class="markdownIt-Anchor" href="#环境安装"></a> 环境安装</h2><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">pip uninstall crypto</span><br><span class="line">pip uninstall pycryptodome</span><br><span class="line">pip install pycryptodome</span><br></pre></td></tr></table></figure><p>前面两个卸载命令是为了防止一些安装环境问题，具体请看<a href="https://blog.csdn.net/chouzhou9701/article/details/106432497/" target="_blank" rel="noopener">文章</a></p><h1 id="pycryptodome"><a class="markdownIt-Anchor" href="#pycryptodome"></a> pycryptodome</h1><h2 id="aes加密"><a class="markdownIt-Anchor" href="#aes加密"></a> AES加密</h2><p><strong>加密模式</strong></p><p>AES 加密最常用的模式就是 ECB模式 和 CBC 模式，当然还有很多其它模式，他们都属于AES加密。ECB模式和CBC 模式俩者区别就是 ECB 不需要 iv偏移量，而CBC需要。</p><p><strong>AES加密使用参数</strong></p><p>以下参数都是在python中使用的。</p><p>参数作用及数据类型<br>秘钥加密的时候用秘钥，解密的时候需要同样的秘钥才能解出来; 数据类型为bytes<br>明文需要加密的参数; 数据类型为bytes<br>模式aes 加密常用的有 ECB 和 CBC 模式（我只用了这两个模式，还有其他模式）;数据类型为aes类内部的枚举量<br>iv 偏移量这个参数在 ECB 模式下不需要，在 CBC 模式下需要；数据类型为bytes</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># ECB模式加密解密</span></span><br><span class="line"><span class="keyword">from</span> Crypto.Cipher <span class="keyword">import</span> AES</span><br><span class="line"></span><br><span class="line">password = <span class="string">b'1234567812345678'</span> <span class="comment">#秘钥，b就是表示为bytes类型</span></span><br><span class="line">text = <span class="string">b'abcdefghijklmnhi'</span> <span class="comment">#需要加密的内容，bytes类型</span></span><br><span class="line">aes = AES.new(password,AES.MODE_ECB) <span class="comment">#创建一个aes对象</span></span><br><span class="line"><span class="comment"># AES.MODE_ECB 表示模式是ECB模式</span></span><br><span class="line">en_text = aes.encrypt(text) <span class="comment">#加密明文</span></span><br><span class="line">print(<span class="string">"密文："</span>,en_text) <span class="comment">#加密明文，bytes类型</span></span><br><span class="line"></span><br><span class="line">den_text = aes.decrypt(en_text) <span class="comment"># 解密密文</span></span><br><span class="line">print(<span class="string">"明文："</span>,den_text)</span><br></pre></td></tr></table></figure><p>以上是针对ECB模式的加密解密，从这个例子中可以看出参数中有几个限制。</p><ol><li>秘钥必须为16字节或者16字节的倍数的字节型数据。</li><li>明文必须为16字节或者16字节的倍数的字节型数据，如果不够16字节需要进行补全，关于补全规则，后面会在补全模式中具体介绍。</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># CBC</span></span><br><span class="line"><span class="keyword">from</span> Crypto.Cipher <span class="keyword">import</span> AES</span><br><span class="line"></span><br><span class="line">password = <span class="string">b'1234567812345678'</span> <span class="comment">#秘钥，b就是表示为bytes类型</span></span><br><span class="line">iv = <span class="string">b'1234567812345678'</span> <span class="comment"># iv偏移量，bytes类型</span></span><br><span class="line">text = <span class="string">b'abcdefghijklmnhi'</span> <span class="comment">#需要加密的内容，bytes类型</span></span><br><span class="line">aes = AES.new(password,AES.MODE_CBC,iv) <span class="comment">#创建一个aes对象</span></span><br><span class="line"><span class="comment"># AES.MODE_CBC 表示模式是CBC模式</span></span><br><span class="line">en_text = aes.encrypt(text) </span><br><span class="line">print(<span class="string">"密文："</span>,en_text) <span class="comment">#加密明文，bytes类型</span></span><br><span class="line"></span><br><span class="line">aes = AES.new(password,AES.MODE_CBC,iv) <span class="comment">#CBC模式下解密需要重新创建一个aes对象</span></span><br><span class="line">den_text = aes.decrypt(en_text)</span><br><span class="line">print(<span class="string">"明文："</span>,den_text)</span><br></pre></td></tr></table></figure><h1 id="hashlib"><a class="markdownIt-Anchor" href="#hashlib"></a> hashlib</h1><h2 id="md5"><a class="markdownIt-Anchor" href="#md5"></a> MD5</h2><blockquote><p>hashlib模块下的&quot;加密&quot;</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> hashlib</span><br><span class="line">string=<span class="string">'任性的90后boy'</span></span><br><span class="line">md5 = hashlib.md5()</span><br><span class="line">md5.update(string.encode(<span class="string">'utf-8'</span>))<span class="comment">#转码，update里的必须是字节型</span></span><br><span class="line"><span class="comment"># 默认是32位小写</span></span><br><span class="line">res = md5.hexdigest() <span class="comment">#返回字符型摘要信息print(md5.digest())#返回字节型的摘要信息</span></span><br><span class="line"><span class="comment"># 切换到16位大写</span></span><br><span class="line"><span class="comment"># res=md5.hexdigest()[8:-8]</span></span><br><span class="line">print(<span class="string">"md5加密结果:"</span>,res)</span><br></pre></td></tr></table></figure><h2 id="sha256"><a class="markdownIt-Anchor" href="#sha256"></a> Sha256</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> hashlib</span><br><span class="line">string=<span class="string">'任性的90后boy'</span></span><br><span class="line">sha256 = hashlib.sha256()</span><br><span class="line">sha256.update(string.encode(<span class="string">'utf-8'</span>))</span><br><span class="line">res = sha256.hexdigest()</span><br><span class="line">print(<span class="string">"sha256加密结果:"</span>,res)</span><br></pre></td></tr></table></figure>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h2 id=&quot;环境安装&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#环境安装&quot;&gt;&lt;/a&gt; 环境安装&lt;/h2&gt;
&lt;figure class=&quot;highlight shell&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;s
      
    
    </summary>
    
    
      <category term="爬虫" scheme="https://nymrli.top/tags/%E7%88%AC%E8%99%AB/"/>
    
      <category term="Python" scheme="https://nymrli.top/tags/Python/"/>
    
  </entry>
  
  <entry>
    <title>V-P-N(Vmess)与Charles无法共用问题解决</title>
    <link href="https://nymrli.top/2022/04/27/V-P-N%E4%B8%8ECharles%E6%97%A0%E6%B3%95%E5%85%B1%E7%94%A8%E9%97%AE%E9%A2%98%E8%A7%A3%E5%86%B3/"/>
    <id>https://nymrli.top/2022/04/27/V-P-N与Charles无法共用问题解决/</id>
    <published>2022-04-27T13:30:51.000Z</published>
    <updated>2022-05-07T02:06:15.361Z</updated>
    
    <content type="html"><![CDATA[<p>最近需要抓一个需要翻墙才能访问的网页的包，发现VPN直连时会导致 Fiddler 和 Charles 抓包工具无法正常进行抓包，网上找了以后发现了一些解决方案：<a href="https://github.com/mopduan/team/issues/13" target="_blank" rel="noopener">Github:VPN直连，导致 Fiddler 和 Charles 抓包工具无法正常进行抓包解决方案 </a>——试了貌似没用、<a href="https://www.jianshu.com/p/651ee335f97f?from=singlemessage" target="_blank" rel="noopener">windows下，实现vpn访问下的charles抓包设置中无网络问题的解决</a>——收此启发指导了在charles的<code>Proxy-&gt;external proxy</code>允许其他端口代理</p><h2 id="1找到vpn软件的代理端口proxy-port"><a class="markdownIt-Anchor" href="#1找到vpn软件的代理端口proxy-port"></a> 1.找到VPN软件的代理端口proxy port</h2><p>我这边使用的是vmess，可以在<code>选项-&gt;参数设置</code>中查看，需要明确的参数是端口和协议，我这边是10808和socks协议</p><p><img src="/2022/04/27/V-P-N与Charles无法共用问题解决/.%5CV-P-N%E4%B8%8ECharles%E6%97%A0%E6%B3%95%E5%85%B1%E7%94%A8%E9%97%AE%E9%A2%98%E8%A7%A3%E5%86%B3%5Cvmess%E5%8F%82%E6%95%B0.jpg" alt="vmess参数"></p><h2 id="2设置charles"><a class="markdownIt-Anchor" href="#2设置charles"></a> 2.设置charles：</h2><p><code>Proxy-&gt;external proxy</code>, 首先允许其他proxy，然后根据刚刚查看到的vmess端口和协议进行填写</p><p><img src="/2022/04/27/V-P-N与Charles无法共用问题解决/.%5CV-P-N%E4%B8%8ECharles%E6%97%A0%E6%B3%95%E5%85%B1%E7%94%A8%E9%97%AE%E9%A2%98%E8%A7%A3%E5%86%B3%5Ccharles.jpg" alt="charles"></p><h2 id="3设置完成开始抓包"><a class="markdownIt-Anchor" href="#3设置完成开始抓包"></a> 3.设置完成，开始抓包</h2><p>完结撒花~</p><h1 id="附录"><a class="markdownIt-Anchor" href="#附录"></a> 附录：</h1><h2 id="requests使用代理"><a class="markdownIt-Anchor" href="#requests使用代理"></a> requests使用代理</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">cookies = &#123;</span><br><span class="line">    <span class="string">'PB3_SESSION'</span>: <span class="string">'"2|1:0|10:1650810241|11:PB3_SESSION|40:djJleDo1Mi4xNDAuMjAxLjIxMTo1OTQ4NjM0Mg==|f661892137fd704b91fa09d8c58fd641a15ab9e83f94c69981dbeed7980fc9e4"'</span>,</span><br><span class="line">    <span class="string">'V2EX_LANG'</span>: <span class="string">'zhcn'</span>,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">headers = &#123;</span><br><span class="line">    <span class="string">'authority'</span>: <span class="string">'cn.v2ex.com'</span>,</span><br><span class="line">    <span class="string">'accept'</span>: <span class="string">'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'</span>,</span><br><span class="line">    <span class="string">'accept-language'</span>: <span class="string">'zh-CN,zh;q=0.9,en;q=0.8'</span>,</span><br><span class="line">    <span class="string">'cache-control'</span>: <span class="string">'no-cache'</span>,</span><br><span class="line">    <span class="string">'pragma'</span>: <span class="string">'no-cache'</span>,</span><br><span class="line">    <span class="string">'sec-ch-ua'</span>: <span class="string">'" Not A;Brand";v="99", "Chromium";v="100", "Google Chrome";v="100"'</span>,</span><br><span class="line">    <span class="string">'sec-ch-ua-mobile'</span>: <span class="string">'?0'</span>,</span><br><span class="line">    <span class="string">'sec-ch-ua-platform'</span>: <span class="string">'"Windows"'</span>,</span><br><span class="line">    <span class="string">'sec-fetch-dest'</span>: <span class="string">'document'</span>,</span><br><span class="line">    <span class="string">'sec-fetch-mode'</span>: <span class="string">'navigate'</span>,</span><br><span class="line">    <span class="string">'sec-fetch-site'</span>: <span class="string">'none'</span>,</span><br><span class="line">    <span class="string">'sec-fetch-user'</span>: <span class="string">'?1'</span>,</span><br><span class="line">    <span class="string">'upgrade-insecure-requests'</span>: <span class="string">'1'</span>,</span><br><span class="line">    <span class="string">'user-agent'</span>: <span class="string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36'</span>,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">http_proxy = <span class="string">"socks5h://127.0.0.1:10808"</span></span><br><span class="line">https_proxy = <span class="string">"socks5h://127.0.0.1:10808"</span></span><br><span class="line">proxies = &#123;</span><br><span class="line">    <span class="string">"https"</span>: https_proxy,</span><br><span class="line">    <span class="string">"http"</span>: http_proxy</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">response = requests.get(<span class="string">'https://cn.v2ex.com/about'</span>, cookies=cookies, headers=headers, proxies=proxies)</span><br></pre></td></tr></table></figure><p>注：一开始在Sublime里运行的，结果一直在<code>response.text</code>时报编码错误，但是通过网页的content-type和meta charset进行确认过没问题，后来经过一个启发想到会不会是控制台有编码显示不了，于是在Pycharm中运行，成功!</p><h2 id="aiohttp使用socks代理"><a class="markdownIt-Anchor" href="#aiohttp使用socks代理"></a> aiohttp使用socks代理</h2><blockquote><p>from: <a href="https://pypi.org/project/aiohttp-socks/%E3%80%81https://www.cnblogs.com/john-xiong/p/13812567.html" target="_blank" rel="noopener">https://pypi.org/project/aiohttp-socks/、https://www.cnblogs.com/john-xiong/p/13812567.html</a></p></blockquote><ol><li><p><code>pip install aiohttp_socks</code></p></li><li><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">connector = ProxyConnector.from_url(<span class="string">'socks5://127.0.0.1:10808'</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">getDataByChromeDriver</span><span class="params">(url)</span>:</span></span><br><span class="line"><span class="keyword">async</span> <span class="keyword">with</span> aiohttp.ClientSession(connector=connector) <span class="keyword">as</span> session:</span><br><span class="line">        <span class="keyword">async</span> <span class="keyword">with</span> session.get(url) <span class="keyword">as</span> response:</span><br><span class="line">            <span class="keyword">return</span> <span class="keyword">await</span> response.text()</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">        loop.run_until_complete(asyncio.wait([getDataByChromeDriver(index) <span class="keyword">for</span> title, index <span class="keyword">in</span> title_list.items()]))</span><br></pre></td></tr></table></figure></li><li><p>运行即可</p></li></ol><h2 id="request-html使用代理"><a class="markdownIt-Anchor" href="#request-html使用代理"></a> request-html使用代理</h2><blockquote><p><a href="https://baijiahao.baidu.com/s?id=1701142223076604985&amp;wfr=spider&amp;for=pc" target="_blank" rel="noopener">Python爬虫一个requests_html模块足矣！（支持JS加载&amp;异步请求）</a></p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> requests_html <span class="keyword">import</span> AsyncHTMLSession</span><br><span class="line"></span><br><span class="line">http_proxy = <span class="string">"socks5h://127.0.0.1:10808"</span></span><br><span class="line">https_proxy = <span class="string">"socks5h://127.0.0.1:10808"</span></span><br><span class="line">proxies = &#123;</span><br><span class="line">    <span class="string">"https"</span>: https_proxy,</span><br><span class="line">    <span class="string">"http"</span>: http_proxy</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">session = AsyncHTMLSession()</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">getDataByChromeDriver</span><span class="params">(index: Union[int, str])</span>:</span></span><br><span class="line">response = <span class="keyword">await</span> session.get(<span class="string">'https://www.qkl123.com/sector/&#123;&#125;'</span>.format(index), headers=headers, proxies=proxies)</span><br><span class="line">    <span class="comment"># ...</span></span><br></pre></td></tr></table></figure><h2 id="request-html异步"><a class="markdownIt-Anchor" href="#request-html异步"></a> request-html异步</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> requests_html <span class="keyword">import</span> AsyncHTMLSession</span><br><span class="line">asession = AsyncHTMLSession()</span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">get_pyclock</span><span class="params">(index)</span>:</span></span><br><span class="line">    r = <span class="keyword">await</span> asession.get(<span class="string">'http://httpbin.org/get'</span>)</span><br><span class="line">    <span class="keyword">await</span> r.html.arender()</span><br><span class="line">    <span class="keyword">return</span> r</span><br><span class="line"></span><br><span class="line">results = asession.run(get_pyclock, get_pyclock,</span><br><span class="line">                       get_pyclock)  <span class="comment"># 这里作者将同一个页面使用异步方式进行了3次渲染，但是实际上使用的时间并不是平时的3倍！可能只是比平时渲染一个页面多花了一点时间而已！这就是异步的好处！</span></span><br><span class="line">print(results)</span><br></pre></td></tr></table></figure><p>and:<a href="https://cloud.tencent.com/developer/article/1575104" target="_blank" rel="noopener">https://cloud.tencent.com/developer/article/1575104</a></p><h3 id="asessionrun无法传参的问题"><a class="markdownIt-Anchor" href="#asessionrun无法传参的问题"></a> asession.run无法传参的问题</h3><p><a href="https://www.cnblogs.com/angelyan/p/13913926.html" target="_blank" rel="noopener">修改requests_html.AsyncHTMLSessions使得支持url参数</a></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;最近需要抓一个需要翻墙才能访问的网页的包，发现VPN直连时会导致 Fiddler 和 Charles 抓包工具无法正常进行抓包，网上找了以后发现了一些解决方案：&lt;a href=&quot;https://github.com/mopduan/team/issues/13&quot; targe
      
    
    </summary>
    
    
      <category term="爬虫" scheme="https://nymrli.top/tags/%E7%88%AC%E8%99%AB/"/>
    
  </entry>
  
  <entry>
    <title>总得学点Elasticsearch吧</title>
    <link href="https://nymrli.top/2022/04/11/%E6%80%BB%E5%BE%97%E5%AD%A6%E7%82%B9Elasticsearch%E5%90%A7/"/>
    <id>https://nymrli.top/2022/04/11/总得学点Elasticsearch吧/</id>
    <published>2022-04-11T14:18:04.000Z</published>
    <updated>2022-04-12T14:29:28.958Z</updated>
    
    <content type="html"><![CDATA[<h2 id="倒排索引"><a class="markdownIt-Anchor" href="#倒排索引"></a> 倒排索引</h2><p>反向索引（inverted index）更熟悉的名字是倒排索引。</p><p>什么是倒排索引: 倒排索引也叫反向索引，通俗来讲正向索引是通过key找value，反向索引则是通过value找key。</p><p>举个具体的例子：我们一般是通过<strong>标题</strong>来找<strong>文章内容</strong>，即通过目录（正向索引）；而反向索引的运用场景就是通过<strong>内容</strong>来找到是<strong>哪一篇文章</strong>（标题，作者……），而怎么建立反向索引呢？怎么更快更好地找到结果呢？这个就是<strong>搜索引擎</strong>做的事了、</p><p>正向索引：例如一个“文档1”经过分词，提取了20个关键词，每个关键词都会记录它在文档中的出现次数和出现位置。那我们就可以直接通过对&quot;文档1&quot;Document这个结构进行解析，从而得到每个关键词的信息。</p><p>反向索引：那么我们希望通过关键词来找到它在哪几篇“文档”中出现怎么办呢？那么其实在遍历文章的时候，如果当前文章出现了keyword，则对Keyword的结构（简单点可以想成是一个hashmap）添加上这篇文章的信息</p><p>再举个生活例子： 当用户在主页上搜索关键词“<u>华为手机</u>”时，假设只存在正向索引（forward index），那么就需要<strong>实时扫描完索引库中的所有文档</strong>，找出所有包含关键词“华为手机”的文档，再根据打分模型进行打分，排出名次后呈现给用户。因为互联网上收录在搜索引擎中的文档的数目是个天文数字，这样的索引结构根本无法满足<strong>实时</strong>返回排名结果的要求。</p><p>所以，<strong>搜索引擎</strong>会将正向索引<strong>重新构建</strong>为倒排索引，即把<strong>文件ID</strong>对应到<strong>关键词</strong>的映射转换为<strong>关键词到文件ID的映射</strong>，每个关键词都对应着一系列的文件，这些文件中都出现这个关键词。从而大大减少了遍历的大小，加快了检索的速度。</p><p><strong>本质上是通过设计数据结构，并花了额外的空间去换检索时的时间。</strong></p><p>论文查重就是这种的典型应用：<a href="https://www.bilibili.com/video/BV1iK4y1M7XR?spm_id_from=444.41.0.0" target="_blank" rel="noopener">B站视频——论文查重是如何实现的</a>——将段落通过标点符号分隔成一个个句子，再将句子每每7个分隔成一个个section，比如长度为9的句子则可以分成3个section，预处理创建<code>preDict={&quot;section1&quot;: [doc1, ], &quot;section2&quot;: [doc1, ], &quot;section&quot;3: [doc1, ]}</code>的倒排索引，当进行查重时，也是分隔成句子后每每7个字进行检查是否在preDict中，如果在则是知道了在哪些文章中出现（最简单版）。视频后又提出了增加LCS最长公共子串的增强，有效地检验出了添加“的”字or调整字位置的偷鸡方法——建立4字倒排，如果4个字已经出现在了别的文章A里，则把当前句子S1与文章A中的句子S2进行LCS，如果公共子串长度大于7则判定为重复。</p><h2 id="搜索引擎elasticsearch"><a class="markdownIt-Anchor" href="#搜索引擎elasticsearch"></a> 搜索引擎——Elasticsearch</h2><p>Elastic Stack 如果你没有听过，那么 ELK 一定听过。ELK 是三款软件的简称，分别是 <code>Elasticsearch</code>、<code>Logstash</code>、<code>Kibana</code> 组成。那什么是 Elasticsearch呢？——全文搜索是很多网站常见功能，比如 GitHub 站内搜索、JD 商品搜索、B 站视频搜索。<code>Elasticsearch</code> 是当今最火的<strong>搜索引擎</strong>之一，它的底层基于另外一个 <code>java</code> 开源搜索引擎 <code>Lucene</code>，是一款开源分布式搜索引擎，并且提供了一系列 <code>REST API</code> 操作接口。</p><h2 id="es的核心概念"><a class="markdownIt-Anchor" href="#es的核心概念"></a> ES的核心概念</h2><ol><li>索引（Index）：ES将数据存储于一个或多个索引中。类比传统的关系型数据库领域来说，索引相当于SQL中的一个数据库，或者一个数据存储方案（schema）。索引由其名称（必须为全小写字符）进行标识。一个ES集群中可以按需创建任意数目的索引。</li><li>类型（type）：类型是索引内部的逻辑分区（category/partition），一个索引内部可定义一个或多个类型（type）。类比传统的关系型数据库领域来说，类型相当于“表”。</li><li>文档（Document）：文档是索引和搜索的原子单位，它是包含了一个或多个域（Fild）的容器，每个域拥有一个名字及一个或多个值，有多个值的域通常称为“多值域”，文档基于JSON格式进行表示。每个文档可以存储不同的域集，但同一类型下的文档至应该有某种程度上的相似之处。</li><li>集群（Cluster）：一个或者多个拥有相同cluster…name配置的节点组成，它们共同承担数据和负载的压力。</li><li>节点（Node）：一个运行中的Elasticsearch实例称为一个节点。<br>ES集群中的节点有三种不同的类型：<ul><li>主节点：负责管理集群范围内的所有变更，例如增加、删除索引，或者增加、删除节点等。主节点并不需要涉及到文档级别的变更和搜索等操作。可以通过属性node.masteri进行设置。</li><li>数据节点：存储数据和其对应的倒排索引。默认每一个节点都是数据节点（包括主节点），可以通过ode.data属性进行设置。</li><li>协调节点：如果node.master和node.data属性均为false，则此节点称为协调节点，用来响应客户请求，均衡每个节点的负载。</li></ul></li></ol><h2 id="python操作elasticsearch"><a class="markdownIt-Anchor" href="#python操作elasticsearch"></a> Python操作Elasticsearch</h2><p>安装es库<code>pip install Elasticsearch</code></p><p>参考：</p><ul><li><a href="https://www.bilibili.com/video/BV1CJ411L7uV?from=search&amp;seid=16833887069453710612&amp;spm_id_from=333.337.0.0" target="_blank" rel="noopener"><em>python</em>操作<em>ElasticSearch</em>-创建、插入、检索</a></li><li><a href="https://www.bilibili.com/video/BV1Nk4y1R7Hf?p=5&amp;spm_id_from=pageDriver" target="_blank" rel="noopener">【狂神说Java】ElasticSearch搜索实战仿京东搜索</a>——Java操作ES+仿京东搜索+vue与thymeleft单页面实现前端</li></ul><h1 id="附录"><a class="markdownIt-Anchor" href="#附录"></a> 附录</h1><h2 id="docker-运行-elasticsearch"><a class="markdownIt-Anchor" href="#docker-运行-elasticsearch"></a> docker 运行 Elasticsearch</h2><blockquote><p>根据官网推荐，ES配套 Kibana 只能安装 <a href="https://link.juejin.cn/?target=https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Fkibana%2F6.8%2Fdocker.html" target="_blank" rel="noopener">6.8.19</a>， 所以安装</p></blockquote><ol><li><p>拉取镜像:<code>docker pull docker.io/elasticsearch:7.17.2</code>，拉取其他版本可见<a href="#%E6%9F%A5%E7%9C%8B%E9%95%9C%E5%83%8F%E7%89%88%E6%9C%AC%EF%BC%9Adocker-search-tag">查看镜像版本：docker-search-tag</a></p></li><li><p>es配置文件</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">mkdir -p /home/apollo3d/cl/dockerdata/elasticsearch/config </span><br><span class="line">mkdir -p /home/apollo3d/cl/dockerdata/elasticsearch/data</span><br><span class="line"><span class="built_in">echo</span> <span class="string">"http.host: 0.0.0.0"</span> &gt;&gt; /home/apollo3d/cl/dockerdata/elasticsearch/config/elasticsearch.yml</span><br></pre></td></tr></table></figure></li><li><p>启动容器:<code>docker run --name es -p 9200:9200 -p 9300:9300 \ -e &quot;discovery.type=single-node&quot; \ -e ES_JAVA_OPTS=&quot;-Xms64m -Xmx512m&quot; \ -v /home/apollo3d/dockerdata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \ -v /home/apollo3d/dockerdata/elasticsearch/data:/usr/share/elasticsearch/data \ -v /home/apollo3d/dockerdata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \ -d docker.io/elasticsearch:7.17.2</code>，注：<code>&lt;image_id&gt;</code>可以通过<code>docker images</code>查看刚刚pull下来的镜像；需要去ESC上打开防火墙的指定端口；</p></li><li><p>测试：浏览器访问 Elasticsearch 地址：<code>ECS_IP:9200</code>， 如果返回JSON数据则<strong>启动正确</strong></p></li><li><p>安装 ik 分词器</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 进入es容器</span></span><br><span class="line">$ docker <span class="built_in">exec</span> -it es bash</span><br><span class="line"><span class="comment"># 下载ik分词器</span></span><br><span class="line">$ ./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.1.1/elasticsearch-analysis-ik-7.1.1.zip</span><br><span class="line"><span class="comment"># 退出容器</span></span><br><span class="line">$ <span class="built_in">exit</span></span><br></pre></td></tr></table></figure></li><li><p>安装 ik 分词器后重启 Elasticsearch：<code>docker restart es</code></p></li><li><p>postman 测试 ik 分词器（注意：把 Content-Type 设置为 application/json ）</p></li></ol><h2 id="启动-es-head-可视化界面"><a class="markdownIt-Anchor" href="#启动-es-head-可视化界面"></a> 启动 es-head (可视化界面):</h2><blockquote><p>Elasticsearch 5.x后不支持插件，head作为单独服务独立运行。</p></blockquote><ol><li><p>拉取镜像：<code>docker pull mobz/elasticsearch-head:5</code></p></li><li><p>启动容器：<code>docker run -d -p 9100:9100 --name esh elasticsearch-head</code></p></li><li><p>访问 es-head：<code>ECS_IP:9100</code></p><p>当你输入 <a href="https://link.zhihu.com/?target=http%3A//192.168.232.128%3A9200/" target="_blank" rel="noopener">http://192.168.232.128:9200/</a> 点击连接时，会发现无法连接。是因为前后端分离开发，存在跨域问题，需要在服务端做 CORS 的配置。我们再次进入 Elasticsearch 容器内部，修改 <code>elasticsearch.yml</code> 配置。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 进入容器</span></span><br><span class="line">$ docker <span class="built_in">exec</span> -it es /bin/bash</span><br><span class="line">[root@7f213e9fb6bb elasticsearch]<span class="comment"># vi config/elasticsearch.yml</span></span><br><span class="line"><span class="comment"># 添加如下两条配置，注意冒号后面有空格，保存并退出。</span></span><br><span class="line">http.cors.enabled: <span class="literal">true</span> </span><br><span class="line">http.cors.allow-origin: <span class="string">"*"</span></span><br></pre></td></tr></table></figure></li><li><p>最后退出容器，并重启 Elasticsearch: <code>docker restart esh</code></p></li><li><p>再次访问 es-head，就能看到可视化的界面了。</p></li></ol><h2 id="启动-kibana可视化界面"><a class="markdownIt-Anchor" href="#启动-kibana可视化界面"></a> ★启动 kibana（可视化界面）</h2><blockquote><p>Kibana 是为 <a href="https://so.csdn.net/so/search?q=Elasticsearch&amp;spm=1001.2101.3001.7020" target="_blank" rel="noopener">Elasticsearch</a>设计的开源分析和可视化平台。可以使用 Kibana 来搜索，查看存储在 Elasticsearch 索引中的数据并与之交互。你可以很容易实现高级的数据分析和可视化，并且以<u>图表的形式</u>展现出来。</p><p>是ELK技术栈中的K，也是ES比较主流的可视化界面，也是官方推荐的</p><p>Elasticsearch 5.x后不支持插件，跟Elasticsearch  head一样，kibana需要作为单独服务独立运行。</p></blockquote><ol><li><p>拉取镜像：<code>docker pull docker.io/kibana:7.17.2</code></p></li><li><p>配置文件</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">mkdir -p ~/dockerdata/elk7/kibana/config/</span><br><span class="line">vi ~/dockerdata/elk7/kibana/config/kibana.yml</span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># ** THIS IS AN AUTO-GENERATED FILE **</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># Default Kibana configuration for docker target</span></span><br><span class="line"></span><br><span class="line">server.name: kibana</span><br><span class="line">server.host: <span class="string">"0"</span> <span class="comment"># 对外访问kibana的地址</span></span><br><span class="line">elasticsearch.hosts: [ <span class="string">"http://127.0.0.1:9200"</span> ]<span class="comment"># elasticsearch的地址</span></span><br><span class="line">xpack.monitoring.ui.container.elasticsearch.enabled: <span class="literal">true</span></span><br></pre></td></tr></table></figure></li><li><p>启动容器: <code>docker run -d --name=kibana --restart=always -p 5601:5601 -v ~/dockerdata/elk7/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml kibana:7.17.2</code></p></li><li><p>查看日志: <code>docker logs -f kibana</code></p></li><li><p>等待30秒，如果出现以下信息，说明启动成功了：<code>{&quot;type&quot;:&quot;log&quot;,&quot;@timestamp&quot;:&quot;2020-08-27T03:00:28Z&quot;,&quot;tags&quot;:[&quot;listening&quot;,&quot;info&quot;],&quot;pid&quot;:6,&quot;message&quot;:&quot;Server running at http://0:5601&quot;}</code></p></li><li><p>访问页面：<code>ECS_IP:5601</code>就可以看到Welcome to Kibana啦~</p></li></ol><p>Kibana server is not ready yet问题：</p><ul><li><a href="https://blog.csdn.net/fbvukn/article/details/107289027?spm=1001.2101.3001.6650.1&amp;utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-1.pc_relevant_antiscanv2&amp;depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-1.pc_relevant_antiscanv2&amp;utm_relevant_index=2" target="_blank" rel="noopener">kibana解决Kibana server is not ready yet问题</a></li><li><a href="https://blog.csdn.net/whatday/article/details/107879989" target="_blank" rel="noopener">kibana 报错 server is not ready yet 可能的原因</a></li></ul><h2 id="docker-compose"><a class="markdownIt-Anchor" href="#docker-compose"></a> docker-compose:</h2><figure class="highlight yml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">version:</span> <span class="string">'3'</span></span><br><span class="line">  </span><br><span class="line"><span class="attr">services:</span></span><br><span class="line"><span class="attr">    elasticsearch:</span></span><br><span class="line"><span class="attr">        image:</span> <span class="string">docker.io/elasticsearch:7.17.2</span></span><br><span class="line"><span class="attr">        container_name:</span> <span class="string">es2</span></span><br><span class="line"><span class="attr">        environment:</span></span><br><span class="line"><span class="bullet">            -</span> <span class="string">discovery.type=single-node</span></span><br><span class="line"><span class="attr">        ports:</span></span><br><span class="line"><span class="bullet">           -</span> <span class="string">"9200:9200"</span></span><br><span class="line"><span class="bullet">           -</span> <span class="string">"9300:9300"</span></span><br><span class="line"><span class="attr">    kinaba:</span></span><br><span class="line"><span class="attr">        image:</span> <span class="string">docker.io/kibana:7.17.2</span></span><br><span class="line"><span class="attr">        container_name:</span> <span class="string">kibana2</span></span><br><span class="line"><span class="attr">        ports:</span></span><br><span class="line"><span class="bullet">            -</span> <span class="string">"5601:5601"</span></span><br><span class="line"><span class="attr">        volumes:</span></span><br><span class="line"><span class="bullet">            -</span> <span class="string">/home/apollo3d/cl/dockerdata/elk7/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml</span></span><br><span class="line"><span class="attr">        depends_on:</span></span><br><span class="line"><span class="bullet">            -</span> <span class="string">elasticsearch</span></span><br></pre></td></tr></table></figure><h2 id="查看镜像版本docker-search-tag"><a class="markdownIt-Anchor" href="#查看镜像版本docker-search-tag"></a> 查看镜像版本：docker-search-tag</h2><p><a href="http://xn--docker-show-repo-tag-2x06a431biq1j2e4clo2cqoe.sh" target="_blank" rel="noopener">使用自制脚本docker-show-repo-tag.sh</a></p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/bin/sh</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># Simple script that will display docker repository tags.</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># Usage:</span></span><br><span class="line"><span class="comment">#   $ docker-show-repo-tags.sh ubuntu centos</span></span><br><span class="line"><span class="keyword">for</span> Repo <span class="keyword">in</span> $* ; <span class="keyword">do</span></span><br><span class="line">  curl -s -S <span class="string">"https://registry.hub.docker.com/v2/repositories/library/<span class="variable">$Repo</span>/tags/"</span> | \</span><br><span class="line">    sed -e <span class="string">'s/,/,\n/g'</span> -e <span class="string">'s/\[/\[\n/g'</span> | \</span><br><span class="line">    grep <span class="string">'"name"'</span> | \</span><br><span class="line">    awk -F\<span class="string">" '&#123;print <span class="variable">$4</span>;&#125;' | \</span></span><br><span class="line"><span class="string">    sort -fu | \</span></span><br><span class="line"><span class="string">    sed -e "</span>s/^/<span class="variable">$&#123;Repo&#125;</span>:/<span class="string">"</span></span><br><span class="line"><span class="string">done</span></span><br></pre></td></tr></table></figure><p>然后在终端运行<code>$ ./docker-show-repo-tags.sh elasticsearch</code>即可获得结果</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line">elasticsearch:8.0.1</span><br><span class="line">elasticsearch:8.1.0</span><br><span class="line">elasticsearch:8.1.1</span><br><span class="line">elasticsearch:8.1.2</span><br></pre></td></tr></table></figure>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h2 id=&quot;倒排索引&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#倒排索引&quot;&gt;&lt;/a&gt; 倒排索引&lt;/h2&gt;
&lt;p&gt;反向索引（inverted index）更熟悉的名字是倒排索引。&lt;/p&gt;
&lt;p&gt;什么是倒排索引: 倒排索引也叫反向索引，通俗来讲正向
      
    
    </summary>
    
    
  </entry>
  
  <entry>
    <title>算法与数据结构——滑动窗口、尺取法</title>
    <link href="https://nymrli.top/2022/04/10/%E7%AE%97%E6%B3%95%E4%B8%8E%E6%95%B0%E6%8D%AE%E7%BB%93%E6%9E%84%E2%80%94%E2%80%94%E6%BB%91%E5%8A%A8%E7%AA%97%E5%8F%A3/"/>
    <id>https://nymrli.top/2022/04/10/算法与数据结构——滑动窗口/</id>
    <published>2022-04-10T03:14:53.000Z</published>
    <updated>2022-04-11T08:35:58.656Z</updated>
    
    <content type="html"><![CDATA[<blockquote><p>Sliding window algorithm is used to perform required operation on specific window size of given large buffer or array.滑动窗口算法是在给定<strong>特定窗口大小</strong>的<u>数组或字符串</u>上执行要求的操作。</p><p>This technique shows how a nested for loop in few problems can be converted to single for loop and hence reducing the time complexity.该技术可以将一部分问题中的嵌套循环转变为一个单循环，因此它可以减少时间复杂度。</p></blockquote><h2 id="基本概念"><a class="markdownIt-Anchor" href="#基本概念"></a> 基本概念</h2><p>滑动窗口是一种基于双指针的一种思想，两个指针指向的元素之间形成一个窗口，可以看做是一种双指针方法的特例。</p><p>窗口的做法分为定长窗口和不定长窗口，有些人把涉及窗口移动的都统称为<strong>滑动窗口</strong>做法。而在此，我们进行进一步的细分：</p><ul><li>定长滑动窗口</li><li>变长滑动窗口——尺取法</li></ul><p><strong>应用</strong>：</p><blockquote><p>利用窗口的特性，可以用来解决一些查找满足一定条件的<strong>连续区间的</strong>性质（长度等）的问题。由于区间连续，因此当区间发生变化时，可以通过<u>旧有的计算结果</u>进行修改，从而对搜索空间进行剪枝，这样便减少了重复计算，降低了时间复杂度。</p></blockquote><p>Q：那么，具体什么情况可以用滑动窗口来解决实际问题呢？</p><ol><li>一般给出的数据结构是<u>数组或者字符串</u></li><li>求取某个子串或者<u>子序列</u><u>最长最短等最值问题</u>或者求某个目标值时</li><li>该问题<u>本身可以通过暴力求解</u>——★滑动窗口跟双指针一样，针对一些问题能降低时间复杂度。</li></ol><h2 id="核心思路"><a class="markdownIt-Anchor" href="#核心思路"></a> <strong>核心思路</strong></h2><p><strong>窗口的形成</strong></p><p>在具体使用之前，我们知道窗口实际是两个指针之间形成的区域，那关键就是这两个指针是如何移动的。</p><p>《挑战程序设计竞赛》这本书中把滑动窗口叫做「虫取法」，这也非常生动形象。因为滑动窗口的两个指针移动的过程和虫子爬动的过程非常像：前脚(right指针)不动，把后脚(left指针)移动过来；后脚不动，把前脚向前移动。</p><p>滑动窗口中用到了左右两个指针，它们移动的思路是：<strong>以右指针作为驱动，拖着左指针向前走。<u>右指针每次只移动一步，而左指针在内部 while 循环中每次可能移动多步</u>。右指针是主动前移，探索未知的新区域；左指针是被迫移动，负责寻找满足题意的区间</strong>。</p><p>为了得到符合要求的最长子数组的长度，应遵循以下两点原则：</p><ul><li>当 start 的值固定时，end 的值应尽可能大；</li><li>当 end 的值固定时, start 的值应尽可能小。</li></ul><p><strong>模板的执行思想是：</strong></p><ol><li>定义两个指针 left 和 right 分别指向区间的开头和结尾，注意是<strong>闭区间</strong>；定义 sums 用来统计该区间内的各个字符出现次数；</li><li>第一重 while 循环是为了判断 right 指针的位置是否超出了数组边界；当 right 每次到了新位置，需要增加 right 指针的求和/计数 --&gt; sum；</li><li>第二重 while 循环是让 left 指针向右移动到 [left, right] 区间符合题意的位置；当 left 每次移动到了新位置，需要减少 left 指针的求和/计数；</li><li>在第二重 while 循环之后，成功找到了一个符合题意的 [left, right] 区间，便跳出循环，更新题目要求最大的区间长度，即 <code>res = max(res, right - left + 1)</code>。</li><li>right 指针每次向右移动一步，开始探索新的区间。</li></ol><h2 id="代码模板"><a class="markdownIt-Anchor" href="#代码模板"></a> 代码模板</h2><h3 id="变长窗口模板"><a class="markdownIt-Anchor" href="#变长窗口模板"></a> 变长窗口模板</h3><p>伪代码</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 滑动窗口模板</span></span><br><span class="line">left,right = <span class="number">0</span>, (<span class="number">0</span> <span class="keyword">or</span> <span class="number">1</span>)</span><br><span class="line">ans = target = <span class="number">0</span></span><br><span class="line"><span class="keyword">while</span> right &lt; len(nums):</span><br><span class="line">   根据 right 更新 target 值</span><br><span class="line">   <span class="keyword">while</span> 窗口内数据不满足要求</span><br><span class="line">      <span class="comment"># 1. 更新 target 值</span></span><br><span class="line">      <span class="comment"># 2. 收缩左边界 ==&gt; 使得新窗口满足要求</span></span><br><span class="line">      <span class="comment"># or: 更新 ans： 窗口相关最小值，则在此更新</span></span><br><span class="line">    <span class="comment"># or: 更新 ans： 求窗口相关最大值，则在此更新</span></span><br><span class="line">    更新right(移动右边界)</span><br><span class="line">返回 ans</span><br></pre></td></tr></table></figure><p>具象化的Python代码</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">findSubArray</span><span class="params">(nums)</span>:</span></span><br><span class="line">    N = len(nums) <span class="comment"># 数组/字符串长度</span></span><br><span class="line">    left, right = <span class="number">0</span>, <span class="number">0</span> <span class="comment"># 双指针，表示当前遍历的区间[left, right]，闭区间</span></span><br><span class="line">    sums = <span class="number">0</span> <span class="comment"># 用于统计 子数组/子区间 是否有效，根据题目可能会改成求和/计数</span></span><br><span class="line">    res = <span class="number">0</span> <span class="comment"># 保存最大的满足题目要求的 子数组/子串 长度</span></span><br><span class="line">    <span class="keyword">while</span> right &lt; N: <span class="comment"># 当右边的指针没有搜索到 数组/字符串 的结尾</span></span><br><span class="line">        sums += nums[right] <span class="comment"># 增加当前右边指针的数字/字符的求和/计数</span></span><br><span class="line">        <span class="keyword">while</span> 区间[left, right]不符合题意：<span class="comment"># 此时需要一直移动左指针，直至找到一个符合题意的区间</span></span><br><span class="line">            sums -= nums[left] <span class="comment"># 移动左指针前需要从counter中减少left位置字符的求和/计数</span></span><br><span class="line">            left += <span class="number">1</span> <span class="comment"># 真正的移动左指针，注意顺序不能跟上面一行代码写反</span></span><br><span class="line">        <span class="comment"># 到 while 结束时，我们找到了一个符合题意要求的 子数组/子串</span></span><br><span class="line">        res = max(res, right - left + <span class="number">1</span>) <span class="comment"># 需要更新结果</span></span><br><span class="line">        right += <span class="number">1</span> <span class="comment"># 移动右指针，去探索新的区间</span></span><br><span class="line">    <span class="keyword">return</span> res</span><br></pre></td></tr></table></figure><p>另一种模板：</p><blockquote><p>(right++在前)</p></blockquote><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">string s, t;</span><br><span class="line">    <span class="comment">// 在 s 中寻找 t 的「最小覆盖子串」</span></span><br><span class="line">    <span class="keyword">int</span> left = <span class="number">0</span>, right = <span class="number">0</span>;</span><br><span class="line">    string res = s;</span><br><span class="line">    </span><br><span class="line">    <span class="keyword">while</span>(right &lt; s.size()) &#123;</span><br><span class="line">        window.add(s[right]);</span><br><span class="line">        right++;</span><br><span class="line">        <span class="comment">// 如果符合要求，说明窗口构造完成，移动 left 缩小窗口</span></span><br><span class="line">        <span class="keyword">while</span> (window 符合要求) &#123;</span><br><span class="line">            <span class="comment">// 如果这个窗口的子串更短，则更新 res</span></span><br><span class="line">            <span class="comment">// or: ans更新位置</span></span><br><span class="line">            res = minLen(res, window);</span><br><span class="line">            window.remove(s[left]);</span><br><span class="line">            left++;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="comment">// or: ans更新位置</span></span><br><span class="line">    &#125;</span><br><span class="line">    <span class="keyword">return</span> res;</span><br></pre></td></tr></table></figure><h3 id="固定长窗口模板"><a class="markdownIt-Anchor" href="#固定长窗口模板"></a> <a href="https://www.cnblogs.com/huansky/p/13488234.html" target="_blank" rel="noopener">固定长窗口模板：</a></h3><blockquote><p>right更新在第一层while最后，且while中没有<code>if (right &gt; k)</code>类似的代码</p></blockquote><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">maxVowels</span><span class="params">(String s, <span class="keyword">int</span> k)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">int</span> right = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> len = s.length();</span><br><span class="line">        <span class="keyword">int</span> maxAns = Integer.MIN_VALUE;</span><br><span class="line">        <span class="keyword">int</span> target = <span class="number">0</span>;</span><br><span class="line"><span class="comment">// 初始构造定长窗口</span></span><br><span class="line">        <span class="keyword">while</span> (right &lt; k) &#123;</span><br><span class="line">            <span class="comment">// 更新这段初始化窗口中的 target 值</span></span><br><span class="line">        &#125;</span><br><span class="line"><span class="comment">// ★更新 ans 值 maxAns = Math.max(maxAns, );</span></span><br><span class="line">        </span><br><span class="line">        <span class="comment">// 让 right (有边界)</span></span><br><span class="line">        <span class="keyword">while</span> (right &lt; len) &#123;</span><br><span class="line">            <span class="comment">// 更新 target 值</span></span><br><span class="line">            <span class="comment">// 更新 ans 值 maxAns = Math.max(maxAns, );</span></span><br><span class="line">            right ++;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> maxAns;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">isVowel</span><span class="params">(<span class="keyword">char</span> ch)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">return</span> ch == <span class="string">'a'</span> || ch == <span class="string">'e'</span> || ch == <span class="string">'i'</span> || ch == <span class="string">'o'</span> || ch == <span class="string">'u'</span> ? <span class="number">1</span> : <span class="number">0</span>;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>另一种模板：</p><blockquote><ul><li>(right++在前) --&gt; 这个是可以调整的，相应地修改<code>if</code></li><li>有<code>if right &gt; k</code>的判断，即窗口是否已经构造完成</li></ul></blockquote><figure class="highlight c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 固定窗口大小为 k</span></span><br><span class="line">   <span class="built_in">string</span> s;</span><br><span class="line">   <span class="comment">// 在 s 中寻找窗口大小为 k 时的所包含最大元音字母个数</span></span><br><span class="line">   <span class="keyword">int</span>  right = <span class="number">0</span>;</span><br><span class="line">   <span class="keyword">while</span>(right &lt; s.size()) &#123;</span><br><span class="line">       window.add(s[right]);</span><br><span class="line">       <span class="comment">// 注：如果放最后，则下面的if改成 right &gt; k</span></span><br><span class="line">       right++;</span><br><span class="line">       <span class="comment">// 如果符合要求，说明窗口构造完成，</span></span><br><span class="line">       <span class="keyword">if</span> (right&gt;=k) &#123;</span><br><span class="line">           <span class="comment">// 这是已经是一个窗口了，根据条件做一些事情</span></span><br><span class="line">          <span class="comment">// ... 可以计算窗口最大值等 </span></span><br><span class="line">           <span class="comment">// 最后不要忘记把 right - k 位置元素从窗口里面移除</span></span><br><span class="line">       &#125;</span><br><span class="line">   &#125;</span><br><span class="line">   <span class="keyword">return</span> res;</span><br></pre></td></tr></table></figure><p>可以发现此时不需要依赖 left 指针了。因为窗口固定所以其实就没必要使用left。可以直接通过 right 指针来控制窗口。</p><p>由于窗口是固定的，因此可以轻易获取到 left 的位置，此处 left = right - k，所以在第二层while中可以通过<code>nums[right-k]</code>来获得left位置得值，从而从窗口中删除。</p><p>注： 虽然提供了两种模板，但不要贪多，跟二分模板一样，只要记住一种就行了。</p><h2 id="easy-point"><a class="markdownIt-Anchor" href="#easy-point"></a> easy point:</h2><ul><li>窗口的长度: <code>right - left + 1</code>，</li><li>假设窗口长度为k，则窗口内容为<code>[right-k+1, right-k+2, ..., right]</code>，要被剔除的<code>left</code>的索引为<code>right-k</code>：固定长窗口的题目中，可以通过right位置找到left位置。</li></ul><h2 id="一起做几题"><a class="markdownIt-Anchor" href="#一起做几题"></a> 一起做几题</h2><ul><li><a href="https://leetcode-cn.com/problems/maximum-number-of-vowels-in-a-substring-of-given-length/" target="_blank" rel="noopener">1456. 定长子串中元音的最大数目</a>——固定长窗口</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">maxVowels</span><span class="params">(String s, <span class="keyword">int</span> k)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">int</span> left = <span class="number">0</span>, right = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> len = s.length();</span><br><span class="line">        <span class="keyword">int</span> maxAns = Integer.MIN_VALUE;</span><br><span class="line">        <span class="keyword">int</span> ans = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">char</span>[] chars = &#123;<span class="string">'a'</span>, <span class="string">'e'</span>, <span class="string">'i'</span>, <span class="string">'u'</span>, <span class="string">'o'</span>&#125;;</span><br><span class="line">        <span class="keyword">while</span> (right - left + <span class="number">1</span> &lt;= k) &#123;</span><br><span class="line">            <span class="keyword">if</span> (contains(chars, s.charAt(right))) &#123;</span><br><span class="line">                ans++;</span><br><span class="line">            &#125;</span><br><span class="line">            right++;</span><br><span class="line">        &#125;</span><br><span class="line">        maxAns = Math.max(maxAns, ans);</span><br><span class="line"></span><br><span class="line">        <span class="keyword">while</span> (right &lt; len) &#123;</span><br><span class="line">            <span class="keyword">if</span> (contains(chars, s.charAt(right))) &#123;</span><br><span class="line">                ans += <span class="number">1</span>;</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="comment">// 被排除的是元音</span></span><br><span class="line">            <span class="keyword">if</span> (contains(chars, s.charAt(right - k))) &#123;</span><br><span class="line">                ans -= <span class="number">1</span>;</span><br><span class="line">            &#125;</span><br><span class="line">            maxAns = Math.max(maxAns, ans);</span><br><span class="line">            right++;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">return</span> maxAns;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">private</span> <span class="keyword">boolean</span> <span class="title">contains</span><span class="params">(<span class="keyword">char</span> chars[], <span class="keyword">char</span> ch)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; chars.length; i++) &#123;</span><br><span class="line">            <span class="keyword">if</span> (chars[i] == ch) &#123;</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">true</span>;</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">false</span>;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">// 更好的做法:</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">maxVowels</span><span class="params">(String s, <span class="keyword">int</span> k)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">int</span> right =<span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> sum = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> max = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">while</span> (right &lt; s.length()) &#123;</span><br><span class="line">            sum += isYuan(s.charAt(right)) ;</span><br><span class="line">            right++;</span><br><span class="line">            <span class="keyword">if</span> (right &gt;=k) &#123;</span><br><span class="line">                max = Math.max(max, sum);</span><br><span class="line">                sum -= isYuan(s.charAt(right-k));</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> max;</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">isYuan</span><span class="params">(<span class="keyword">char</span> s)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">return</span> s==<span class="string">'a'</span> || s==<span class="string">'e'</span> ||s==<span class="string">'i'</span> ||s==<span class="string">'o'</span> ||s==<span class="string">'u'</span> ? <span class="number">1</span>:<span class="number">0</span>;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><ul><li><a href="https://leetcode-cn.com/problems/longest-substring-without-repeating-characters/" target="_blank" rel="noopener"> #3 无重复字符的最长子串</a> ——用set来模拟窗口</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">   </span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">lengthOfLongestSubstring</span><span class="params">(String s)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">int</span> len = s.length();</span><br><span class="line">        <span class="keyword">if</span> (len &lt;= <span class="number">1</span>) &#123;</span><br><span class="line">            <span class="keyword">return</span> len;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">int</span> ans = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> left = <span class="number">0</span>, right = <span class="number">0</span>;</span><br><span class="line">        HashSet&lt;Character&gt; set = <span class="keyword">new</span> HashSet&lt;&gt;();</span><br><span class="line">        <span class="keyword">while</span> (right &lt; len) &#123;</span><br><span class="line">            <span class="comment">// 把删除之前s.charAt(right)位置内所有的字符</span></span><br><span class="line">            <span class="keyword">while</span> (set.contains(s.charAt(right))) &#123;</span><br><span class="line">                set.remove(s.charAt(left++));</span><br><span class="line">            &#125;</span><br><span class="line">            set.add(s.charAt(right));</span><br><span class="line">            ans = Math.max(ans, right - left + <span class="number">1</span>);</span><br><span class="line">            right++;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> ans;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><ul><li><a href="https://leetcode-cn.com/problems/minimum-size-subarray-sum/" target="_blank" rel="noopener"> #209 长度最小的子数组</a> —— 变长窗口板子</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">     <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">minSubArrayLen</span><span class="params">(<span class="keyword">int</span> target, <span class="keyword">int</span>[] nums)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">int</span> left = <span class="number">0</span>, right = <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">int</span> minLen = Integer.MAX_VALUE;</span><br><span class="line">        <span class="keyword">int</span> len = nums.length;</span><br><span class="line">        <span class="keyword">int</span> sum = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">while</span> ( right &lt; len) &#123;</span><br><span class="line">            sum += nums[right];</span><br><span class="line">            <span class="keyword">while</span> (sum &gt;= target) &#123;</span><br><span class="line">                <span class="comment">// 由于需要求最小, 因此这边在left收缩过程中更新minVal</span></span><br><span class="line">                minLen = Math.min(minLen, right - left +<span class="number">1</span>);</span><br><span class="line">                sum -= nums[left];</span><br><span class="line">                left ++;</span><br><span class="line">            &#125;</span><br><span class="line">            right ++;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> minLen == Integer.MAX_VALUE ? <span class="number">0</span> : minLen;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><ul><li><a href="https://leetcode-cn.com/problems/max-consecutive-ones-iii/" target="_blank" rel="noopener"> #1004 最大连续1的个数 III</a> —— 变长窗口板子略有变形</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">longestOnes</span><span class="params">(<span class="keyword">int</span>[] nums, <span class="keyword">int</span> k)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">int</span> ans = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> left= <span class="number">0</span>, right = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> len = nums.length;</span><br><span class="line">        <span class="keyword">int</span> zeroTimes = <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">while</span> (right &lt; len) &#123;</span><br><span class="line">            <span class="keyword">int</span> now = nums[right];</span><br><span class="line">            <span class="comment">// 把当前修改成1</span></span><br><span class="line">            <span class="keyword">if</span> (now == <span class="number">0</span>) &#123;</span><br><span class="line">                zeroTimes ++;</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="comment">// 如果欠了k，则需要从左边进行挪动补偿；保证下一次添加nums[right]时 k一定非负</span></span><br><span class="line">            <span class="keyword">while</span>( zeroTimes &gt; k) &#123;</span><br><span class="line">                <span class="keyword">if</span> (nums[left] == <span class="number">0</span>) &#123;</span><br><span class="line">                    zeroTimes--;</span><br><span class="line">                &#125;</span><br><span class="line">                left ++;</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="comment">// 获得最大长度</span></span><br><span class="line">            ans = Math.max(ans, right - left + <span class="number">1</span>);</span><br><span class="line">            right ++;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">return</span> ans;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><ul><li><a href="https://leetcode-cn.com/problems/maximum-average-subarray-i/" target="_blank" rel="noopener"> #643 子数组最大平均数 I</a>——固定长窗口</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">double</span> <span class="title">findMaxAverage</span><span class="params">(<span class="keyword">int</span>[] nums, <span class="keyword">int</span> k)</span> </span>&#123;</span><br><span class="line">      <span class="keyword">int</span> i = <span class="number">0</span>;</span><br><span class="line">      <span class="keyword">int</span> sum = <span class="number">0</span>;</span><br><span class="line">      <span class="keyword">while</span> (i &lt; k) &#123;</span><br><span class="line">          sum += nums[i++];</span><br><span class="line">      &#125;</span><br><span class="line">      <span class="keyword">double</span> maxV = -<span class="number">10000</span>;</span><br><span class="line">      maxV = Math.max(maxV, (<span class="keyword">double</span>) sum / k);</span><br><span class="line">      <span class="keyword">for</span> (; i &lt; nums.length; i++) &#123;</span><br><span class="line">          sum = sum - nums[i - k] + nums[i];</span><br><span class="line">          maxV = Math.max(maxV, (<span class="keyword">double</span>) sum / k);</span><br><span class="line">      &#125;</span><br><span class="line">      <span class="keyword">return</span> maxV;</span><br><span class="line">  &#125;</span><br></pre></td></tr></table></figure><ul><li><a href="https://leetcode-cn.com/problems/contains-duplicate-ii/" target="_blank" rel="noopener"> #219 存在重复元素 II</a>——使用set作为窗口</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">containsNearbyDuplicate</span><span class="params">(<span class="keyword">int</span>[] nums, <span class="keyword">int</span> k)</span> </span>&#123;</span><br><span class="line">        HashSet&lt;Integer&gt; set = <span class="keyword">new</span> HashSet&lt;&gt;();</span><br><span class="line">        <span class="keyword">int</span> right = <span class="number">0</span>;</span><br><span class="line">        </span><br><span class="line">        <span class="comment">// 初始化构造窗口</span></span><br><span class="line">        <span class="keyword">while</span> (set.size() &lt; k) &#123;</span><br><span class="line">            set.add(nums[right++]);</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">while</span>(right &lt; nums.length)</span><br><span class="line">            <span class="keyword">if</span> (set.contains(nums[right])) &#123;</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">true</span>;</span><br><span class="line">            &#125;</span><br><span class="line">            set.remove(nums[right-k]);</span><br><span class="line">            set.add(nums[right]);</span><br><span class="line">            right++;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">false</span>;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><ul><li><a href="https://leetcode-cn.com/problems/find-all-anagrams-in-a-string/" target="_blank" rel="noopener">438. 找到字符串中所有字母异位词</a>——固定长窗口</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">  <span class="function"><span class="keyword">public</span> List&lt;Integer&gt; <span class="title">findAnagrams</span><span class="params">(String s, String p)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">int</span> len = s.length();</span><br><span class="line">        <span class="keyword">int</span> right = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> plen = p.length();</span><br><span class="line">        ArrayList&lt;Integer&gt; list = <span class="keyword">new</span> ArrayList&lt;&gt;();</span><br><span class="line"><span class="comment">// 保证p比s短</span></span><br><span class="line">        <span class="keyword">if</span> (plen &gt; len) &#123;</span><br><span class="line">            <span class="keyword">return</span> list;</span><br><span class="line">        &#125;</span><br><span class="line">      </span><br><span class="line">        <span class="keyword">int</span>[] mp = <span class="keyword">new</span> <span class="keyword">int</span>[<span class="number">26</span>];</span><br><span class="line">        <span class="keyword">int</span>[] targetmp = <span class="keyword">new</span> <span class="keyword">int</span>[<span class="number">26</span>];</span><br><span class="line">        <span class="keyword">int</span> mpSize = <span class="number">0</span>;</span><br><span class="line"><span class="comment">// 记录p中各个字符出现的次数</span></span><br><span class="line">        <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; plen; i++) &#123;</span><br><span class="line">            targetmp[ p.charAt(i) - <span class="string">'a'</span>] ++;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">while</span> (right &lt; len) &#123;</span><br><span class="line">            <span class="keyword">char</span> c = s.charAt(right);</span><br><span class="line">            mp[c-<span class="string">'a'</span>]++; mpSize++;</span><br><span class="line">            <span class="comment">// while和if效果是一样的，因为mp大小是通过right一次次增大的</span></span><br><span class="line">            <span class="keyword">if</span> (mpSize &gt; plen) &#123;</span><br><span class="line">                <span class="comment">// 缩小左边界</span></span><br><span class="line">                <span class="keyword">char</span> removeChar = s.charAt(right - plen);</span><br><span class="line">                mpSize--;</span><br><span class="line">                mp[removeChar-<span class="string">'a'</span>]--;</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="keyword">if</span> (right(mp, targetmp)) &#123;</span><br><span class="line">                list.add(right - plen + <span class="number">1</span>);</span><br><span class="line">            &#125;</span><br><span class="line">            right ++;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">return</span> list;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">private</span> <span class="keyword">boolean</span> <span class="title">right</span><span class="params">(<span class="keyword">int</span>[] chars1, <span class="keyword">int</span>[] chars2)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; chars1.length; i++) &#123;</span><br><span class="line">            <span class="keyword">if</span> (chars1[i] != chars2[i]) &#123;</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">false</span>;</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">true</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><ul><li><a href="https://leetcode-cn.com/problems/get-equal-substrings-within-budget/" target="_blank" rel="noopener">1208. 尽可能使字符串相等</a>——变长窗口板子</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">     <span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">equalSubstring</span><span class="params">(String s, String t, <span class="keyword">int</span> maxCost)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">int</span> len = s.length();</span><br><span class="line">        <span class="keyword">int</span> left = <span class="number">0</span>, right = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> ans = <span class="number">0</span>;</span><br><span class="line">        <span class="comment">// 现在已经花费的cost</span></span><br><span class="line">        <span class="keyword">int</span> cost = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">while</span> (right &lt; len) &#123;</span><br><span class="line">            cost += Math.abs(s.charAt(right) - t.charAt(right));</span><br><span class="line">            <span class="comment">// 此时cost不满足条件</span></span><br><span class="line">            <span class="keyword">while</span> (cost &gt; maxCost) &#123;</span><br><span class="line">                <span class="comment">// 通过移动left, 将cost补偿回来</span></span><br><span class="line">                cost -= Math.abs(s.charAt(left) - t.charAt(left));</span><br><span class="line">                left++;</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="comment">// 求最大长度，则更新在第一层while里面</span></span><br><span class="line">            ans = Math.max(ans, right - left + <span class="number">1</span>);</span><br><span class="line">            right ++;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> ans;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><ul><li><a href="https://leetcode-cn.com/problems/sliding-window-maximum/" target="_blank" rel="noopener">239. 滑动窗口最大值(困难)</a></li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">     <span class="keyword">public</span> <span class="keyword">int</span>[] maxSlidingWindow(<span class="keyword">int</span>[] nums, <span class="keyword">int</span> k) &#123;</span><br><span class="line">        <span class="keyword">int</span> len = nums.length;</span><br><span class="line">        <span class="keyword">int</span>[] ans = <span class="keyword">new</span> <span class="keyword">int</span>[len - k + <span class="number">1</span>];</span><br><span class="line">        <span class="keyword">int</span> right = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> idx = <span class="number">0</span>;</span><br><span class="line">        LinkedList&lt;Integer&gt; list = <span class="keyword">new</span> LinkedList&lt;&gt;();</span><br><span class="line"></span><br><span class="line">        <span class="keyword">while</span> (right &lt; len) &#123;</span><br><span class="line">            <span class="keyword">int</span> now = nums[right];</span><br><span class="line">            <span class="comment">// 保持list递减</span></span><br><span class="line">            <span class="keyword">while</span> (!list.isEmpty() &amp;&amp; list.peekLast() &lt; now) &#123;</span><br><span class="line">                list.pollLast();</span><br><span class="line">            &#125;</span><br><span class="line">            list.addLast(now);</span><br><span class="line">            right++;</span><br><span class="line">            <span class="keyword">if</span> (right &gt;= k) &#123;</span><br><span class="line">                ans[idx++] = list.peekFirst();</span><br><span class="line">                <span class="comment">// 下一次的时候将不会再在窗口里面， 所以判断是否出局</span></span><br><span class="line">                <span class="keyword">if</span> (list.peekFirst() == nums[right - k]) &#123;</span><br><span class="line">                    list.removeFirst();</span><br><span class="line">                &#125;</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> ans;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><ul><li><a href="https://leetcode-cn.com/problems/permutation-in-string/" target="_blank" rel="noopener">567. 字符串的排列</a>——固定长度窗口</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 一开始都没看出来怎么需要用到窗口， 后来发现s1的排列必须是s2的子串, 所以字符是相同的, 子串长度也是固定。 所以是道定长窗口题</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span> </span>&#123;</span><br><span class="line">  <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">checkInclusion</span><span class="params">(String s1, String s2)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">int</span>[] arr1 = <span class="keyword">new</span> <span class="keyword">int</span>[<span class="number">26</span>];</span><br><span class="line">        <span class="keyword">int</span>[] arr2 = <span class="keyword">new</span> <span class="keyword">int</span>[<span class="number">26</span>];</span><br><span class="line">        <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; s1.length(); i++) &#123;</span><br><span class="line">            arr1[s1.charAt(i) - <span class="string">'a'</span>]++;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">int</span> k = s1.length(), right = <span class="number">0</span>;</span><br><span class="line">        <span class="keyword">int</span> len = s2.length();</span><br><span class="line">        <span class="keyword">int</span> cnt = <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">while</span> (right &lt; len) &#123;</span><br><span class="line">            arr2[s2.charAt(right) - <span class="string">'a'</span>]++;</span><br><span class="line">            cnt ++;</span><br><span class="line"></span><br><span class="line">            <span class="keyword">while</span> (cnt &gt; k) &#123;</span><br><span class="line">                arr2[ s2.charAt(right - k) - <span class="string">'a'</span>]--;</span><br><span class="line">                cnt--;</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="keyword">if</span> (contains(arr1, arr2)) &#123;</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">true</span>;</span><br><span class="line">            &#125;</span><br><span class="line">            right++;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">false</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">boolean</span> <span class="title">contains</span><span class="params">(<span class="keyword">int</span>[] arr1, <span class="keyword">int</span>[] arr2)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; arr1.length; i++) &#123;</span><br><span class="line">            <span class="comment">// 一旦arr2不符合</span></span><br><span class="line">            <span class="keyword">if</span> (arr1[i] &gt; arr2[i]) &#123;</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">false</span>;</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">true</span>;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h2 id="参考文章"><a class="markdownIt-Anchor" href="#参考文章"></a> 参考文章：</h2><ul><li><a href="https://leetcode-cn.com/problems/max-consecutive-ones-iii/solution/fen-xiang-hua-dong-chuang-kou-mo-ban-mia-f76z/" target="_blank" rel="noopener">分享滑动窗口模板，秒杀滑动窗口问题</a>——形象的理解：虫取法</li><li><a href="https://www.cnblogs.com/huansky/p/13488234.html" target="_blank" rel="noopener">滑动窗口算法基本原理与实践</a>——例题+模板</li><li><a href="https://blog.csdn.net/Dby_freedom/article/details/89066140" target="_blank" rel="noopener">算法与数据结构（一）：滑动窗口法总结</a>——概念+例题</li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;blockquote&gt;
&lt;p&gt;Sliding window algorithm is used to perform required operation on specific window size of given large buffer or array.滑动窗口算法
      
    
    </summary>
    
    
      <category term="数据结构" scheme="https://nymrli.top/tags/%E6%95%B0%E6%8D%AE%E7%BB%93%E6%9E%84/"/>
    
  </entry>
  
  <entry>
    <title>[翻译]设计Pastebin.com(or Bit.ly)系统</title>
    <link href="https://nymrli.top/2022/04/05/%E7%BF%BB%E8%AF%91-%E8%AE%BE%E8%AE%A1Pastebin-com-or-Bit-ly-%E7%B3%BB%E7%BB%9F/"/>
    <id>https://nymrli.top/2022/04/05/翻译-设计Pastebin-com-or-Bit-ly-系统/</id>
    <published>2022-04-05T09:34:50.000Z</published>
    <updated>2022-04-05T11:30:43.887Z</updated>
    
    <content type="html"><![CDATA[<h1 id="设计pastebincom"><a class="markdownIt-Anchor" href="#设计pastebincom"></a> <a href="http://xn--Pastebin-xx5xji.com" target="_blank" rel="noopener">设计Pastebin.com</a> (or <a href="http://Bit.ly" target="_blank" rel="noopener">Bit.ly</a>)系统</h1><p>注：本文档直接链接到系统设计主题中的相关领域，以避免重复。请参考链接内容来了解总体的讨论要点、利弊权衡以及替代方案。</p><p>**设计<a href="https://www.baidu.com/s?wd=bit.ly&amp;rsv_spt=1&amp;rsv_iqid=0x97c01cce002338c1&amp;issp=1&amp;f=8&amp;rsv_bp=1&amp;rsv_idx=2&amp;ie=utf-8&amp;rqlang=cn&amp;tn=baiduhome_pg&amp;rsv_enter=0&amp;rsv_dl=tb&amp;oq=bit.ly&amp;rsv_btype=t&amp;rsv_t=e8762eu5%2B6DVVq%2BNlFhLqorUp3QSNtSJ5UjqS%2FLmbVRjAw3gHUzdZ5VW%2F%2BJORNxD3z1%2F&amp;rsv_sug3=7&amp;rsv_sug1=7&amp;rsv_sug7=100&amp;rsv_pq=ce550397001e64ee&amp;prefixsug=bit.ly&amp;rsp=1&amp;rsv_sug4=5598" target="_blank" rel="noopener">Bit.ly</a>**是一个类似的问题。只是Pastebin还具有一个额外的功能，即存储粘贴的内容，而不是原始的未缩短过的url。</p><h2 id="第一步列出用例和约束的大纲"><a class="markdownIt-Anchor" href="#第一步列出用例和约束的大纲"></a> 第一步：列出用例和约束的大纲</h2><blockquote><p>确定应用场景：收集需求和范围问题；提问来弄清阐明用例和约束，讨论来作出假设。</p></blockquote><p>在没有面试官来阐明清这些问题的情况下，我们将定义一些用例和约束。</p><h3 id="用例"><a class="markdownIt-Anchor" href="#用例"></a> 用例</h3><h4 id="我们将确定问题的范围只处理以下使用情况"><a class="markdownIt-Anchor" href="#我们将确定问题的范围只处理以下使用情况"></a> 我们将确定问题的范围，只处理以下使用情况</h4><ul><li><strong>用户</strong>：输入一段文本后，获得一个随机生成的链接<ul><li>过期<ul><li>默认设置是不会过期的</li><li>可以选择一个时间过期</li></ul></li></ul></li><li><strong>用户</strong>：输入一个paste的url链接，便可以看到内容</li><li><strong>用户</strong>：是匿名的</li><li><strong>服务</strong>：可以追踪得到页面分析结果<ul><li>每月的访问数据</li></ul></li><li><strong>服务</strong>：删除过期的pasten内容</li><li><strong>服务</strong>：具有高可用性</li></ul><h4 id="范围之外"><a class="markdownIt-Anchor" href="#范围之外"></a> 范围之外</h4><ul><li><strong>用户</strong>：注册一个帐户<ul><li><strong>用户</strong>：认证邮箱</li></ul></li><li><strong>用户</strong>：登录注册账户<ul><li><strong>用户</strong>：编辑文档</li></ul></li><li><strong>用户</strong>：可以设置可见性</li><li><strong>用户</strong>：可以设置短链接</li></ul><h3 id="约束和假设"><a class="markdownIt-Anchor" href="#约束和假设"></a> 约束和假设</h3><h4 id="情况假设"><a class="markdownIt-Anchor" href="#情况假设"></a> 情况假设</h4><ul><li>流量不是均匀分布的</li><li>产生一个短链接应该是快速的</li><li>粘贴paste内容只能是文本</li><li>页面浏览分析不需要是实时的</li><li>1000万用户</li><li>每月1000万次的粘贴paste写入量</li><li>每月1亿次的粘贴paste阅读</li><li>10:1的读与写比例</li></ul><h3 id="计算用量"><a class="markdownIt-Anchor" href="#计算用量"></a> 计算用量</h3><p><strong>向你的面试官说明你是否应该对用量进行粗略估计。</strong></p><ul><li>每次粘贴paste的大小<ul><li>每次粘贴的内容为1KB</li><li><code>shortlink</code>- 7 bytes</li><li><code>expiration_length_in_minutes</code> - 4 bytes</li><li><code>created_at</code> - 5 bytes</li><li><code>paste_path</code> - 255 bytes</li><li><code>total</code> = ~1.27 KB</li></ul></li><li>每月12.7GB的新粘贴内容<ul><li>每个1.27 KB 的粘贴paste * 每月1000万个paste</li><li>3年内约有450GB的新粘贴内容</li><li>3年内有3.6亿个短链接</li><li>假设大多数是新的粘贴内容，而不是对现有内容的更新</li></ul></li><li>平均每秒钟写4次粘贴内容</li><li>平均每秒40次读取请求</li></ul><p>方便的转换指南：</p><ul><li>每月250万秒</li><li>每秒1个请求=每月250万个请求</li><li>每秒40个请求=每月1亿个请求</li><li>每秒400个请求=每月10亿个请求</li></ul><h2 id="第二步-创建一个高水平的设计"><a class="markdownIt-Anchor" href="#第二步-创建一个高水平的设计"></a> 第二步: 创建一个高水平的设计</h2><blockquote><p>勾勒出一个包含所有重要组成部分的高层次设计。</p></blockquote><p><img src="http://i.imgur.com/BKsBnmG.png" alt="Imgur"></p><h2 id="第三步设计核心组件"><a class="markdownIt-Anchor" href="#第三步设计核心组件"></a> 第三步：设计核心组件</h2><p><strong>用例: 用户输入一段文本后，获得一个随机生成的链接</strong></p><p>我们可以使用一个<a href="https://github.com/donnemartin/system-design-primer#relational-database-management-system-rdbms" target="_blank" rel="noopener">关系数据库</a>作为一个大的哈希表，将生成的url映射到文件服务器和包含粘贴paste文件的路径。</p><p>我们不采用管理文件服务器的方式，而是使用一个可管理的<strong>对象存储</strong>，如Amazon S3或<a href="https://github.com/donnemartin/system-design-primer#document-store" target="_blank" rel="noopener">NoSQL文档存储</a>。</p><p>作为一个大型哈希表的关系数据库的替代品，我们可以使用<a href="https://github.com/donnemartin/system-design-primer#key-value-store" target="_blank" rel="noopener">NoSQL键值存储</a>。我们应该讨论<a href="https://github.com/donnemartin/system-design-primer#sql-or-nosql" target="_blank" rel="noopener">选择SQL或NoSQL之间的利弊权衡</a>。下面的讨论使用了关系型数据库的方法。</p><ul><li><strong>客户端</strong>向作为反向代理的<strong>web服务器</strong>发送一个创建粘贴paste请求</li><li><strong>web服务器</strong>将该请求转发给 <strong>Write API 服务器</strong></li><li><strong>Write API 服务器</strong>做以下工作。<ul><li>生成一个唯一的url<ul><li>通过查看<strong>SQL数据库</strong>是否有重复的网址，来检查该网址是否是唯一的。</li><li>如果url不是唯一的，它将生成另一个网址</li><li>如果我们支持自定义url，我们可以使用用户提供的url（也检查是否有重复的）。</li></ul></li><li>保存到<strong>SQL数据库</strong>中的<code>粘贴paste</code>表</li><li>将粘贴的数据保存到<strong>对象存储</strong>中</li><li>返回url</li></ul></li></ul><p><strong>与你的面试官明确说明你要写多少代码</strong></p><p><code>pastes</code>表的结构可以设计成下面的样子。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">shortlink char(7) NOT NULL</span><br><span class="line">expiration_length_in_minutes int NOT NULL</span><br><span class="line">created_at datetime NOT NULL</span><br><span class="line">paste_path varchar(255) NOT NULL</span><br><span class="line">PRIMARY KEY(shortlink)</span><br></pre></td></tr></table></figure><p>将主键设置为基于<code>shortlink</code>列的<a href="https://github.com/donnemartin/system-design-primer#use-good-indices" target="_blank" rel="noopener">索引</a>，数据库会使用这个索引来强制执行唯一性。我们将在<code>creative_at</code>上创建一个额外的索引，以加快查询速度（用日志时间代替扫描整个表），并将数据保存在内存中。从内存中连续读取1MB的数据需要250微秒，而从SSD中读取需要4倍，从磁盘中读取需要80倍的时间。<sup><a href="https://github.com/donnemartin/system-design-primer#latency-numbers-every-programmer-should-know" target="_blank" rel="noopener">1</a></sup></p><p>为了生成唯一的url，我们可以:</p><ul><li>对用户的<code>ip_address</code> + 时间戳进行<a href="https://en.wikipedia.org/wiki/MD5" target="_blank" rel="noopener"><strong>MD5</strong></a> 哈希<ul><li>MD5是一个广泛使用的散列函数，产生一个128位的哈希值</li><li>MD5是均匀分布的</li><li>另外，我们也可以取随机生成的数据的MD5哈希值</li></ul></li><li>使用<a href="https://www.kerstner.at/2012/07/shortening-strings-using-base-62-encoding/" target="_blank" rel="noopener"><strong>Base62</strong></a>对MD5哈希结果值进行编码<ul><li>Base 62 编码字符为<code>[a-zA-Z0-9]</code>，不需要转义特殊字符，这对Urls来说很有效。</li><li>原始输入只有一个哈希结果，而且Base 62是确定性的（不涉及随机性）。</li><li>Base 64 是另一种流行的编码，但由于有额外的<code>+</code>和<code>/</code>字符，所以对urls有问题。</li><li>下面的<a href="http://stackoverflow.com/questions/742013/how-to-code-a-url-shortener" target="_blank" rel="noopener">Base 62伪代码</a>在O(k)时间内运行，其中k=7。</li></ul></li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">base_encode</span><span class="params">(num, base=<span class="number">62</span>)</span>:</span></span><br><span class="line">    digits = []</span><br><span class="line">    <span class="keyword">while</span> num &gt; <span class="number">0</span></span><br><span class="line">      remainder = modulo(num, base)</span><br><span class="line">      digits.push(remainder)</span><br><span class="line">      num = divide(num, base)</span><br><span class="line">    digits = digits.reverse</span><br></pre></td></tr></table></figure><ul><li>取输出的前7个字符，结果是62^7个可能的值，应该足以处理我们3年内3.6亿个短链接的约束。</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">url = base_encode(md5(ip_address+timestamp))[:URL_LENGTH]</span><br></pre></td></tr></table></figure><p>我们将使用一个公共<a href="https://github.com/donnemartin/system-design-primer#representational-state-transfer-rest" target="_blank" rel="noopener"><strong>REST API</strong></a>。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ curl -X POST --data <span class="string">'&#123; "expiration_length_in_minutes": "60", \</span></span><br><span class="line"><span class="string">    "paste_contents": "Hello World!" &#125;'</span> https://pastebin.com/api/v1/paste</span><br></pre></td></tr></table></figure><p>Response:</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">&#123;</span><br><span class="line">    <span class="attr">"shortlink"</span>: <span class="string">"foobar"</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>对于内部通信，我们可以使用<a href="https://github.com/donnemartin/system-design-primer#remote-procedure-call-rpc" target="_blank" rel="noopener">远程过程调用</a>。</p><h3 id="用例用户输入一个paste-url并查看其内容"><a class="markdownIt-Anchor" href="#用例用户输入一个paste-url并查看其内容"></a> 用例：用户输入一个paste url并查看其内容</h3><ul><li><strong>客户端</strong>向<strong>web服务器</strong>发送一个获取粘贴paste的请求</li><li><strong>Web服务器</strong>将该请求转发给 <strong>Read API 服务器</strong></li><li><strong>Read API服务器</strong>做以下工作。<ul><li>检查<strong>SQL数据库</strong>中生成的url<ul><li>如果url在<strong>SQL数据库</strong>中，则从<strong>对象存储</strong>中获取粘贴内容</li><li>否则，给用户返回一个错误信息</li></ul></li></ul></li></ul><p>REST API:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ curl https://pastebin.com/api/v1/paste?shortlink=foobar</span><br></pre></td></tr></table></figure><p>Response:</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">&#123;</span><br><span class="line">    <span class="attr">"paste_contents"</span>: <span class="string">"Hello World"</span></span><br><span class="line">    <span class="string">"created_at"</span>: <span class="string">"YYYY-MM-DD HH:MM:SS"</span></span><br><span class="line">    <span class="string">"expiration_length_in_minutes"</span>: <span class="string">"60"</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h3 id="用例网页跟踪分析服务"><a class="markdownIt-Anchor" href="#用例网页跟踪分析服务"></a> 用例：网页跟踪分析服务</h3><p>由于实时分析不是一个要求，我们可以简单地对<strong>Web服务器</strong>的日志进行 <strong>MapReduce</strong> 以生成点击数。</p><p><strong>与你的面试官明确说明你要写多少代码</strong>。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">HitCounts</span><span class="params">(MRJob)</span>:</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">extract_url</span><span class="params">(self, line)</span>:</span></span><br><span class="line">        <span class="string">"""Extract the generated url from the log line."""</span></span><br><span class="line">        ...</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">extract_year_month</span><span class="params">(self, line)</span>:</span></span><br><span class="line">        <span class="string">"""Return the year and month portions of the timestamp."""</span></span><br><span class="line">        ...</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">mapper</span><span class="params">(self, _, line)</span>:</span></span><br><span class="line">        <span class="string">"""Parse each log line, extract and transform relevant lines.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">        Emit key value pairs of the form:</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">        (2016-01, url0), 1</span></span><br><span class="line"><span class="string">        (2016-01, url0), 1</span></span><br><span class="line"><span class="string">        (2016-01, url1), 1</span></span><br><span class="line"><span class="string">        """</span></span><br><span class="line">        url = self.extract_url(line)</span><br><span class="line">        period = self.extract_year_month(line)</span><br><span class="line">        <span class="keyword">yield</span> (period, url), <span class="number">1</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">reducer</span><span class="params">(self, key, values)</span>:</span></span><br><span class="line">        <span class="string">"""Sum values for each key.</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">        (2016-01, url0), 2</span></span><br><span class="line"><span class="string">        (2016-01, url1), 1</span></span><br><span class="line"><span class="string">        """</span></span><br><span class="line">        <span class="keyword">yield</span> key, sum(values)</span><br></pre></td></tr></table></figure><h3 id="用例删除过期paste的服务"><a class="markdownIt-Anchor" href="#用例删除过期paste的服务"></a> 用例：删除过期paste的服务</h3><p>要删除过期的粘贴，我们可以直接扫描<strong>SQL数据库</strong>中所有过期时间戳大于当前时间戳的条目。然后，所有过期的条目将被从表中删除（或标记为过期）。</p><h2 id="第四步扩展设计"><a class="markdownIt-Anchor" href="#第四步扩展设计"></a> 第四步：扩展设计</h2><blockquote><p>鉴于约束条件，识别并解决瓶颈问题。</p></blockquote><p><img src="http://i.imgur.com/4edXG0T.png" alt="Imgur"></p><p><strong>重要：不要简单地从最初的设计直接跳到最终的设计中去!</strong></p><p>说明你会迭代地做这件事。1) <strong>基准测试/负载测试</strong>；2) <strong>剖析</strong>瓶颈；3) 解决瓶颈问题，同时评估替代方案和利弊权衡；4) 重复。参看<a href="../scaling_aws/README.md">设计一个可以在AWS上扩展到数百万用户的系统</a>，作为一个如何对初始设计进行迭代扩展的样例。</p><p>讨论初始设计可能遇到的瓶颈以及如何解决每个瓶颈是很重要的。例如，通过添加一个带有多个<strong>Web服务器</strong>的<strong>负载平衡器</strong>、<strong>CDN</strong>、<strong>主从复制</strong>，可以解决哪些问题？各自的替代方案和<strong>利弊权衡</strong>是什么？</p><p>我们将介绍一些组件来完成设计并解决可扩展性问题。内部负载均衡器没有显示，以减少混乱。</p><p><em>为了避免重复讨论</em>，请参考下面的<a href="https://github.com/donnemartin/system-design-primer#index-of-system-design-topics" target="_blank" rel="noopener">系统设计主题</a>来了解主要的论文要点、利弊权衡和替代方案。</p><ul><li><a href="https://github.com/donnemartin/system-design-primer#domain-name-system" target="_blank" rel="noopener">DNS</a></li><li><a href="https://github.com/donnemartin/system-design-primer#content-delivery-network" target="_blank" rel="noopener">CDN</a></li><li><a href="https://github.com/donnemartin/system-design-primer#load-balancer" target="_blank" rel="noopener">Load balancer</a></li><li><a href="https://github.com/donnemartin/system-design-primer#horizontal-scaling" target="_blank" rel="noopener">Horizontal scaling</a></li><li><a href="https://github.com/donnemartin/system-design-primer#reverse-proxy-web-server" target="_blank" rel="noopener">Web server (reverse proxy)</a></li><li><a href="https://github.com/donnemartin/system-design-primer#application-layer" target="_blank" rel="noopener">API server (application layer)</a></li><li><a href="https://github.com/donnemartin/system-design-primer#cache" target="_blank" rel="noopener">Cache</a></li><li><a href="https://github.com/donnemartin/system-design-primer#relational-database-management-system-rdbms" target="_blank" rel="noopener">Relational database management system (RDBMS)</a></li><li><a href="https://github.com/donnemartin/system-design-primer#fail-over" target="_blank" rel="noopener">SQL write master-slave failover</a></li><li><a href="https://github.com/donnemartin/system-design-primer#master-slave-replication" target="_blank" rel="noopener">Master-slave replication</a></li><li><a href="https://github.com/donnemartin/system-design-primer#consistency-patterns" target="_blank" rel="noopener">Consistency patterns</a></li><li><a href="https://github.com/donnemartin/system-design-primer#availability-patterns" target="_blank" rel="noopener">Availability patterns</a></li></ul><p><strong>数据库分析</strong>可以使用数据仓库解决方案，如亚马逊 Redshift 或谷歌 BigQuery。</p><p>像亚马逊S3这样的<strong>对象存储</strong>可以从容地处理每月12.7 GB 的新内容的约束。</p><p>为了解决每秒40次的平均读取请求（高峰时更高），热门内容的流量应该由<strong>内存缓存</strong>而不是数据库来处理。<strong>内存缓存</strong>对于处理分布不均的流量和流量高峰也很有用。<strong>SQL Read 多副本</strong>应该能够处理缓存的缺失，只要副本没有被副本写入所拖累。</p><p>对于单个<strong>SQL Write 主从</strong>来说，每秒<em>平均</em>4次粘贴写（高峰时更多）应该是可以做到的。否则，我们就需要采用额外的SQL扩展模式。</p><ul><li><a href="https://github.com/donnemartin/system-design-primer#federation" target="_blank" rel="noopener">Federation</a></li><li><a href="https://github.com/donnemartin/system-design-primer#sharding" target="_blank" rel="noopener">Sharding</a></li><li><a href="https://github.com/donnemartin/system-design-primer#denormalization" target="_blank" rel="noopener">Denormalization</a></li><li><a href="https://github.com/donnemartin/system-design-primer#sql-tuning" target="_blank" rel="noopener">SQL Tuning</a></li></ul><p>我们还应该考虑将一些数据转移到<strong>NoSQL数据库</strong>。</p><h2 id="额外的讨论要点"><a class="markdownIt-Anchor" href="#额外的讨论要点"></a> 额外的讨论要点</h2><blockquote><p>根据问题的范围和剩余时间，可以深入研究其他的话题。</p></blockquote><h4 id="nosql"><a class="markdownIt-Anchor" href="#nosql"></a> NoSQL</h4><ul><li><a href="https://github.com/donnemartin/system-design-primer#key-value-store" target="_blank" rel="noopener">Key-value store</a></li><li><a href="https://github.com/donnemartin/system-design-primer#document-store" target="_blank" rel="noopener">Document store</a></li><li><a href="https://github.com/donnemartin/system-design-primer#wide-column-store" target="_blank" rel="noopener">Wide column store</a></li><li><a href="https://github.com/donnemartin/system-design-primer#graph-database" target="_blank" rel="noopener">Graph database</a></li><li><a href="https://github.com/donnemartin/system-design-primer#sql-or-nosql" target="_blank" rel="noopener">SQL vs NoSQL</a></li></ul><h3 id="缓存"><a class="markdownIt-Anchor" href="#缓存"></a> 缓存</h3><ul><li>缓存到哪里？<ul><li><a href="https://github.com/donnemartin/system-design-primer#client-caching" target="_blank" rel="noopener">Client caching</a></li><li><a href="https://github.com/donnemartin/system-design-primer#cdn-caching" target="_blank" rel="noopener">CDN caching</a></li><li><a href="https://github.com/donnemartin/system-design-primer#web-server-caching" target="_blank" rel="noopener">Web server caching</a></li><li><a href="https://github.com/donnemartin/system-design-primer#database-caching" target="_blank" rel="noopener">Database caching</a></li><li><a href="https://github.com/donnemartin/system-design-primer#application-caching" target="_blank" rel="noopener">Application caching</a></li></ul></li><li>缓存什么？<ul><li><a href="https://github.com/donnemartin/system-design-primer#caching-at-the-database-query-level" target="_blank" rel="noopener">Caching at the database query level</a></li><li><a href="https://github.com/donnemartin/system-design-primer#caching-at-the-object-level" target="_blank" rel="noopener">Caching at the object level</a></li></ul></li><li>何时更新缓存？<ul><li><a href="https://github.com/donnemartin/system-design-primer#cache-aside" target="_blank" rel="noopener">Cache-aside</a></li><li><a href="https://github.com/donnemartin/system-design-primer#write-through" target="_blank" rel="noopener">Write-through</a></li><li><a href="https://github.com/donnemartin/system-design-primer#write-behind-write-back" target="_blank" rel="noopener">Write-behind (write-back)</a></li><li><a href="https://github.com/donnemartin/system-design-primer#refresh-ahead" target="_blank" rel="noopener">Refresh ahead</a></li></ul></li></ul><h3 id="异步性和微服务"><a class="markdownIt-Anchor" href="#异步性和微服务"></a> 异步性和微服务</h3><ul><li><a href="https://github.com/donnemartin/system-design-primer#message-queues" target="_blank" rel="noopener">Message queues</a></li><li><a href="https://github.com/donnemartin/system-design-primer#task-queues" target="_blank" rel="noopener">Task queues</a></li><li><a href="https://github.com/donnemartin/system-design-primer#back-pressure" target="_blank" rel="noopener">Back pressure</a></li><li><a href="https://github.com/donnemartin/system-design-primer#microservices" target="_blank" rel="noopener">Microservices</a></li></ul><h3 id="通信"><a class="markdownIt-Anchor" href="#通信"></a> 通信</h3><ul><li>对利弊权衡进行讨论：<ul><li>与客户的外部通信 - <a href="https://github.com/donnemartin/system-design-primer#representational-state-transfer-rest" target="_blank" rel="noopener">HTTP APIs following REST</a><ul><li>内部通信 - <a href="https://github.com/donnemartin/system-design-primer#remote-procedure-call-rpc" target="_blank" rel="noopener">RPC</a></li></ul></li></ul></li><li><a href="https://github.com/donnemartin/system-design-primer#service-discovery" target="_blank" rel="noopener">服务发现</a></li></ul><h3 id="安全"><a class="markdownIt-Anchor" href="#安全"></a> 安全</h3><p>见 <a href="https://github.com/donnemartin/system-design-primer#security" target="_blank" rel="noopener">security section</a>.</p><h3 id="延迟数"><a class="markdownIt-Anchor" href="#延迟数"></a> 延迟数</h3><p>见 <a href="https://github.com/donnemartin/system-design-primer#latency-numbers-every-programmer-should-know" target="_blank" rel="noopener">每个程序员应该知道的延迟数字</a>。</p><h3 id="持续推进"><a class="markdownIt-Anchor" href="#持续推进"></a> 持续推进</h3><ul><li>继续对你的系统进行基准测试和监测，以解决出现的瓶颈问题</li><li>扩展是一个反复的过程</li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h1 id=&quot;设计pastebincom&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#设计pastebincom&quot;&gt;&lt;/a&gt; &lt;a href=&quot;http://xn--Pastebin-xx5xji.com&quot; target=&quot;_blank&quot; rel=
      
    
    </summary>
    
    
      <category term="翻译" scheme="https://nymrli.top/tags/%E7%BF%BB%E8%AF%91/"/>
    
      <category term="系统设计" scheme="https://nymrli.top/tags/%E7%B3%BB%E7%BB%9F%E8%AE%BE%E8%AE%A1/"/>
    
  </entry>
  
  <entry>
    <title>理了一天彻底弄懂元类——分享给你一起弄懂</title>
    <link href="https://nymrli.top/2022/04/05/%E7%90%86%E4%BA%86%E4%B8%80%E5%A4%A9%E5%BD%BB%E5%BA%95%E5%BC%84%E6%87%82%E5%85%83%E7%B1%BB%E2%80%94%E2%80%94%E5%88%86%E4%BA%AB%E7%BB%99%E4%BD%A0%E4%B8%80%E8%B5%B7%E5%BC%84%E6%87%82/"/>
    <id>https://nymrli.top/2022/04/05/理了一天彻底弄懂元类——分享给你一起弄懂/</id>
    <published>2022-04-05T05:26:39.000Z</published>
    <updated>2022-04-05T05:42:12.847Z</updated>
    
    <content type="html"><![CDATA[<blockquote><p>这是一个在看开源代码时配到了单例元类写法后的一个原理贴，可以帮你彻底弄清元类、以及<code>__new__</code>和<code>__call__</code>，相信我，看完你绝对会有收获。</p><p>首先，关于元类的结论说在前头，先有个印象——<strong>小总结</strong>：</p><ul><li>现在我们能知道为什么元类必须继承type了：因为我们实例化对象<code>Foo(xxx)</code>时调用了<code>type.__call__</code>,而<code>type.__call__</code>又会调用<code>type.__new__</code>因此如果type子类重写实现了<code>__new__</code>（返回的类实例对象的类型作控制）、<code>__call__</code>（对实例化的流程做控制），则可以对<strong>类对象的类型和类属性</strong>起到自定义的功能，而重写就必须继承type=&gt;需要元类必须继承type</li><li>所以按照上述的逻辑，如果定义了一个元类让自定义类用的话<code>class Foo(metaclass=MyMetaClass)</code>，在其实例化过程中<code>Foo()</code>会直接调用重写后的<code>MyMetaClass.__call__</code>，而只要记住在<code>MyMetaClass.__call__</code>中使用到<code>return super(Singleton, cls).__call__(*args, **kwargs)</code>就可以把<code>type.__call__</code>生成的实例返回啦。所以这也是为什么<u>编写元类，一般都是继承了type，然后根据想控制实例化流程就重写<code>__call__</code>方法，想添加属性就重写<code>__new__</code>方法就行了。</u></li><li><strong>★元类产生影响的时间点是在实例化的时候</strong></li></ul></blockquote><p>看开源代码时，看到了下面一段代码，于是对withMetaclass产生了好奇，经过了解发现其作用是six对python2和python3使用元类兼容的写法。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Python2和3兼容使用元类写法</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span><span class="params">(withMetaclass<span class="params">(Singleton)</span>)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">pass</span></span><br></pre></td></tr></table></figure><p>因此，上述代码在Python3中相当于</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Python3元类使用写法</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span><span class="params">(metaclass=Singleton)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">pass</span></span><br></pre></td></tr></table></figure><p>那么，问题来了，withMetaclass到底是怎么实现兼容的呢？下面是其实现代码</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">withMetaclass</span><span class="params">(meta, *bases)</span>:</span></span><br><span class="line">    <span class="class"><span class="keyword">class</span> <span class="title">MetaClass</span><span class="params">(meta)</span>:</span></span><br><span class="line">        <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, this_bases, d)</span>:</span></span><br><span class="line">            <span class="keyword">return</span> meta(name, bases, d)</span><br><span class="line">    <span class="keyword">return</span> type.__new__(MetaClass, <span class="string">'temporary_class'</span>, (), &#123;&#125;)</span><br></pre></td></tr></table></figure><p>可以看到其中出现了不少我们很少看到的使用方法。接下来我们就仔细的学习上述写法为什么可以成功。</p><p>元类使用可以参考：<a href="https://zhuanlan.zhihu.com/p/49158035" target="_blank" rel="noopener">Python3 元类（metaclass）</a></p><h3 id="预置知识type和object"><a class="markdownIt-Anchor" href="#预置知识type和object"></a> 预置知识：type和object</h3><blockquote><p>object 和 type的关系很像鸡和蛋的关系，先有object还是先有type没法说，obejct和type是共生的关系，必须同时出现的。</p><p>记住一点：<strong>在Python里面，所有的东西都是对象的概念</strong>，即包括类（类是type的实例对象）</p></blockquote><p>最重要的两点</p><ul><li>object类是所有类的超类（也是type类的父类）</li><li>type是所有类的类（类型，所有类都是type的实例对象，object<u>类型</u>也是type的实例对象；type 创建的对象拥有创建对象的能力(也就是类)）–&gt;是所有类的元类</li></ul><p>此外：</p><ul><li>type是所有<strong>元类</strong>的父亲。我们可以通过<strong>继承type来创建元类</strong>（通过重写<code>type.__new__</code>和<code>type.__call__</code>来拦截自定义类的创建过程）。</li><li>object是所有类的父亲。</li><li>实例是对象关系链的末端，不能再被子类化和实例化。</li></ul><p>了解到这些关键的点后，我们继续看代码中出现的一些内容：</p><h3 id="__new__"><a class="markdownIt-Anchor" href="#__new__"></a> <code>__new__</code></h3><blockquote><p><code>__new__()</code> 是一种负责<strong>创建类实例</strong>的<strong>静态方法</strong>，它无需使用 staticmethod 装饰器修饰，且该方法会优先 <code>__init__()</code> 初始化方法被调用。</p></blockquote><p><code>__new__()</code>通常会返回该类的一个实例，但有时也可能会返回其他类的实例，其<code>super().__new__(cls)</code>中会调用<code>object.__init__</code>来<code>Create and return a new object.</code>，因此我们可以通过改写子类的<code>__new__</code>可以添加一些逻辑来控制实例的产生，然后再通过<code>super().__new__(cls)</code>来生成一个instance并返回。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">demoClass</span>:</span></span><br><span class="line">    instances_created = <span class="number">0</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="comment"># __new__(): &lt;class '__main__.demoClass'&gt; ('abc',) &#123;&#125;</span></span><br><span class="line">        print(<span class="string">"__new__():"</span>, cls, args, kwargs)</span><br><span class="line">        <span class="comment"># 1. 通过父类__new__生成一个实例: 调用父类object.__new__生成实例(Create and return a new object.) </span></span><br><span class="line">        instance = super().__new__(cls)</span><br><span class="line">        <span class="comment"># 2. 自己重写要实现的逻辑</span></span><br><span class="line">        instance.number = cls.instances_created</span><br><span class="line">        cls.instances_created += <span class="number">1</span></span><br><span class="line">        <span class="comment"># 3. 将父类生成的实例返回</span></span><br><span class="line">        <span class="keyword">return</span> instance</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, attribute)</span>:</span></span><br><span class="line">        <span class="comment"># __init__(): &lt;__main__.demoClass object at 0x00000185A6466EB0&gt; abc</span></span><br><span class="line">        print(<span class="string">"__init__():"</span>, self, attribute)</span><br><span class="line">        self.attribute = attribute</span><br><span class="line"></span><br><span class="line">test1 = demoClass(<span class="string">"abc"</span>)</span><br><span class="line">test2 = demoClass(<span class="string">"xyz"</span>)</span><br><span class="line">print(test1.number, test1.instances_created)</span><br><span class="line">print(test2.number, test2.instances_created)</span><br></pre></td></tr></table></figure><p>Q:什么情况下重写类的<code>__new__()</code>呢？答案很简单，在<code>__init__()</code>不够用的时候。</p><p><code>__new__()</code>通常会返回该类的一个实例，但有时也可能会返回其他类的实例，如果发生了这种情况，则会跳过对 <code>__init__()</code>方法的调用。而在某些情况下（比如需要修改<strong>不可变类实例</strong>（<a href="http://c.biancheng.net/python/" target="_blank" rel="noopener">Python</a> 的某些内置类型）的创建行为），利用这一点会事半功倍。比如：<a href="http://c.biancheng.net/view/5484.html%EF%BC%8C%E5%AF%B9" target="_blank" rel="noopener">http://c.biancheng.net/view/5484.html，对</a> Python 不可变的内置类型（如 int、str、float 等）进行了子类化，这是因为一旦创建了这样不可变的对象实例，就无法在<code>__init__()</code>方法中对其进行修改。</p><p>注：由于 <code>__new__()</code> 不限于返回同一个类的实例，所以很容易被滥用，不负责任地使用这种方法可能会对代码有害，所以要谨慎使用。</p><h3 id="metaclass元类"><a class="markdownIt-Anchor" href="#metaclass元类"></a> <a href="http://c.biancheng.net/view/2293.html" target="_blank" rel="noopener">MetaClass元类</a></h3><blockquote><p>承接上文<code>__new__</code>，Python中大量使用<code>__new__()</code>方法且合理的地方，就是 MetaClass 元类。</p><p>MetaClass元类，并不是某一个类的名字，它是一个概念，是一种Python的思想。当然其本质也是一个类，但和普通类的用法不同，它可以对<strong>类内部的定义</strong>（包括类属性和类方法）进行<strong>动态的修改</strong>。可以这么说，使用元类的主要目的就是为了实现在创建类时，能够动态地改变类中定义的属性或者方法。其可以将<strong>创建对象的过程拦截</strong>下来，从而对这个对象进行自定义（这个需要<strong>类继承type</strong>，与前文继承object的做区别）。</p><p>明确一点：元类可以理解成是自定义类继承的父类（从兼容写法中也能看出），但元类的特点是<strong>不会出现在自定义类的继承关系(<code>__mro__</code>)之中</strong></p></blockquote><p>举个例子，根据实际场景的需要，我们要为多个类添加一个 name 属性和一个 say() 方法。显然有多种方法可以实现，但其中一种方法就是使用 MetaClass 元类。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 定义一个元类，继承type。因为只有继承type才能通过重写__new__来拦截创建过程</span></span><br><span class="line"><span class="comment"># ▲注意，继承type后__new__能拿到的参数信息跟不继承type的有天壤之别，原因继续看下去</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">FirstMetaClass</span><span class="params">(type)</span>:</span></span><br><span class="line">    <span class="comment"># cls代表元类的类: FirstMetaClass</span></span><br><span class="line">    <span class="comment"># name代表自定义类的类名: CLanguage</span></span><br><span class="line">    <span class="comment"># bases代表被动态修改的类的所有父类</span></span><br><span class="line">    <span class="comment"># attr代表被动态修改的类的所有属性、方法组成的字典</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, bases, attrs)</span>:</span></span><br><span class="line">        <span class="comment"># 动态为该类添加一个name属性</span></span><br><span class="line">        attrs[<span class="string">'name'</span>] = <span class="string">"C语言中文网"</span></span><br><span class="line">        attrs[<span class="string">'say'</span>] = <span class="keyword">lambda</span> self: print(<span class="string">"调用 say() 实例方法"</span>)</span><br><span class="line">        <span class="keyword">return</span> super().__new__(cls,name,bases,attrs)</span><br><span class="line">        </span><br><span class="line"><span class="comment"># 定义类时，指定元类</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CLanguage</span><span class="params">(object, metaclass=FirstMetaClass)</span>:</span></span><br><span class="line">    <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line">clangs = CLanguage()</span><br><span class="line">print(clangs.name)</span><br><span class="line">clangs.say()</span><br></pre></td></tr></table></figure><p>可以看到，在创建类时，通过在标注父类的同时指定元类（格式为<code>metaclass=元类名</code>），则当 <a href="http://c.biancheng.net/python/" target="_blank" rel="noopener">Python</a> 解释器在创建该类实例时，<code>FirstMetaClass(type)</code>元类中的<code>__new__</code>方法就会被调用，其中bases和attrs能拿到自定义类的参数，从而实现动态修改类属性或者类方法的目的。</p><p><strong>元类和父类的区别：</strong></p><p>在定义子类的时候，我们有两个选择：①是传需要继承的父类；②自定义的元类。</p><ul><li>父类是子类的模板，子类的功能是跟父类紧耦合的，子类和父类一般是一一对应的</li><li>元类是子类的修饰器，可以为该子类和其他子类都添加自定义功能，并且不在继承关系中(<code>Class.__mro__</code>查看)，子类和元类是一对多的关系。元类并不是特地为某个子类服务的</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestMeta3</span><span class="params">(type)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, bases, attrs)</span>:</span></span><br><span class="line">        print(cls)    <span class="comment"># 当前类 </span></span><br><span class="line">        print(<span class="string">"name"</span>, name)<span class="comment"># 如果是通过metaclass触发的，此处为调用metaclass的类的类型</span></span><br><span class="line">        print(<span class="string">"bases"</span>, bases)<span class="comment"># 如果是通过metaclass触发的，此处为调用metaclass的类的父类</span></span><br><span class="line">        print(<span class="string">"attrs"</span>, attrs)<span class="comment"># 如果是通过metaclass触发的，此处为调用metaclass的类的属性</span></span><br><span class="line">        <span class="keyword">return</span> type.__new__(cls, name, bases, attrs)</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Pa3</span>:</span></span><br><span class="line">    <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># python3中可以直接通过metaclass关键字参数来指定类的元类</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Eg3</span><span class="params">(Pa3, metaclass=TestMeta3)</span>:</span></span><br><span class="line"><span class="meta">    @classmethod</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">get</span><span class="params">(self)</span>:</span></span><br><span class="line">        kkk = []</span><br><span class="line">        kkk.append(self.__skiless__)</span><br><span class="line">        <span class="keyword">return</span> kkk</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">acc2</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> <span class="string">'a2'</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">输出</span></span><br><span class="line"><span class="string">&lt;class '__main__.TestMeta3'&gt;</span></span><br><span class="line"><span class="string">name Eg3</span></span><br><span class="line"><span class="string">bases (&lt;class '__main__.Pa3'&gt;,)</span></span><br><span class="line"><span class="string">attrs &#123;'__module__': '__main__', '__qualname__': 'Eg3', 'get': &lt;classmethod object at 0x00000263511C6FA0&gt;, 'acc2': &lt;function Eg3.acc2 at 0x00000263511C5310&gt;&#125;</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure><p>在定义的时候，发现竟然有输出。因为定义的时候，python解释器会在当前类中查找metaclass[3]，如果找到了，就<strong>使用该metaclass创建Eg3类</strong>。所以打印出来的name、bases、attrs都和Eg3有关。</p><p><strong>with_metaclass</strong></p><blockquote><p>由于python2和python3中元类使用方法的不同，我们需要使用一种兼容的方式[1]，如下所示：</p></blockquote><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">withMetaclass</span><span class="params">(meta, *bases)</span>:</span></span><br><span class="line">    <span class="string">"""Create a base class with a metaclass."""</span></span><br><span class="line">    <span class="class"><span class="keyword">class</span> <span class="title">MetaClass</span><span class="params">(meta)</span>:</span></span><br><span class="line"><span class="comment"># 如果删除__new__，则类.__mro__中能看到meta类</span></span><br><span class="line">        <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, this_bases, d)</span>:</span></span><br><span class="line">            <span class="comment"># 因为meta是类，所以这边是在调用meta的__call__()。▲bases，d为ConfigHandler的父类和属性</span></span><br><span class="line">            <span class="keyword">return</span> meta(name, bases, d)</span><br><span class="line"><span class="comment"># 返回一个新类型, type.__new__()要求第一个必须是type的子类</span></span><br><span class="line">    <span class="keyword">return</span> type.__new__(MetaClass, <span class="string">'temporary_class'</span>, (), &#123;&#125;)</span><br><span class="line"><span class="comment"># 下面两句话等价，withMetaclass是为了兼容python2和python3,2中没有metaclass关键字</span></span><br><span class="line"><span class="comment"># 其在创建实例时，__new__方法会被MetaClass拦截(其实就是子类没定义__new__，走了父类的__new__)</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span><span class="params">(withMetaclass<span class="params">(Singleton)</span>)</span></span></span><br><span class="line"><span class="class"># <span class="title">python3</span>写法</span></span><br><span class="line"><span class="class"><span class="title">class</span> <span class="title">ConfigHandler</span><span class="params">(metaclass=Singleton)</span></span></span><br><span class="line"><span class="class"></span></span><br><span class="line"># 因为ConfigHandler相当于继承了 type.__new__返回的类MetaClass，所以在c = ConfigHandler()实例化的时候，会触发Metaclass的__new__然后调用meta.__call__从而返回一个对象</span><br></pre></td></tr></table></figure><p><code>with_metaclass</code>返回的临时类中，本身无任何属性，但包含了元类和基类的所有信息，并在<strong>下一步定义类时</strong>将所有信息解包出来[1]。</p><p>见：<a href="https://zhuanlan.zhihu.com/p/354828950" target="_blank" rel="noopener">★Python 元类及with_metaclass</a></p><h3 id="type"><a class="markdownIt-Anchor" href="#type"></a> <a href="http://c.biancheng.net/view/2292.html" target="_blank" rel="noopener"><code>type</code>动态创建类</a></h3><ul><li>type() 函数属于 <a href="http://c.biancheng.net/python/" target="_blank" rel="noopener">Python</a> 内置函数，通常用来查看某个变量的具体类型。<ul><li><code>type(obj)</code></li></ul></li><li>其实，type() 函数还有一个更高级的用法，即<strong>创建</strong>一个自定义<strong>类型</strong>（也就是创建一个类）。<ul><li><code>type(name, bases, dict)</code>：其中 name 表示类的名称；bases 表示一个元组，其中存储的是该类的父类；dict 表示一个字典，用于表示类内定义的属性或者方法。</li></ul></li></ul><p>实际上<code>type(name, bases, dict)</code>是调用了type类的<code>type.__init__(cls, what, bases=None, dict=None)</code>方法，创建了一个type的实例（类类型就是一个type实例），类型是<code>&lt;class 'type'&gt;</code></p><p><strong>&lt;class ‘type’&gt;是所有类型的类型。&lt;class ‘object’&gt;也是所有对象的超类（除了它自己，包括type）</strong></p><p>▲. 此外type还有<code>type.__new__(*args, **kwargs)</code>，其作用是<code>Create and return a new object.</code>，可以写成<code>type.__new__(ClassTpye, name, base, dicts)</code>，但ClassType必须是type的子类。会返回一个跟ClassType有关系的新类型</p><h3 id="通过元类创建单例类"><a class="markdownIt-Anchor" href="#通过元类创建单例类"></a> 通过元类创建单例类</h3><blockquote><p>现在让我们正式看，我在开源代码里看到的内容：</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 注意这边继承了type, 所以下面的__call__是重写type的__call__，即创建实例的方法</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Singleton</span><span class="params">(type)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    Singleton Metaclass</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    _inst = &#123;&#125;</span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__call__</span><span class="params">(cls, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> cls <span class="keyword">not</span> <span class="keyword">in</span> cls._inst:</span><br><span class="line">            cls._inst[cls] = super(Singleton, cls).__call__(*args, **kwargs)</span><br><span class="line">        <span class="keyword">return</span> cls._inst[cls]</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">withMetaclass</span><span class="params">(meta, *bases)</span>:</span></span><br><span class="line">    <span class="string">"""Create a base class with a metaclass."""</span></span><br><span class="line">    <span class="comment"># 这需要一点解释：基本思想是为一个级别的类实例化创建一个虚拟元类，用实际的元类替换它自己。</span></span><br><span class="line">    </span><br><span class="line">    <span class="comment"># KeyPoint1. 继承meta类</span></span><br><span class="line">    <span class="class"><span class="keyword">class</span> <span class="title">MetaClass</span><span class="params">(meta)</span>:</span></span><br><span class="line">        <span class="comment"># 实际上下面的__new__不影响</span></span><br><span class="line">        <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, this_bases, d)</span>:</span></span><br><span class="line">            <span class="comment"># cls为withMetaclass; name为使用者的类型; this_bases为使用者的父类们; d为使用者的属性</span></span><br><span class="line">            <span class="keyword">return</span> meta(name, bases, d)</span><br><span class="line">    <span class="comment"># KeyPoint2. type.__new__创建一个名称叫temporary_class，类型为MetaClass的类</span></span><br><span class="line">    <span class="comment"># ▲注意type.__new__中的类类型必须是type的子类</span></span><br><span class="line">    <span class="keyword">return</span> type.__new__(MetaClass, <span class="string">'temporary_class'</span>, (), &#123;&#125;)</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span><span class="params">(withMetaclass<span class="params">(Singleton)</span>)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        print(<span class="string">"__init__"</span>)</span><br><span class="line">        <span class="keyword">pass</span></span><br><span class="line"><span class="meta">    @LazyProperty</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">serverHost</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> os.environ.get(<span class="string">"HOST"</span>, setting.HOST)</span><br><span class="line"><span class="comment"># res=withMetaclass(Singleton)的类型为&lt;class 'util.six.withMetaclass.&lt;locals&gt;.MetaClass'&gt;</span></span><br><span class="line"><span class="comment"># 将其传给ConfigHandler作为父类，在定义 ConfigHandler 时会触发MetaClass.__new__, 于是调用meta(name, bases, d)，此处的meta为Singleton，而name为ConfigHandler类, bases为空, d为ConfigHandler的属性和方法。</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># print(type(ConfigHandler)) ==&gt;  &lt;class 'util.singleton.Singleton'&gt;</span></span><br><span class="line"><span class="comment"># 因此 c = ConfigHandler() ==&gt; Singleton的__call__方法，</span></span><br><span class="line">当c = ConfigHandler()时会因为type.__new__(MetaClass, <span class="string">'temporary_class'</span>, (), &#123;&#125;)去找MetaClass的__call__进行调用，MetaClass没有__call__则找到了其父类meta(Singleton)的__call__</span><br></pre></td></tr></table></figure><p>注：类也是对象，是元类的对象，即我们实例化一个类时，调用其元类的<code>__call__(cls, *args, **kwargs)</code>方法进行创建对象。</p><h2 id="__call__"><a class="markdownIt-Anchor" href="#__call__"></a> <code>__call__</code></h2><blockquote><p>一个非常特殊的实例方法，即<code>__call__()</code>。该方法的功能是在类中<strong>重载了对象的 () 运算符</strong>，使得类实例对象可以像调用普通函数那样，以“<strong>对象名()</strong>”的形式使用。</p></blockquote><p>实际上，如果不重写<code>__call__</code>的话，<code>Class.__call__(*args, **kwargs)</code>还承担着产生类实例的功能（会调用父类（可以通过<code>Class.__class__</code>来查看父类）的<code>type.__call__</code>其会返回一个实例）</p><p>案例一:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 默认继承的是object, 而不是type</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Meta</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name)</span>:</span></span><br><span class="line">        print(<span class="string">"init"</span>)</span><br><span class="line">        self.name = name</span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__call__</span><span class="params">(self, *args, **kwargs)</span>:</span></span><br><span class="line">        print(<span class="string">"call"</span>)</span><br><span class="line"><span class="comment"># 当没有重写__call__时，无论是显式的调用__call__，还是通过()运算符调用，都会调用type.__call__返回一个实例</span></span><br><span class="line">res = Meta.__call__(<span class="string">"asd"</span>)</span><br><span class="line">print(res, type(res))</span><br><span class="line">res = Meta(<span class="string">"asd"</span>)</span><br><span class="line">print(res, type(res))</span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">init</span></span><br><span class="line"><span class="string">&lt;__main__.Meta object at 0x0000014FB5115EE0&gt; &lt;class '__main__.Meta'&gt;</span></span><br><span class="line"><span class="string">init</span></span><br><span class="line"><span class="string">&lt;__main__.Meta object at 0x0000014FB5115A90&gt; &lt;class '__main__.Meta'&gt;</span></span><br><span class="line"><span class="string">上述两种都能创建对象</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="comment"># 当重写__call__以后, __call__()返回实例的效果就失效了==&gt;因为上述代码没有return</span></span><br><span class="line"><span class="comment"># 此时 Meta()与Meta.__call__()不再等价</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">call</span></span><br><span class="line"><span class="string">None &lt;class 'NoneType'&gt;</span></span><br><span class="line"><span class="string">init</span></span><br><span class="line"><span class="string">&lt;__main__.Meta object at 0x0000016CC2745EE0&gt; &lt;class '__main__.Meta'&gt;</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure><p>Q:我们在实例化一个对象的时候<code>f = Foo(1, y=2)</code>，可以发现在<code>__init__()</code>中并没有返回实例，但调用<code>Foo(1, y=2)</code><strong>确实</strong>返回了一个对象，而且，<code>__init__</code>预期一个<code>self</code>参数，但是当我们调用<code>Foo(1, y=2)</code>时这里并没有这个参数。那么类实例化的过程到底是怎么样的呢？</p><p>A:构造顺序——<a href="https://www.jianshu.com/p/f63ad9d550f1" target="_blank" rel="noopener">理解python的类实例化</a></p><p>首先明确一点，Python中的<strong>类也是对象</strong>！类、函数、方法以及实例都是对象——<strong>类类型是type的对象</strong>，并且无论何时你将一对括号放在它们的名字后面时，就会调用<code>type.__call__()</code>方法。为什么呢？因为type是类型的父类</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>Foo.__class__</span><br><span class="line">&lt;<span class="class"><span class="keyword">class</span> '<span class="title">type</span>'&gt;</span></span><br></pre></td></tr></table></figure><p>所以<code>Foo</code>是类型<code>type</code>的一个对象，并且调用type类的<code>__call__(self, *args, **kwargs)</code>返回一个<code>Foo</code>类的对象。让我们看下<code>type</code>中的<code>__call__</code>方法是什么样的。这个方法相当的复杂，但是我们将其C代码转成Python代码，并尝试尽量简化它，结果如下。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">type</span><span class="params">(object)</span>:</span></span><br><span class="line"><span class="comment"># 这边的obj_type跟cls一样</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__call__</span><span class="params">(obj_type, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="comment"># 通过__new__创建一个空的类实例，如果obj_type没有__new__则使用type.__new__</span></span><br><span class="line">        obj = obj_type.__new__(*args, **kwargs)</span><br><span class="line">        <span class="comment"># 进行类型检查</span></span><br><span class="line">        <span class="keyword">if</span> obj <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">None</span> <span class="keyword">and</span> issubclass(obj, obj_type):</span><br><span class="line">            <span class="comment"># 对类进行__init__初始化</span></span><br><span class="line">            obj.__init__(*args, **kwargs)</span><br><span class="line">        <span class="comment"># 返回类实例</span></span><br><span class="line">        <span class="keyword">return</span> obj</span><br></pre></td></tr></table></figure><p>可见<code>__new__</code>方法为对象分配了内存空间，构建它为一个“空&quot;对象然后<code>__init__</code>方法被调用来初始化它。</p><p>那我们定义了一个具体类来讲解这个过程。首先明确一点：<strong>Foo相对于产生了一个type实例化对象</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Foo</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, x, y=<span class="number">0</span>)</span>:</span></span><br><span class="line">        self.x = x</span><br><span class="line">        self.y = y</span><br></pre></td></tr></table></figure><p>获得实例化对象**<code>Foo(*args, **kwargs)</code><strong>也可以看作是<code>type对象()</code>即</strong>调用了type中()运算符的触发的函数<code>type.__call__</code>从而创建一个Foo的实例**</p><ul><li>至于<code>type.__call__</code>发生了什么就是上面抽象代码中介绍的那般，调用<code>type.__new__(Foo, *args, **kwargs)</code>然后返回一个对象实例obj。</li><li><code>obj</code>随后通过调用<code>obj.__init__(*args, **kwargs)</code>被初始化。</li><li><code>obj</code>被<code>type.__call__</code>中返回。</li></ul><p>▲注意：<code>Foo.__call__</code>重载的是<code>foo对象</code>的()运算符，而<code>Foo()</code>实例化foo对象，则执行的是<code>type对象</code>的()运算符。</p><p><strong>小总结</strong>：</p><ul><li>现在我们能知道为什么元类必须继承type了：因为我们实例化对象<code>Foo(xxx)</code>时调用了<code>type.__call__</code>,而<code>type.__call__</code>又会调用<code>type.__new__</code>因此如果type子类重写实现了<code>__new__</code>（返回的类实例对象的类型作控制）、<code>__call__</code>（对实例化的流程做控制），则可以对<strong>类对象的类型和类属性</strong>起到自定义的功能，而重写就必须继承type=&gt;需要元类必须继承type</li><li>所以按照上述的逻辑，如果定义了一个元类让自定义类用的话<code>class Foo(metaclass=MyMetaClass)</code>，在其实例化过程中<code>Foo()</code>会直接调用重写后的<code>MyMetaClass.__call__</code>，而只要记住在<code>MyMetaClass.__call__</code>中使用到<code>return super(Singleton, cls).__call__(*args, **kwargs)</code>就可以把<code>type.__call__</code>生成的实例返回啦。所以这也是为什么<u>编写元类，一般都是继承了type，然后根据想控制实例化流程就重写<code>__call__</code>方法，想添加属性就重写<code>__new__</code>方法就行了。</u></li><li><strong>★元类产生影响的时间点是在实例化的时候</strong></li></ul><p>注意点：元类继承了type，所以实例化元类是在产生一个类类型，就要以type创建类类型的参数去产生。而元类的使用一般都是自定义类<code>class MyClass(metaclass=元类)</code>，然后实例化自定义类<code>MyClass(xxx)</code></p><p><strong>总结</strong>：看完上述知识点后，我们能知道为什么withclass能起到metaclass的作用(类的<code>__mro__</code>中不出现指定的元类)了：</p><ul><li>首先分析流程：<code>return type.__new__(Metaclass)</code>返回了一个类型供自定义类继承，由于MetaClass继承的是真正的元类(元类都继承type)，所以在自定义类实例化的时候会被Metaclass的<code>__new__</code>方法拦截，在<code>MetaClass.__new__</code>里return了一个自定义实例，并把对象加入到了Singleton字典中了。</li><li>其次讲解为什么MetaClass中没有MetaClass：因为根据<a href="#"><code>__new__</code></a>知识点中讲到的，<code>__new__</code>控制了实例产生，<code>return type.__new__(Metaclass)</code>中创建了<code>Metaclass</code>，但其在<code>__new__</code>中返回的并不是MetaClass，因此<code>__mro__</code>中不会出现<code>Metaclass</code></li><li>最后还要讲讲Singleton中的执行逻辑：</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Singleton</span><span class="params">(type)</span>:</span></span><br><span class="line">    _inst = &#123;&#125;</span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__call__</span><span class="params">(cls, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> cls <span class="keyword">not</span> <span class="keyword">in</span> cls._inst:</span><br><span class="line">            <span class="comment"># super(Singleton, cls).__call__调用的是type.__call__(自定义类类名name, 自定义类父类bases, 自定义类属性attrs)</span></span><br><span class="line">            cls._inst[cls] = super(Singleton, cls).__call__(*args, **kwargs)</span><br><span class="line">        <span class="keyword">return</span> cls._inst[cls]</span><br></pre></td></tr></table></figure>]]></content>
    
    <summary type="html">
    
      
      
        &lt;blockquote&gt;
&lt;p&gt;这是一个在看开源代码时配到了单例元类写法后的一个原理贴，可以帮你彻底弄清元类、以及&lt;code&gt;__new__&lt;/code&gt;和&lt;code&gt;__call__&lt;/code&gt;，相信我，看完你绝对会有收获。&lt;/p&gt;
&lt;p&gt;首先，关于元类的结论说在前头，先有个
      
    
    </summary>
    
    
      <category term="Python" scheme="https://nymrli.top/tags/Python/"/>
    
  </entry>
  
  <entry>
    <title>PlantUML作图</title>
    <link href="https://nymrli.top/2022/04/04/PlantUML%E4%BD%9C%E5%9B%BE/"/>
    <id>https://nymrli.top/2022/04/04/PlantUML作图/</id>
    <published>2022-04-04T03:30:28.000Z</published>
    <updated>2022-04-04T05:03:10.461Z</updated>
    
    <content type="html"><![CDATA[<blockquote><p>最近在看开源代码，有些项目的架构和执行逻辑都非常有学习、借鉴意义，因此想作图可视化记录下来。</p><p>在写文档的过程中，经常需要进行画图。虽然说现在有很多类似viso之类的可视化画图工具，但是还是要花费大量时间在拖拉组件上，效率十分低下。而<a href="http://plantuml.com/" target="_blank" rel="noopener">Plantuml</a>与markdown类似，本质上是也算一门可以快速画图的<strong>设计语言</strong>，学习起来也很方便，学完后直接书写、不需要拖拽就能做出规整的图，能大大节约在布局上细枝末节的时间。</p><p>此外，PlantUML在vscode, webstorm都有相关的插件可以使用。</p></blockquote><p>在线体验网站有很多使用的样例介绍，在此我就不过多介绍了。本文会以我使用到的功能图来提醒说明一下一些语法规范和注意点。</p><h2 id="时序图"><a class="markdownIt-Anchor" href="#时序图"></a> 时序图</h2><ul><li><p>注释</p><ul><li><p>图上不显示的注释：<code>/' 注释内容 '/</code></p></li><li><p>图中显示的注释：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 左侧注释</span></span><br><span class="line">Alice-&gt;Bob : hello</span><br><span class="line">note left: this is a first note</span><br><span class="line"></span><br><span class="line"><span class="comment"># 右侧注释</span></span><br><span class="line">Bob--&gt;Alice : ok</span><br><span class="line">note right: this is another note</span><br><span class="line"></span><br><span class="line"><span class="comment"># 跨越两个的注释</span></span><br><span class="line">note over Alice, Bob <span class="comment">#FFAAAA: This is displayed\n over Bob and Alice.</span></span><br></pre></td></tr></table></figure></li></ul></li><li><p><strong>返回信息</strong></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Alice -&gt; Bob: Another authentication Request</span><br><span class="line"><span class="comment"># 同样可以写成Bob --&gt; Alice, 只不过下面的更直观能看出是返回</span></span><br><span class="line">Alice &lt;-- Bob: Another authentication Response</span><br></pre></td></tr></table></figure></li><li><p><strong>生命线的激活与撤销</strong></p><ul><li>关键字<code>activate</code>和<code>deactivate</code>用来表示参与者的生命活动。一旦参与者被激活，它的生命线（柱子）就会显示出来。<ul><li><code>activate</code>和<code>deactivate</code>适用于以上情形。</li><li><code>destroy</code>表示一个参与者的生命线的终结。</li></ul></li><li>给生命线上色语法：<code>activate Holder [#color]</code></li></ul><p>▲.一个参与者在最外层只用于有一个生命线，但在大生命线中可以通过activate创建多个嵌套的小生命线，用deactivate跳出当前生命线</p><p>★因此，最好生命线的创建和退出同时写出，即写了一个<code>activate ?</code>就写一个<code>deactivate ?</code>， 需要完成的内容在其中编写</p><p>此外，需要注意的是，当前结果的返回应该在deactivate之前。比如<code>A &lt;-- C: return</code>后才是<code>deactivate C</code>，当然如果遵循了成对书写activate的习惯的话，就不存在这个问题了。</p></li><li><p><strong>return关键字</strong>：以用于生成一个带有可选文本标签的返回信息。返回的点是导致最近一次激活生命线的点。但<strong>不建议使用</strong>，因为在维护修改的时候无法直观的看出到底是从哪返回哪。而是采取<code>A &lt;-- B: return</code>来声明从B到A的虚线返回</p></li><li><p><strong>分隔符</strong>： 通过使用<code>==</code>关键词来将你的图表分割成多个逻辑步骤。</p></li><li><p><strong>声明参与者</strong>： 声明的顺序将是（默认的）<strong>显示顺序</strong>。</p><p>格式为: <code>类型 参与者名称 [as AliasName] [order orderVaalue] [#color]</code></p><p>可以选择的类型有：</p><ul><li><code>actor</code>（角色）</li><li><code>participant</code>(默认参与者)</li><li><code>boundary</code>（边界）</li><li><code>control</code>（控制）</li><li><code>entity</code>（实体）</li><li><code>database</code>（数据库）</li><li><code>collections</code>（集合）</li><li><code>queue</code>（队列）</li></ul></li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br></pre></td><td class="code"><pre><span class="line">@startuml</span><br><span class="line">/&apos; 声明参与者 &apos;/</span><br><span class="line">Actor       actor       as A</span><br><span class="line">Participant scheduler as S</span><br><span class="line">Participant Fetcher as F</span><br><span class="line">Participant check as C</span><br><span class="line">Participant DoValidator as D</span><br><span class="line">Participant Queue as Q</span><br><span class="line">Participant ProxyHandler as P</span><br><span class="line">Participant ProxyFetcher as PF</span><br><span class="line">A -&gt; S: runScheduler()</span><br><span class="line"></span><br><span class="line">/&apos;scheduler主要逻辑&apos;/</span><br><span class="line">activate S</span><br><span class="line">S -&gt; S: __runProxyFetch()</span><br><span class="line">== 定时器设置 ==</span><br><span class="line">S -&gt; S: scheduler.add_job(__runProxyFetch)</span><br><span class="line">S -&gt; S: scheduler.add_job(__runProxyCheck)</span><br><span class="line">S -&gt; S: scheduler.configure()</span><br><span class="line">S -&gt; S: scheduler.start()</span><br><span class="line"></span><br><span class="line">== 定时器执行 ==</span><br><span class="line">S -&gt; S: __runProxyFetch()</span><br><span class="line">/&apos; --__runProxyFetch-- &apos;/</span><br><span class="line">activate S #orange</span><br><span class="line">S -&gt; F: __runProxyFetch()</span><br><span class="line">activate F  #pink</span><br><span class="line">F-&gt;F: run()</span><br><span class="line">F-&gt;PF: freeProxyXX()</span><br><span class="line">activate PF</span><br><span class="line">PF--&gt;F: proxy</span><br><span class="line">deactivate PF</span><br><span class="line">F-&gt;Q: proxy_queue.put(proxy)</span><br><span class="line">Q--&gt;F</span><br><span class="line">F--&gt;S</span><br><span class="line">deactivate F</span><br><span class="line"></span><br><span class="line">S -&gt; C: Checker(&quot;raw&quot;, proxy_queue)</span><br><span class="line">activate C</span><br><span class="line">C -&gt; C: _ThreadChecker.run()</span><br><span class="line">C -&gt; D: DoValidator.validator(proxy)</span><br><span class="line"></span><br><span class="line">activate D</span><br><span class="line">D -&gt; D: cls.httpValidator(proxy)</span><br><span class="line">D -&gt; D: cls.httpsValidator(proxy)</span><br><span class="line">D -&gt; C</span><br><span class="line">deactivate D</span><br><span class="line"></span><br><span class="line">alt work_type </span><br><span class="line">C-&gt;C: __ifRaw()</span><br><span class="line">else</span><br><span class="line">C-&gt;C: __ifUse()</span><br><span class="line">end</span><br><span class="line">C--&gt;S</span><br><span class="line"></span><br><span class="line">deactivate C</span><br><span class="line">deactivate S</span><br><span class="line">/&apos; --__runProxyFetch-- &apos;/</span><br><span class="line"></span><br><span class="line">/&apos; --__runProxyCheck-- &apos;/</span><br><span class="line">S -&gt; S: __runProxyCheck()</span><br><span class="line">activate S #orange</span><br><span class="line">S-&gt;P: &lt;&lt;create proxy_handler&gt;&gt;</span><br><span class="line">P--&gt;S</span><br><span class="line">S-&gt;Q: &lt;&lt;create proxy_queue&gt;&gt;</span><br><span class="line">Q--&gt;S</span><br><span class="line"></span><br><span class="line">alt proxy_handler.db.getCount() &lt; 0</span><br><span class="line">S-&gt;S: __runProxyFetch()</span><br><span class="line">else</span><br><span class="line">S-&gt;P: proxy_handler.getAll()</span><br><span class="line">P--&gt;S</span><br><span class="line">end</span><br><span class="line">S-&gt;C: Checker(&quot;use&quot;, proxy_queue)</span><br><span class="line">C--&gt;S</span><br><span class="line">deactivate S</span><br><span class="line">/&apos; --__runProxyCheck-- &apos;/</span><br><span class="line">S --&gt; A</span><br><span class="line">deactivate S</span><br><span class="line">@enduml</span><br></pre></td></tr></table></figure>]]></content>
    
    <summary type="html">
    
      
      
        &lt;blockquote&gt;
&lt;p&gt;最近在看开源代码，有些项目的架构和执行逻辑都非常有学习、借鉴意义，因此想作图可视化记录下来。&lt;/p&gt;
&lt;p&gt;在写文档的过程中，经常需要进行画图。虽然说现在有很多类似viso之类的可视化画图工具，但是还是要花费大量时间在拖拉组件上，效率十分低下。而&lt;
      
    
    </summary>
    
    
      <category term="计算机基础" scheme="https://nymrli.top/tags/%E8%AE%A1%E7%AE%97%E6%9C%BA%E5%9F%BA%E7%A1%80/"/>
    
  </entry>
  
  <entry>
    <title>开源项目代码阅读记录</title>
    <link href="https://nymrli.top/2022/03/29/%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE%E4%BB%A3%E7%A0%81%E9%98%85%E8%AF%BB%E8%AE%B0%E5%BD%95/"/>
    <id>https://nymrli.top/2022/03/29/开源项目代码阅读记录/</id>
    <published>2022-03-29T09:54:46.000Z</published>
    <updated>2022-05-29T10:01:40.807Z</updated>
    
    <content type="html"><![CDATA[<h1 id="爬虫系列"><a class="markdownIt-Anchor" href="#爬虫系列"></a> 爬虫系列</h1><h2 id="抢购-秒杀"><a class="markdownIt-Anchor" href="#抢购-秒杀"></a> 抢购、秒杀</h2><h3 id="taobao_seckill"><a class="markdownIt-Anchor" href="#taobao_seckill"></a> <strong><a href="https://github.com/jerry3747/taobao_seckill" target="_blank" rel="noopener"> taobao_seckill</a></strong></h3><blockquote><p>淘宝秒杀物品：分为selenium版和request网络请求版，微小的区别在于，api先请求的是购物车的物品信息，selenium则是勾选选择框。</p><p>流程大致都相同：①keep_wait等待临近抢购时间-----&gt;②将需要的信息缓存住，不再刷新（比如要抢购的物品信息、cookies…）-----&gt;③进行结算(/点击按钮)-----&gt;④提交订单(/点击按钮)-----&gt;⑤支付订单(/点击按钮)；</p></blockquote><ul><li><p><a href="https://www.cnblogs.com/amiza/p/10175543.html" target="_blank" rel="noopener">使用browsercookie来管理浏览器cookies</a>——从浏览器提取保存的cookies的工具。它是一个很有用的爬虫工具，通过加载你浏览器的cookies到一个cookiejar对象里面，让你轻松下载需要登录的网页内容。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>cj = browsercookie.chrome()<span class="comment"># browsercookie.load() 在不知道或者不关心浏览器时使用</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>r = requests.get(url, cookies=cj)</span><br></pre></td></tr></table></figure><p>一般情况下，作为开发者的我们都是将cookies手动写入或者是传入程序的，而browsercookie用途，我想可以是将需要传入cookies的程序打包给不会提取Cookies的用户使用。（类似通过selenium登陆后，拿取cookies再发送网络请求，这边是让/借助用户自己的浏览器来拿取cookies）</p></li><li><p>使用tkinter搭建简单的GUI：点击“开始”的按钮后会将密码框和抢购设置时间框的内容传入，作为实例化ChromeDriverSpider的参数，并运行实例的sec_kill方法开始执行抢购</p></li><li><p>配置chromeDriver启动配置项</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">build_chrome_options</span><span class="params">(self)</span>:</span></span><br><span class="line">    <span class="string">"""配置启动项"""</span></span><br><span class="line">    chrome_options = webdriver.ChromeOptions()</span><br><span class="line">    chrome_options.accept_untrusted_certs = <span class="keyword">True</span></span><br><span class="line">    chrome_options.assume_untrusted_cert_issuer = <span class="keyword">True</span></span><br><span class="line">    arguments = [<span class="string">'--no-sandbox'</span>, <span class="string">'--disable-impl-side-painting'</span>, <span class="string">'--disable-setuid-sandbox'</span>, <span class="string">'--disable-seccomp-filter-sandbox'</span>,</span><br><span class="line">    <span class="string">'--disable-breakpad'</span>, <span class="string">'--disable-client-side-phishing-detection'</span>, <span class="string">'--disable-cast'</span>,</span><br><span class="line">    <span class="string">'--disable-cast-streaming-hw-encoding'</span>, <span class="string">'--disable-cloud-import'</span>, <span class="string">'--disable-popup-blocking'</span>,</span><br><span class="line">    <span class="string">'--ignore-certificate-errors'</span>, <span class="string">'--disable-session-crashed-bubble'</span>, <span class="string">'--disable-ipv6'</span>,</span><br><span class="line">    <span class="string">'--allow-http-screen-capture'</span>, <span class="string">'--start-maximized'</span>]</span><br><span class="line">    <span class="keyword">for</span> arg <span class="keyword">in</span> arguments:</span><br><span class="line">    chrome_options.add_argument(arg)</span><br><span class="line">    chrome_options.add_argument(<span class="string">f'--user-agent=<span class="subst">&#123;choice(get_useragent_data())&#125;</span>'</span>)</span><br><span class="line">    <span class="keyword">return</span> chrome_options</span><br></pre></td></tr></table></figure></li></ul><h3 id="taobao_api"><a class="markdownIt-Anchor" href="#taobao_api"></a> <strong><a href="https://github.com/gaoxt/taobao_api" target="_blank" rel="noopener">taobao_api</a></strong></h3><blockquote><p>selenium和api结合：登陆逻辑使用了selenium，刷新购物车和下单调用了手机淘宝taobao_api，自动付款也使用了selenium，能自动模拟触摸输入密码付款，接口签名sign生成方法参考get_sign_val。</p></blockquote><ul><li><p>同步阿里云时间服务器</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#一次性设置</span></span><br><span class="line">sudo sntp -sS ntp.aliyun.com</span><br><span class="line"><span class="comment">#永久设置</span></span><br><span class="line">sudo systemsetup -setnetworktimeserver ntp.aliyun.com</span><br><span class="line">sudo systemsetup -setusingnetworktime on</span><br></pre></td></tr></table></figure></li><li><p>展示QRCode</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">   <span class="comment"># 找到二维码图片的链接</span></span><br><span class="line">   img = WebDriverWait(browser, <span class="number">2</span>, <span class="number">0.1</span>).until(</span><br><span class="line">       EC.presence_of_element_located((By.XPATH, <span class="string">"//div[@id='J_QRCodeImg']/img"</span>))).get_attribute(<span class="string">'src'</span>)</span><br><span class="line">   <span class="comment"># 获得图片字节流内容</span></span><br><span class="line">   img_content = requests.get(img, timeout=<span class="number">5</span>).content</span><br><span class="line">   <span class="comment"># 保存到本地图片中</span></span><br><span class="line">   file_name = <span class="string">'taobao_qrcode.png'</span></span><br><span class="line">   fp = open(file_name, <span class="string">'wb'</span>)</span><br><span class="line">   fp.write(img_content)</span><br><span class="line">   fp.close()</span><br><span class="line"><span class="comment"># 将其解码成链接， pyzbar为二维码和条形码库 ==&gt; 个人觉得保存后可以直接转成PIL的Image然后img.show()</span></span><br><span class="line">   <span class="keyword">from</span> pyzbar.pyzbar <span class="keyword">import</span> decode</span><br><span class="line">   barcode_url = <span class="string">''</span></span><br><span class="line">   barcodes = decode(Image.open(file_name))</span><br><span class="line">   <span class="keyword">for</span> barcode <span class="keyword">in</span> barcodes:</span><br><span class="line">       <span class="comment"># 解析获得二维码后的内容；如果是条形码，则返回条形码数</span></span><br><span class="line">       barcode_url = barcode.data.decode(<span class="string">"utf-8"</span>)</span><br><span class="line"><span class="comment"># 生成二维码并通过控制台打印</span></span><br><span class="line">   qr = qrcode.QRCode()</span><br><span class="line">   qr.add_data(barcode_url)</span><br><span class="line">   qr.print_ascii(invert=<span class="keyword">True</span>)</span><br></pre></td></tr></table></figure><p>注：<a href="http://zbar.sourceforge.net/" target="_blank" rel="noopener">ZBar</a>是一个开源软件套件，用于从各种来源（如视频流、图像文件和原始强度传感器）读取条形码；<code>pyzbar</code> 是通过 <code>Python2</code>和<code>3</code>接口，使用 <code>ZBar</code> 库读取一维条形码和QR码 。=&gt;<a href="https://blog.csdn.net/qq_37924224/article/details/109582507" target="_blank" rel="noopener">读取条形码</a>、<a href="https://blog.csdn.net/hxj0323/article/details/112969622" target="_blank" rel="noopener">读取二维码</a></p></li></ul><h3 id="jd_seckill"><a class="markdownIt-Anchor" href="#jd_seckill"></a> <strong><a href="https://github.com/andyzys/jd_seckill" target="_blank" rel="noopener">jd_seckill</a></strong></h3><blockquote><ol><li>预约商品</li><li>秒杀抢购商品: ①<strong>访问</strong>商品的抢购<strong>链接</strong>（用于设置cookie等—&gt;②访问抢购订单<strong>结算页面</strong>----&gt;③<strong>提交抢</strong>购（秒杀）<strong>订单</strong> [_get_seckill_order_data生成提交抢购订单所需的请求体参数、获取秒杀初始化信息（包括：地址，发票，token）]</li></ol></blockquote><ul><li><p>从UAList中通过<code>random.choice(USER_AGENTS)</code>随机UA</p></li><li><p>使用内置configparser模块解析config.ini</p><ul><li>RawConfigParser是最基础的INI文件读取类；ConfigParser、SafeConfigParser<a href="https://blog.csdn.net/miner_k/article/details/77857292" target="_blank" rel="noopener">支持对%(value)s变量的解析</a>，SafeConfigParser类实现了ConfigParser相同的接口，新增如下方法：set存在保存的方法</li><li>section下的option(key)可以不填value值，即为空。</li><li>from: <a href="https://blog.csdn.net/tianzhu123/article/details/6893755" target="_blank" rel="noopener">python中RawConfigParser类、ConfigParser类、SafeConfigParser类中函数介绍</a>、</li></ul></li><li><p>多进程池：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> concurrent.futures <span class="keyword">import</span> ProcessPoolExecutor</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> ProcessPoolExecutor(work_count) <span class="keyword">as</span> pool:</span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(work_count):</span><br><span class="line">        pool.submit(self.seckill)</span><br></pre></td></tr></table></figure></li><li><p>秒杀时间比较</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">buy_time = datetime.strptime(global_config.getRaw(<span class="string">'config'</span>,<span class="string">'buy_time'</span>), <span class="string">"%Y-%m-%d %H:%M:%S.%f"</span></span><br><span class="line">now_time = datetime.now</span><br><span class="line"><span class="keyword">while</span> <span class="keyword">True</span>:</span><br><span class="line">    <span class="keyword">if</span> now_time() &gt;= buy_time:</span><br><span class="line">        logger.info(<span class="string">'时间到达，开始执行……'</span>)</span><br><span class="line">        <span class="keyword">break</span></span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line">time.sleep(self.sleep_interval)</span><br></pre></td></tr></table></figure></li><li><p><a href="https://github.com/woxiqingxian/jd_seckill/pull/2/commits/63a41a0609912d5643bc11f3419fe7dd8e84a00a" target="_blank" rel="noopener">获取京东服务器时间</a>：<code>https://api.m.jd.com/client.action?functionId=queryMaterialProducts&amp;client=wh5</code></p></li></ul><h3 id="jd-assistant"><a class="markdownIt-Anchor" href="#jd-assistant"></a> <strong><a href="https://github.com/tychxn/jd-assistant" target="_blank" rel="noopener">jd-assistant★</a></strong></h3><blockquote><p>是<a href="#jd_seckill">jd_seckill</a>的优化进阶版本</p><ul><li><p>普通商品购买主要流程：<code>(1)清空购物车</code> --&gt; <code>(2)添加商品到购物车</code> --&gt; <code>(3)提交订单</code></p></li><li><p>预约抢购商品特点：</p><ol><li>需要提前点击预约</li><li>大部分此类商品在预约后自动加入购物车，但是无法勾选✓，也无法️进入到结算页面</li><li>到了抢购的时间点后将商品加入购物车，此时才能勾选并下单</li></ol></li><li><p>普通商品：能加入购物车🛒，然后进入购物车结算下单.</p></li><li><p>抢购商品：需提提前预约，开始抢购时有“立即抢购”按钮🔘，点击按钮后商品加入购物车，然后结算下单.</p></li></ul><p>⭐️抢购受多种因素影响：<strong>网速、账号质量、运气</strong>等等，仅供娱乐，认真就输了.</p></blockquote><ul><li><p>deprecated、check_login装饰器：传入了self参数，并通过self实例调用了类的其他成员方法</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">check_login</span><span class="params">(func)</span>:</span></span><br><span class="line">    <span class="string">"""用户登陆态校验装饰器。若用户未登陆，则调用扫码登陆"""</span></span><br><span class="line"><span class="meta">    @functools.wraps(func)</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">new_func</span><span class="params">(self, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> self.is_login:</span><br><span class="line">            logger.info(<span class="string">"&#123;0&#125; 需登陆后调用，开始扫码登陆"</span>.format(func.__name__))</span><br><span class="line">            self.login_by_QRcode()</span><br><span class="line">        <span class="keyword">return</span> func(self, *args, **kwargs)</span><br><span class="line">    <span class="keyword">return</span> new_func</span><br></pre></td></tr></table></figure></li><li><p>密码加密——<strong><a href="https://github.com/Legrandin/pycryptodome" target="_blank" rel="noopener">pycryptodome</a></strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> Crypto.PublicKey <span class="keyword">import</span> RSA</span><br><span class="line"><span class="keyword">from</span> Crypto.Cipher <span class="keyword">import</span> PKCS1_v1_5 <span class="keyword">as</span> Cipher_pkcs1_v1_5</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">encrypt_pwd</span><span class="params">(password, public_key=RSA_PUBLIC_KEY)</span>:</span></span><br><span class="line">    rsa_key = RSA.importKey(public_key)</span><br><span class="line">    encryptor = Cipher_pkcs1_v1_5.new(rsa_key)</span><br><span class="line">    cipher = b64encode(encryptor.encrypt(password.encode(<span class="string">'utf-8'</span>)))</span><br><span class="line">    <span class="keyword">return</span> cipher.decode(<span class="string">'utf-8'</span>)</span><br></pre></td></tr></table></figure></li><li><p>提供JS脚本文件来获取商品的信息</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 以下代码在订单结算页面的开发者工具Console中执行，用于获取必要的参数</span></span><br><span class="line"><span class="keyword">var</span> eid = $(<span class="string">'#eid'</span>).val();</span><br><span class="line"><span class="keyword">var</span> fp = $(<span class="string">'#fp'</span>).val();</span><br><span class="line"><span class="keyword">var</span> trackId = getTakId();</span><br><span class="line"><span class="keyword">var</span> riskControl = $(<span class="string">'#riskControl'</span>).val();</span><br><span class="line"><span class="built_in">console</span>.log(<span class="string">`eid = <span class="subst">$&#123;eid&#125;</span>\nfp = <span class="subst">$&#123;fp&#125;</span>\ntrack_id = <span class="subst">$&#123;trackId&#125;</span>\nrisk_control = <span class="subst">$&#123;riskControl&#125;</span>`</span>);</span><br></pre></td></tr></table></figure></li><li><p>cookies验证-通过读取本地保存的cookies时要先验证下cookies是否有效：requests库的使用方式</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">_validate_cookies</span><span class="params">(self)</span>:</span></span><br><span class="line">    <span class="string">"""验证cookies是否有效（是否登陆）</span></span><br><span class="line"><span class="string">    通过访问用户订单列表页进行判断：若未登录，将会重定向到登陆页面。</span></span><br><span class="line"><span class="string">    :return: cookies是否有效 True/False</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    url = <span class="string">'https://order.jd.com/center/list.action'</span></span><br><span class="line">    payload = &#123;</span><br><span class="line">        <span class="string">'rid'</span>: str(int(time.time() * <span class="number">1000</span>)),</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="keyword">try</span>:</span><br><span class="line">        resp = self.sess.get(url=url, params=payload, allow_redirects=<span class="keyword">False</span>)</span><br><span class="line">        <span class="keyword">if</span> resp.status_code == requests.codes.OK:</span><br><span class="line">            <span class="keyword">return</span> <span class="keyword">True</span></span><br><span class="line">    <span class="keyword">except</span> Exception <span class="keyword">as</span> e:</span><br><span class="line">        logger.error(e)</span><br><span class="line">    <span class="comment"># 创建新的session</span></span><br><span class="line">    self.sess = requests.session()</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">False</span></span><br></pre></td></tr></table></figure></li><li><p>二维码登录：下载二维码<code>_get_QRcode</code>–&gt;询问获取二维码扫描状态<code>_get_QRcode_ticket</code>—&gt;验证二维码信息(<code>_validate_QRcode_ticket</code>)–&gt;设置<code>is_login=True和save_cookies</code>（二维码有效期约120s，浏览器大约2s发送一个请求询问扫描状态）——看代码是assistant参考了2018年的**<a href="https://github.com/adyzng/jd-autobuy" target="_blank" rel="noopener">jd-autobuy</a>**</p></li><li><p>扫描二维码：写入图片和打开图片</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">save_image</span><span class="params">(resp, image_file)</span>:</span></span><br><span class="line">    <span class="keyword">with</span> open(image_file, <span class="string">'wb'</span>) <span class="keyword">as</span> f:</span><br><span class="line">        <span class="comment"># 从网络请求中下载内容</span></span><br><span class="line">        <span class="keyword">for</span> chunk <span class="keyword">in</span> resp.iter_content(chunk_size=<span class="number">1024</span>):</span><br><span class="line">            f.write(chunk)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">open_image</span><span class="params">(image_file)</span>:</span></span><br><span class="line">    <span class="keyword">if</span> os.name == <span class="string">"nt"</span>:</span><br><span class="line">        os.system(<span class="string">'start '</span> + image_file)  <span class="comment"># for Windows</span></span><br><span class="line">    <span class="keyword">else</span>:</span><br><span class="line">        <span class="keyword">if</span> os.uname()[<span class="number">0</span>] == <span class="string">"Linux"</span>:</span><br><span class="line">            <span class="keyword">if</span> <span class="string">"deepin"</span> <span class="keyword">in</span> os.uname()[<span class="number">2</span>]:</span><br><span class="line">                os.system(<span class="string">"deepin-image-viewer "</span> + image_file)  <span class="comment"># for deepin</span></span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                os.system(<span class="string">"eog "</span> + image_file)  <span class="comment"># for Linux</span></span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            os.system(<span class="string">"open "</span> + image_file)  <span class="comment"># for Mac</span></span><br></pre></td></tr></table></figure><p>注：判断操作系统还可以借助内置的platform库<code>platform.system().lower()</code>得到[linux、windows、mac]</p></li></ul><p>▲cookies和二维码结合使用：</p><ol><li>如果登陆成功则吧cookies持久化</li><li>使用二维码登陆前，先加载验证之前持久化的cookies是否有效，如果有效则无需再次扫码登陆(在QRCode_login的起始位置判断validation中修改的is_login是否为True)</li></ol><h3 id="jd-seckill-maotai"><a class="markdownIt-Anchor" href="#jd-seckill-maotai"></a> <strong><a href="https://github.com/hello8817/jd-seckill-maotai" target="_blank" rel="noopener"> jd-seckill-maotai</a></strong></h3><blockquote><ul><li><h2 id="主要功能"><a class="markdownIt-Anchor" href="#主要功能"></a> 主要功能</h2><ul><li>登陆京东商城（<a href="http://www.jd.com" target="_blank" rel="noopener">www.jd.com</a>）<ul><li>用京东APP扫码给出的二维码</li></ul></li><li>预约茅台<ul><li>定时自动预约</li></ul></li><li>秒杀预约后等待抢购<ul><li>定时开始自动抢购</li></ul></li></ul></li></ul><p><strong>特别声明：</strong></p><ul><li>本仓库发布的<code>jd_seckill_maotai</code>项目中涉及的任何脚本，仅用于测试和学习研究，禁止用于商业用途，不能保证其合法性，准确性，完整性和有效性，请根据情况自行判断。</li><li>本项目内所有资源文件，禁止任何公众号、自媒体进行任何形式的转载、发布。</li><li><code>ChinaVolvocars</code> 对任何脚本问题概不负责，包括但不限于由任何脚本错误导致的任何损失或损害.</li><li>请勿将<code>jd_seckill_maotai</code>项目的任何内容用于商业或非法目的，否则后果自负。</li><li>如果任何单位或个人认为该项目的脚本可能涉嫌侵犯其权利，则应及时通知并提供身份证明，所有权证明，我们将在收到认证文件后删除相关脚本。</li><li>您必须在下载后的24小时内从计算机或手机中完全删除以上内容。</li><li>本项目遵循<code>GPL-3.0 License</code>协议，如果本特别声明与<code>GPL-3.0 License</code>协议有冲突之处，以本特别声明为准。</li><li><em><strong>您使用或者复制了本仓库且本人制作的任何代码或项目，则视为<code>已接受</code>此声明，请仔细阅读</strong></em><br><em><strong>您在本声明未发出之时点使用或者复制了本仓库且本人制作的任何代码或项目且此时还在使用，则视为<code>已接受</code>此声明，请仔细阅读</strong></em></li></ul></blockquote><blockquote><p>根据12月14日以来抢茅台的日志分析，大胆推断再接再厉返回Json消息中<code>resultCode</code>与<strong>小白信用</strong>的关系。<br>这里主要分析出现频率最高的<code>90016</code>和<code>90008</code>。</p><table><thead><tr><th>案例</th><th>小白信用</th><th>90016</th><th>90008</th><th>抢到耗时</th></tr></thead><tbody><tr><td>张三</td><td>63.8</td><td>59.63%</td><td>40.37%</td><td>暂未抢到</td></tr><tr><td>李四</td><td>92.9</td><td>72.05%</td><td>27.94%</td><td>4天</td></tr><tr><td>王五</td><td>99.6</td><td>75.70%</td><td>24.29%</td><td>暂未抢到</td></tr><tr><td>赵六</td><td>103.4</td><td>91.02%</td><td>8.9%</td><td>2天</td></tr></tbody></table><p>风控放行后才会进行抢购，这时候用的应该是水库计数模型，假设无法一次性拿到所有数据的情况下来尽量的做到抢购成功用户的均匀分布，这样就和概率相关了。</p><p><strong>抢购结果确认</strong></p><p>抢购是否成功通常在程序开始的一分钟内可见分晓！<br>搜索日志，出现“抢购成功，订单号xxxxx&quot;，代表成功抢到了，务必半小时内支付订单！程序暂时不支持自动停止，需要手动STOP！<br>若两分钟还未抢购成功，基本上就是没抢到！程序暂时不支持自动停止，需要手动STOP！</p></blockquote><ul><li><p>在程序开始运行后，会检测本地时间与京东服务器时间，输出的差值为【本地时间-京东服务器时间】，即-50为本地时间比京东服务器时间慢50ms。 本代码的执行的抢购时间以本地电脑/服务器时间为准</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, ...)</span>:</span></span><br><span class="line">    <span class="comment"># 初始化误差</span></span><br><span class="line">self.diff_time = self.local_time() - self.jd_time()</span><br><span class="line"><span class="comment"># ....</span></span><br><span class="line"><span class="keyword">while</span> <span class="keyword">True</span>:</span><br><span class="line">    <span class="comment"># 本地时间减去与京东的时间差(ms)，能够将时间误差提升到0.1秒附近, 具体精度依赖获取京东服务器时间的网络时间损耗</span></span><br><span class="line">    <span class="keyword">if</span> self.local_time() - self.diff_time &gt;= self.buy_time_ms:</span><br><span class="line">        logger.info(<span class="string">'时间到达，开始执行……'</span>)</span><br><span class="line">        <span class="keyword">break</span></span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            time.sleep(self.sleep_interval)</span><br></pre></td></tr></table></figure><p>补充：Python的时间戳默认都是秒级别的，如果要ms级别的，则要乘以1000</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">通过time，以元组（struct_time）为核心实现时间戳和格式化时间字符串的相互转换。</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="comment"># 获得10位秒级时间戳</span></span><br><span class="line">seconds = time.time()</span><br><span class="line"><span class="comment"># 获得13位毫秒级时间戳==&gt;java默认13位</span></span><br><span class="line">millis = int(round(time.time() * <span class="number">1000</span>))</span><br><span class="line"><span class="comment"># 同样，time下的mktime也需要*1000</span></span><br><span class="line">time.mktime(self.buy_time.timetuple()) * <span class="number">1000.0</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">通过datetime，以datetime类实例对象为核心实现时间戳和格式化时间字符串的相互转换。</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="keyword">from</span> datetime <span class="keyword">import</span> datetime</span><br><span class="line"><span class="comment"># 获得当前时间</span></span><br><span class="line">now = datetime.now()</span><br><span class="line">timestamp = datetime.timestamp(now)</span><br><span class="line">print(<span class="string">"时间戳 ="</span>, timestamp)</span><br><span class="line"><span class="comment"># 也是10位s级</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="number">1648622589.624272</span></span><br></pre></td></tr></table></figure></li></ul><h3 id="jd_mask"><a class="markdownIt-Anchor" href="#jd_mask"></a> <strong><a href="https://github.com/Lasx/jd_mask" target="_blank" rel="noopener">jd_mask</a></strong></h3><blockquote><p>此tool只单独支持<strong>预约-抢购-成功后直接提交订单</strong>的商品，如[口罩]，只提供学习参考用途.</p></blockquote><p>Ⓜ️阅读了多份京东抢购的代码后，发现基本都是从一个模板里不断优化发展出来的，反应了开源的意义，也反应了京东部分细节的修改，同时也能得到一个告示：在实现相同功能前，不妨多看看、借鉴下以往可行的代码。</p><h3 id="automatic_ticket_purchase"><a class="markdownIt-Anchor" href="#automatic_ticket_purchase"></a> <strong><a href="https://github.com/MakiNaruto/Automatic_ticket_purchase" target="_blank" rel="noopener">Automatic_ticket_purchase</a></strong></h3><p>大麦网抢购流程：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">graph LR</span><br><span class="line">A[开始] --&gt; B(登陆)</span><br><span class="line">    B --&gt; C&#123;是否利用cookies登陆&#125;</span><br><span class="line">    C --&gt; |否| D[页面登陆]</span><br><span class="line">    C --&gt; |是| E[登陆验证]</span><br><span class="line">    D --&gt; E</span><br><span class="line">    E --&gt; G&#123;是否登陆成功&#125;</span><br><span class="line">    G --&gt; |是| F[获取购票必要信息]</span><br><span class="line">    G --&gt; |否|ED</span><br><span class="line">    F --&gt; L[检测抢票现在购票状态]</span><br><span class="line">    L --&gt; M&#123;目标票可进行抢购&#125;</span><br><span class="line">    M --&gt; |即将开抢|L</span><br><span class="line">    M --&gt; |缺货登记|ED((结束))</span><br><span class="line">M --&gt; |立刻购买|N&#123;抢购&#125;</span><br></pre></td></tr></table></figure><blockquote><p>跟限时、定时秒杀不同的是，大麦网的抢购信息是靠不断刷新的，需要通过不断检测页面内容，判断目前票的可否购买情况。如果刷新为可购买则进行抢购购买。</p></blockquote><ul><li><p>点击购买是对指定接口发送请求，并更新cookies(cookies嫌少不嫌多)：step2_click_buy_now</p></li><li><p>提交订单：step3_submit_order</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">response = self.session.post(<span class="string">'https://buy.damai.cn/multi/trans/createOrder'</span>,</span><br><span class="line">                             headers=headers,</span><br><span class="line">                             params=params,</span><br><span class="line">                             data=submit_order_info,</span><br><span class="line">                             cookies=self.login_cookies)</span><br><span class="line">buy_status = json.loads(response.text)</span><br><span class="line"><span class="keyword">if</span> buy_status.get(<span class="string">'success'</span>) <span class="keyword">is</span> <span class="keyword">True</span> <span class="keyword">and</span> buy_status.get(<span class="string">'module'</span>).get(<span class="string">'alipayOrderId'</span>):</span><br><span class="line">    print(<span class="string">'-'</span> * <span class="number">10</span>, <span class="string">'抢票成功, 请前往 大麦网-&gt;我的大麦-&gt;交易中心-&gt;订单管理 确认订单'</span>, <span class="string">'-'</span> * <span class="number">10</span>)</span><br><span class="line">    print(<span class="string">'alipayOrderId: '</span>, buy_status.get(<span class="string">'module'</span>).get(<span class="string">'alipayOrderId'</span>))</span><br><span class="line">    print(<span class="string">'支付宝支付链接: '</span>, buy_status.get(<span class="string">'module'</span>).get(<span class="string">'alipayWapCashierUrl'</span>))</span><br></pre></td></tr></table></figure></li></ul><h3 id="nikerobot"><a class="markdownIt-Anchor" href="#nikerobot"></a> <strong><a href="https://github.com/cxiaoer/NikeRobot" target="_blank" rel="noopener"> NikeRobot</a></strong></h3><ul><li><p>pdb调试断点：==&gt;实际上Pycharm的调试更加好用</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> DEBUG:</span><br><span class="line">pdb.set_trace()</span><br></pre></td></tr></table></figure></li><li><p>将类成员属性转成字典形式: <code>NikeLoginParam().__dict__</code></p></li><li><p>自己管理线程：创建继承Thread的执行类，重写run方法，在main中进行创建多个实例MyThread().start()，并join住。</p></li></ul><h3 id="hpv4g"><a class="markdownIt-Anchor" href="#hpv4g"></a> <strong><a href="https://github.com/xiangmingzhe0928/hpv4g" target="_blank" rel="noopener">hpv4g</a>★</strong></h3><ul><li><p>是否使用代理：<code>init_ip_proxy_pool</code></p></li><li><p>多线程参数设置: python3.8中ThreadPoolExecutor的默认worker：<code>max_workers = min(32, os.cpu_count() + 4)</code></p></li><li><p>线程池：ThreadPoolExecutor</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> concurrent.futures <span class="keyword">import</span> ThreadPoolExecutor, wait, FIRST_COMPLETED</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> ThreadPoolExecutor(max_workers=max_workers) <span class="keyword">as</span> t:</span><br><span class="line">    fs = [t.submit(sec_kill_task, miao_miao, params[i % _params_len],</span><br><span class="line">                   <span class="keyword">None</span> <span class="keyword">if</span> <span class="keyword">not</span> _ip_proxys_len <span class="keyword">else</span> &#123;</span><br><span class="line">                       <span class="string">'http'</span>: <span class="keyword">None</span> <span class="keyword">if</span> (index := i % _ip_proxys_len) == <span class="number">0</span> <span class="keyword">else</span> ip_proxys[index]&#125;) <span class="keyword">for</span> i <span class="keyword">in</span></span><br><span class="line">          range(max_workers + <span class="number">5</span>)]</span><br><span class="line">    <span class="comment"># 120S后结束任务, 有一个完成则完成</span></span><br><span class="line">    wait(fs, <span class="number">120</span>, return_when=FIRST_COMPLETED)</span><br><span class="line">    <span class="keyword">global</span> KILL_FLAG</span><br><span class="line">    KILL_FLAG = <span class="keyword">True</span></span><br><span class="line">    print(<span class="string">'&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;本轮未成功秒杀到疫苗&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;'</span>)</span><br></pre></td></tr></table></figure></li><li><p>缓存配置——非默认参数的装饰器：考虑提前缓存疫苗列表 秒杀开始后跳过查询列表等操作 直接调用秒杀接口</p></li><li><p>获取服务器当前时间戳</p></li><li><p>直接设置logging中的log</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> logging</span><br><span class="line"><span class="comment"># 设置配置</span></span><br><span class="line">logging.basicConfig(handlers=[logging.FileHandler(filename=LOG_NAME,</span><br><span class="line">                                                  encoding=<span class="string">'utf-8'</span>, mode=<span class="string">'a+'</span>)],</span><br><span class="line">                    format=<span class="string">'%(asctime)s %(message)s'</span>,</span><br><span class="line">                    level=getattr(logging, args.log))</span><br><span class="line"><span class="comment"># 使用</span></span><br><span class="line">logging.info(<span class="string">"xxx"</span>)</span><br><span class="line">logging.debug(<span class="string">"yyy"</span>)</span><br></pre></td></tr></table></figure></li><li><p>argueparser使用：</p><ul><li><p>action参数: <code>parser.add_argument('-reload_cache', action='store_true', help='刷新--region_code疫苗列表本地缓存')</code>，出现reload_cache参数则为true，所以默认为false</p></li><li><p><code>parser.add_argument('--log', default='WARNING', choices=['DEBUG', 'INFO', 'WARNING', 'ERROR'],help='日志级别 默认WARNING')</code></p></li><li><p>add_argument中的type实际上是个类型转化器，如type=int，如果无法转成int则报错，因此可以借此参数对输入进行约束</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">_valid_int_type</span><span class="params">(i)</span>:</span></span><br><span class="line">    valid_int = int(i)</span><br><span class="line">    <span class="keyword">if</span> valid_int &lt; <span class="number">1</span>:</span><br><span class="line">        <span class="keyword">raise</span> argparse.ArgumentTypeError(<span class="string">f'invalid int argument:<span class="subst">&#123;i&#125;</span>'</span>)</span><br><span class="line">    <span class="keyword">return</span> valid_int</span><br></pre></td></tr></table></figure></li></ul></li><li><p>requests使用：</p><ul><li><p><code>raise_for_status</code>状态码非成功时抛出异常</p></li><li><p>自定义<code>_get(url, params=None, error_exit=True, **kwargs)</code>方法，添加参数：error_exit:返回4XX 5XX错误时 是否退出</p></li><li><p>cookies除了放在requests.get(cookies=xxxCookieDict)，还可以作为字符串设置在header[“<strong>cookie</strong>”]中（无s），如<code>self._headers[&quot;cookie&quot;]=&quot;k1=v1;_xxhm_=%7B%2x%7D;k3=wxapptoken:v3&quot;</code>，通过<code>;</code>来分割</p></li><li><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># disable ssl warnings</span></span><br><span class="line">requests.packages.urllib3.disable_warnings()</span><br></pre></td></tr></table></figure></li></ul></li></ul><h3 id="抢购型代码总结"><a class="markdownIt-Anchor" href="#抢购型代码总结"></a> 抢购型代码总结：</h3><ul><li><p>实现主要分为selenium(自动测试工具)模拟点击；api模拟网络请求</p></li><li><p>登陆之后会对cookies进行持久化，使用pickle的好处是无法明文看懂、复制</p></li><li><p>可以通过持久化的cookies来登陆，但会检测cookies是否仍然有效</p></li><li><p>定时抢购会等待sleep到临近时间(while True+sleep(time)的方式 or sleepUntil+while try)，然后再进行多次抢购请求（如果抢购的2分钟内没抢到，大多没希望了，只好等待下次）；如果是<a href="#Automatic_ticket_purchase">等待页面刷新的</a>，则不断请求页面，等到结果改变。</p><ul><li>注意：等待间隔不要超过时间阈值，如180s前检查，如果不符合不能让其等待190s</li></ul></li><li><p>推送工具：提供一个自己写的</p><p>config.ini</p><figure class="highlight ini"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="section">[pusher]</span></span><br><span class="line"><span class="attr">pusher_type</span> = pushplus</span><br><span class="line"><span class="attr">push_title</span> = 飞鱼秒杀</span><br><span class="line"></span><br><span class="line"><span class="comment"># https://sct.ftqq.com/</span></span><br><span class="line"><span class="section">[serverchan]</span></span><br><span class="line"><span class="attr">sec_key</span> = SCU35113Te369cebc21f6e483c03fffc400c4c5c05bdad63995c32</span><br><span class="line"></span><br><span class="line"><span class="comment"># https://open.dingtalk.com/document/group/custom-robot-access</span></span><br><span class="line"><span class="section">[dingding]</span></span><br><span class="line"><span class="attr">access_token</span> =</span><br><span class="line"><span class="attr">secret</span> =</span><br><span class="line"></span><br><span class="line"><span class="comment"># http://www.pushplus.plus/</span></span><br><span class="line"><span class="section">[pushplus]</span></span><br><span class="line"><span class="attr">pushplus_token</span> = fedcf6b08f6f4aaeb2948b7d7010eb93</span><br></pre></td></tr></table></figure></li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> base64</span><br><span class="line"><span class="keyword">import</span> hashlib</span><br><span class="line"><span class="keyword">import</span> hmac</span><br><span class="line"><span class="keyword">import</span> os.path</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> urllib.parse</span><br><span class="line"><span class="keyword">from</span> functools <span class="keyword">import</span> reduce</span><br><span class="line"><span class="keyword">from</span> pathlib <span class="keyword">import</span> Path</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">__author__ = <span class="string">'Mrli'</span></span><br><span class="line"></span><br><span class="line">CONFIG_INI_FILENAME = <span class="string">"push_config.ini"</span></span><br><span class="line">CONFIG_INI_PATH = Path(__file__).resolve().parent.with_name(<span class="string">"push_config.ini"</span>)</span><br><span class="line"></span><br><span class="line">REMIND_MSG = <span class="string">"""</span></span><br><span class="line"><span class="string">[pusher]</span></span><br><span class="line"><span class="string">pusher_type = pushplus</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">[serverchan]</span></span><br><span class="line"><span class="string">sec_key =</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">[dingding]</span></span><br><span class="line"><span class="string">access_token =</span></span><br><span class="line"><span class="string">secret =</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">[pushplus]</span></span><br><span class="line"><span class="string">pushplus_token = </span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">PusherException</span><span class="params">(Exception)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, message)</span> -&gt; <span class="keyword">None</span>:</span></span><br><span class="line">        super().__init__(message)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">IPushUtil</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">push</span><span class="params">(self, content: str, title: str = <span class="string">""</span>)</span> -&gt; bool:</span></span><br><span class="line">        <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">PushplusPush</span><span class="params">(IPushUtil)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, push_title, options: dict)</span>:</span></span><br><span class="line">        self.pushplus_token = options.get(<span class="string">"pushplus_token"</span>)</span><br><span class="line">        self.push_title = push_title</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">push</span><span class="params">(self, content: str, title: str = <span class="string">""</span>)</span> -&gt; bool:</span></span><br><span class="line">        d = &#123;</span><br><span class="line">            <span class="string">"token"</span>: self.pushplus_token,</span><br><span class="line">            <span class="string">"template"</span>: <span class="string">"markdown"</span>,</span><br><span class="line">            <span class="string">"title"</span>: <span class="string">"&#123;push_title&#125;-&#123;title&#125;"</span>.format(push_title=self.push_title, title=title),</span><br><span class="line">            <span class="string">"content"</span>: content</span><br><span class="line">        &#125;</span><br><span class="line">        res = requests.post(<span class="string">"http://www.pushplus.plus/send"</span>, data=d)</span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> (<span class="number">200</span> &lt;= res.json().get(<span class="string">"code"</span>) &lt; <span class="number">300</span>):</span><br><span class="line">            print(res.json())</span><br><span class="line">        <span class="keyword">return</span> <span class="number">200</span> &lt;= res.json().get(<span class="string">"code"</span>) &lt; <span class="number">300</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ServerChanPush</span><span class="params">(IPushUtil)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, push_title, options: dict)</span>:</span></span><br><span class="line">        self.sec_key = options.get(<span class="string">"sec_key"</span>)</span><br><span class="line">        self.push_title = push_title</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">push</span><span class="params">(self, content: str, title: str = <span class="string">""</span>)</span> -&gt; bool:</span></span><br><span class="line">        data = &#123;</span><br><span class="line">            <span class="string">'text'</span>: <span class="string">"&#123;push_title&#125;-&#123;title&#125;"</span>.format(push_title=self.push_title, title=title),</span><br><span class="line">            <span class="string">'desp'</span>: content</span><br><span class="line">        &#125;</span><br><span class="line">        res = requests.post(url=<span class="string">'https://sc.ftqq.com/&#123;&#125;.send'</span>.format(self.sec_key), data=data)</span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> (res.json().get(<span class="string">"errmsg"</span>) == <span class="string">"success"</span>):</span><br><span class="line">            print(res.json())</span><br><span class="line">        <span class="keyword">return</span> res.json().get(<span class="string">"errmsg"</span>) == <span class="string">"success"</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DingDingPush</span><span class="params">(IPushUtil)</span>:</span></span><br><span class="line">    URL = <span class="string">"https://oapi.dingtalk.com/robot/send"</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, push_title, options: dict)</span>:</span></span><br><span class="line">        self.access_token = options.get(<span class="string">"access_token"</span>)</span><br><span class="line">        self.secret = options.get(<span class="string">"secret"</span>)</span><br><span class="line">        self.target_url = self.get_url()</span><br><span class="line">        self.push_title = push_title</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">get_url</span><span class="params">(self)</span>:</span></span><br><span class="line">        timestamp = round(time.time() * <span class="number">1000</span>)</span><br><span class="line">        secret_enc = bytes(self.secret, encoding=<span class="string">"utf-8"</span>)</span><br><span class="line">        string_to_sign = <span class="string">"&#123;&#125;\n&#123;&#125;"</span>.format(timestamp, self.secret)</span><br><span class="line">        string_to_sign_enc = bytes(string_to_sign, encoding=<span class="string">"utf-8"</span>)</span><br><span class="line">        hmac_code = hmac.new(</span><br><span class="line">            secret_enc, string_to_sign_enc, digestmod=hashlib.sha256</span><br><span class="line">        ).digest()</span><br><span class="line">        sign = urllib.parse.quote_plus(base64.b64encode(hmac_code))</span><br><span class="line">        <span class="keyword">return</span> self.URL + <span class="string">"?access_token=&#123;access_token&#125;&amp;timestamp=&#123;timestamp&#125;&amp;sign=&#123;sign&#125;"</span>.format(</span><br><span class="line">            access_token=self.access_token, timestamp=timestamp, sign=sign)</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">push</span><span class="params">(self, content: str, title: str = <span class="string">""</span>)</span> -&gt; bool:</span></span><br><span class="line">        msg = self.gen_markdown_msg(title, content)</span><br><span class="line">        <span class="keyword">return</span> self.send(msg)</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">send</span><span class="params">(self, message)</span>:</span></span><br><span class="line">        resp = requests.post(self.target_url, json=message)</span><br><span class="line">        <span class="keyword">return</span> resp.json()</span><br><span class="line"></span><br><span class="line"><span class="meta">    @staticmethod</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">gen_text_msg</span><span class="params">(content, at=None, at_all=False)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> at <span class="keyword">is</span> <span class="keyword">None</span>:</span><br><span class="line">            at = []</span><br><span class="line">        <span class="keyword">return</span> &#123;</span><br><span class="line">            <span class="string">"msgtype"</span>: <span class="string">"text"</span>,</span><br><span class="line">            <span class="string">"text"</span>: &#123;<span class="string">"content"</span>: content&#125;,</span><br><span class="line">            <span class="string">"at"</span>: &#123;<span class="string">"atMobiles"</span>: at, <span class="string">"isAtAll"</span>: at_all&#125;,</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">gen_markdown_msg</span><span class="params">(self, title, text, at=None, at_all=False)</span>:</span></span><br><span class="line">        <span class="function"><span class="keyword">def</span> <span class="title">generateText</span><span class="params">()</span>:</span></span><br><span class="line">            res = <span class="string">""</span></span><br><span class="line">            <span class="comment"># 最顶行显示标题</span></span><br><span class="line">            res += <span class="string">"# "</span> + <span class="string">"&#123;&#125;-"</span>.format(self.push_title) + title + <span class="string">"\n"</span></span><br><span class="line">            <span class="comment"># 内容</span></span><br><span class="line">            res += text</span><br><span class="line">            <span class="comment"># at对象</span></span><br><span class="line">            res += reduce(<span class="keyword">lambda</span> x, y: x + <span class="string">"@"</span> + y, at, <span class="string">""</span>)</span><br><span class="line">            <span class="keyword">return</span> res</span><br><span class="line"></span><br><span class="line">        <span class="keyword">return</span> &#123;</span><br><span class="line">            <span class="string">"msgtype"</span>: <span class="string">"markdown"</span>,</span><br><span class="line">            <span class="string">"markdown"</span>: &#123;</span><br><span class="line">                <span class="string">"title"</span>: title,</span><br><span class="line">                <span class="string">"text"</span>: generateText()</span><br><span class="line">            &#125;,</span><br><span class="line">            <span class="string">"at"</span>: &#123;<span class="string">"atMobiles"</span>: at, <span class="string">"isAtAll"</span>: at_all&#125;,</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Pusher</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, logger=None)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> logger:</span><br><span class="line">            self.cout = logger.info</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            self.cout = <span class="keyword">print</span></span><br><span class="line">        self._pusher = self.init()</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">init</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="string">"""</span></span><br><span class="line"><span class="string">        实例化pusher</span></span><br><span class="line"><span class="string">        :return:</span></span><br><span class="line"><span class="string">        """</span></span><br><span class="line">        <span class="keyword">from</span> configparser <span class="keyword">import</span> RawConfigParser</span><br><span class="line">        cp = RawConfigParser()</span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> os.path.exists(CONFIG_INI_PATH):</span><br><span class="line">            <span class="keyword">raise</span> PusherException(</span><br><span class="line">                <span class="string">"请创建&#123;filename&#125;配置文件\npusher配置信息如下:\n&#123;msg&#125;"</span>.format(filename=CONFIG_INI_FILENAME, msg=REMIND_MSG))</span><br><span class="line">        cp.read(CONFIG_INI_PATH, encoding=<span class="string">"utf8"</span>)</span><br><span class="line">        pusher_type = cp.get(<span class="string">"pusher"</span>, <span class="string">"pusher_type"</span>).lower()</span><br><span class="line">        push_title = cp.get(<span class="string">"pusher"</span>, <span class="string">"push_title"</span>)</span><br><span class="line">        <span class="comment"># 是否使用了pusher</span></span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> pusher_type:</span><br><span class="line">            self.cout(<span class="string">"初始化Pusher: 当前未配置Pusher, 如果需要推送功能, 则在&#123;filename&#125;"</span>.format(filename=CONFIG_INI_FILENAME))</span><br><span class="line">            <span class="keyword">return</span> <span class="keyword">None</span></span><br><span class="line"></span><br><span class="line">        generator_info = dict(cp.items(pusher_type))</span><br><span class="line">        <span class="comment"># 检查pusher配置</span></span><br><span class="line">        <span class="keyword">if</span> pusher_type <span class="keyword">and</span> <span class="keyword">not</span> self._valid(generator_info):</span><br><span class="line">            <span class="keyword">raise</span> PusherException(<span class="string">"&#123;&#125;_pusher配置错误，不能为空~"</span>.format(pusher_type))</span><br><span class="line"></span><br><span class="line">        <span class="keyword">if</span> pusher_type == <span class="string">"serverchan"</span>:</span><br><span class="line">            <span class="keyword">return</span> ServerChanPush(push_title, generator_info)</span><br><span class="line">        <span class="keyword">elif</span> pusher_type == <span class="string">"dingding"</span>:</span><br><span class="line">            <span class="keyword">return</span> DingDingPush(push_title, generator_info)</span><br><span class="line">        <span class="keyword">elif</span> pusher_type == <span class="string">"pushplus"</span>:</span><br><span class="line">            <span class="keyword">return</span> PushplusPush(push_title, generator_info)</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            <span class="keyword">raise</span> PusherException(<span class="string">"不可知pusher类型~"</span>)</span><br><span class="line"></span><br><span class="line"><span class="meta">    @staticmethod</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">_valid</span><span class="params">(config_dict: dict)</span>:</span></span><br><span class="line">        <span class="string">"""</span></span><br><span class="line"><span class="string">        判断字典值是否为空</span></span><br><span class="line"><span class="string">        :param dict:</span></span><br><span class="line"><span class="string">        :return:</span></span><br><span class="line"><span class="string">        """</span></span><br><span class="line">        <span class="keyword">for</span> v <span class="keyword">in</span> config_dict.values():</span><br><span class="line">            <span class="keyword">if</span> <span class="keyword">not</span> v:</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">False</span></span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">True</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">push</span><span class="params">(self, content: str, title: str = <span class="string">""</span>)</span> -&gt; bool:</span></span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> self._pusher:</span><br><span class="line">            self.cout(<span class="string">"当前未配置pusher, 消息无法发送"</span>)</span><br><span class="line">            <span class="keyword">return</span> <span class="keyword">False</span></span><br><span class="line">        <span class="keyword">return</span> self._pusher.push(content, title)</span><br></pre></td></tr></table></figure><h2 id="weibospider"><a class="markdownIt-Anchor" href="#weibospider"></a> <a href="https://github.com/dataabc/weiboSpider" target="_blank" rel="noopener">weiboSpider</a></h2><blockquote><p>可以连续爬取<strong>一个</strong>或<strong>多个</strong>新浪微博用户（如<a href="https://weibo.cn/u/1223178222" target="_blank" rel="noopener">胡歌</a>、<a href="https://weibo.cn/u/1669879400" target="_blank" rel="noopener">迪丽热巴</a>、<a href="https://weibo.cn/u/1729370543" target="_blank" rel="noopener">郭碧婷</a>）的数据，并将结果信息写入<strong>文件</strong>或<strong>数据库</strong>。</p></blockquote><ul><li>①允许用户指定自定义配置文件，否则使用默认配置文件；②对配置文件进行校验，并对非法项进行提示</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">config = _get_config()</span><br><span class="line">    config_util.validate_config(config)</span><br><span class="line">        </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">_get_config</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""获取config.json数据"""</span></span><br><span class="line">    src = os.path.split(</span><br><span class="line">        os.path.realpath(__file__))[<span class="number">0</span>] + os.sep + <span class="string">'config_sample.json'</span></span><br><span class="line">    config_path = os.getcwd() + os.sep + <span class="string">'config.json'</span></span><br><span class="line">    <span class="keyword">if</span> FLAGS.config_path:</span><br><span class="line">        config_path = FLAGS.config_path</span><br><span class="line">        </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">validate_config</span><span class="params">(config)</span>:</span></span><br><span class="line">    <span class="string">"""验证配置是否正确"""</span></span><br><span class="line"></span><br><span class="line">    <span class="comment"># 验证filter、pic_download、video_download</span></span><br><span class="line">    argument_list = [<span class="string">'filter'</span>, <span class="string">'pic_download'</span>, <span class="string">'video_download'</span>]</span><br><span class="line">    <span class="keyword">for</span> argument <span class="keyword">in</span> argument_list:</span><br><span class="line">        <span class="keyword">if</span> config[argument] != <span class="number">0</span> <span class="keyword">and</span> config[argument] != <span class="number">1</span>:</span><br><span class="line">            logger.warning(<span class="string">u'%s值应为0或1,请重新输入'</span>, config[argument])</span><br><span class="line">            sys.exit()</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 验证since_date</span></span><br><span class="line">    since_date = config[<span class="string">'since_date'</span>]</span><br><span class="line">    <span class="keyword">if</span> (<span class="keyword">not</span> _is_date(str(since_date))) <span class="keyword">and</span> (<span class="keyword">not</span> isinstance(since_date, int)):</span><br><span class="line">        logger.warning(<span class="string">u'since_date值应为yyyy-mm-dd形式或整数,请重新输入'</span>)</span><br><span class="line">        sys.exit()</span><br></pre></td></tr></table></figure><h2 id="proxy_pool"><a class="markdownIt-Anchor" href="#proxy_pool"></a> <strong><a href="https://github.com/jhao104/proxy_pool" target="_blank" rel="noopener">proxy_pool</a>★</strong></h2><blockquote><p>爬虫代理IP池项目,主要功能为定时采集网上发布的免费代理验证入库，定时验证入库的代理保证代理的可用性，提供API和CLI两种使用方式。同时你也可以扩展代理源以增加代理池IP的质量和数量。</p><p>模块组成：获取功能、存储功能、校验功能、接口管理</p><p>⭐️程序主要是启动了<code>startServer</code>的API接口服务、<code>startScheduler</code>定时服务</p><ul><li><code>startServer-&gt;runFlask</code>：是向外提供了通过<strong>proxyHandler</strong>来获得Redis中的proxy数据</li><li><code>startScheduler-&gt;sche.add_task(__runProxyFetch)、sche.add_task(__runProxyCheck)</code>，<ul><li><code>__runProxyFetch:proxy_fetcher.run()-&gt;proxy_queue-&gt;Checker(&quot;raw&quot;, proxy_queue)</code>获得各个代理网站的代理信息后，进行校验，校验成功则入库</li><li><code>__runProxyCheck:proxy in proxy_handler.getAll()-&gt;proxy_queue-&gt;Checker(&quot;use&quot;, proxy_queue)</code>：通过proxy_handler拿到库里所有现存的数据后，进行有效性校验，无效的则删除，有效的则更新信息</li></ul></li></ul><p>作为存储功能的接口proxyHandler，也是两个API服务与定时服务的中介。程序也是通过存储功能，将核心的两个功能：<u>定时抓取的proxy数据</u>与<u>提供proxy数据给用户使用</u>成功联系在了一起</p></blockquote><ul><li><p>通过元类实现单例模式：ConfigHandler，其可以在任意模块中以c = ConfigHandler()的形式获得，而不是<code>ConfigHandler.getInstance()</code></p></li><li><p><code>@LazyProperty</code>懒加载属性的装饰器： 只有用到时才会加载并将值注入到<code>__dict__</code>、加载一次后值就不再变化、；讲解可见：<a href="https://www.jianshu.com/p/708dc26f9b92%E2%80%94%E2%80%94%E6%8F%8F%E8%BF%B0%E7%AC%A6or%E4%BF%AE%E9%A5%B0%E7%AC%A6%E5%AE%9E%E7%8E%B0" target="_blank" rel="noopener">https://www.jianshu.com/p/708dc26f9b92——描述符or修饰符实现</a></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">LazyProperty</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="comment"># 在被注解类方法被解释器运行的时候就会创建LazyProperty实例并返回</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, func)</span>:</span></span><br><span class="line">        self.func = func</span><br><span class="line">    <span class="string">"""通过python描述符来实现"""</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__get__</span><span class="params">(self, instance, owner)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> instance <span class="keyword">is</span> <span class="keyword">None</span>:</span><br><span class="line">            <span class="keyword">return</span> self</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            <span class="comment"># 会将结果值通过setattr方法存入到instance对象实例的__dict__中</span></span><br><span class="line">            value = self.func(instance)</span><br><span class="line">            setattr(instance, self.func.__name__, value)</span><br><span class="line">            <span class="keyword">return</span> value</span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span><span class="params">(withMetaclass<span class="params">(Singleton)</span>)</span>:</span></span><br><span class="line">    <span class="comment"># 返回一个LazyProperty实例 </span></span><br><span class="line"><span class="meta">    @LazyProperty</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">serverHost</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> os.environ.get(<span class="string">"HOST"</span>, setting.HOST)</span><br><span class="line">c = ConfigHandler()</span><br><span class="line"><span class="comment"># 会触发ConfigHandler.__dict__["serverHost"], 然后接而触发LazyProperty的__get__，value = self.func(instance)会得到真正serverHost函数的值后将其设置在ConfigHandler instance对象的__dict__中，由于对象的__dict__["serverHost"]=value优先级高于类的__dict__["serverHost"]=LazyProperty()对象，因此之后调用得到的是value结果</span></span><br><span class="line">print(c.serverHost)</span><br></pre></td></tr></table></figure><p><code>__get__</code>只有访问类属性的时候才会生效，这边是通过setattr将serverHost设置成了ConfigHandler的类属性</p></li><li><p>封装了一个请求工具类<code>WebRequest</code>:</p><ul><li>增加了异常处理的功能</li><li>增加了日志功能</li><li>请求头会得到随机UA</li><li>设置重试</li></ul></li><li><p>使用click创建子命令：</p><ol><li>得到一个click_group</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">CONTEXT_SETTINGS = dict(help_option_names=[<span class="string">'-h'</span>, <span class="string">'--help'</span>])</span><br><span class="line"></span><br><span class="line"><span class="meta">@click.group(context_settings=CONTEXT_SETTINGS)</span></span><br><span class="line"><span class="meta">@click.version_option(version=VERSION)</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">cli</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""ProxyPool cli工具"""</span></span><br></pre></td></tr></table></figure><ol start="2"><li>指定group下的子命令</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@cli.command(name="server")</span></span><br><span class="line"><span class="comment"># 还可以设置参数: @click.option('--count', default=1, help='Number of greetings.') --&gt; def server(count)</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">server</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">""" 启动api服务 """</span></span><br><span class="line">    click.echo(BANNER)</span><br><span class="line">    startServer()</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    cli()</span><br></pre></td></tr></table></figure><p>然后通过bash脚本同时开启两个进程</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/usr/bin/env bash</span></span><br><span class="line">python proxyPool.py server &amp;</span><br><span class="line">python proxyPool.py schedule</span><br></pre></td></tr></table></figure></li><li><p>DbClient DB工厂类</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> urllib.parse <span class="keyword">import</span> urlparse</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DbClient</span><span class="params">(withMetaclass<span class="params">(Singleton)</span>)</span>:</span></span><br><span class="line">   <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, db_conn)</span>:</span></span><br><span class="line">        self.parseDbConn(db_conn)</span><br><span class="line">        self.__initDbClient()</span><br><span class="line"></span><br><span class="line"><span class="meta">    @classmethod</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">parseDbConn</span><span class="params">(cls, db_conn)</span>:</span></span><br><span class="line">        <span class="comment"># 可以关注一下urlparse的使用</span></span><br><span class="line">        db_conf = urlparse(db_conn)</span><br><span class="line">        cls.db_type = db_conf.scheme.upper().strip()</span><br><span class="line">        <span class="comment"># _NetlocResultMixinBase对象来获取想要的参数</span></span><br><span class="line">        cls.db_host = db_conf.hostname</span><br><span class="line">        cls.db_port = db_conf.port</span><br><span class="line">        cls.db_user = db_conf.username</span><br><span class="line">        cls.db_pwd = db_conf.password</span><br><span class="line">        cls.db_name = db_conf.path[<span class="number">1</span>:]</span><br><span class="line">        <span class="keyword">return</span> cls</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__initDbClient</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="string">"""</span></span><br><span class="line"><span class="string">        init DB Client</span></span><br><span class="line"><span class="string">        :return:</span></span><br><span class="line"><span class="string">        """</span></span><br><span class="line">        __type = <span class="keyword">None</span></span><br><span class="line">        <span class="keyword">if</span> <span class="string">"SSDB"</span> == self.db_type:</span><br><span class="line">            __type = <span class="string">"ssdbClient"</span></span><br><span class="line">        <span class="keyword">elif</span> <span class="string">"REDIS"</span> == self.db_type:</span><br><span class="line">            __type = <span class="string">"redisClient"</span></span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            <span class="keyword">pass</span></span><br><span class="line">        <span class="keyword">assert</span> __type, <span class="string">'type error, Not support DB type: &#123;&#125;'</span>.format(self.db_type)</span><br><span class="line">        self.client = getattr(__import__(__type), <span class="string">"%sClient"</span> % self.db_type.title())(host=self.db_host,</span><br><span class="line">port=self.db_port,</span><br><span class="line">username=self.db_user,</span><br><span class="line">password=self.db_pwd,</span><br><span class="line">db=self.db_name)</span><br></pre></td></tr></table></figure></li><li><p>python中使用redis</p><blockquote><p>Redis中代理存放的结构为hash：hash  是一个键值(key=&gt;value)对集合；是一个 string 类型的 field（字段） 和 value（值） 的<strong>映射表</strong>，hash 特别适合用于存储对象。</p><p>本项目中应用：key为ip:port, value为代理属性的字典;</p></blockquote><ul><li>python与创建连接</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># pip install redis</span></span><br><span class="line"><span class="keyword">from</span> redis <span class="keyword">import</span> Redis, BlockingConnectionPool</span><br><span class="line">conn_dict = &#123;</span><br><span class="line">    <span class="string">"host"</span>: <span class="string">"49.235.118.244"</span>,</span><br><span class="line">    <span class="string">"port"</span>: <span class="number">6379</span>,</span><br><span class="line">    <span class="string">"db"</span>: <span class="number">0</span></span><br><span class="line">&#125;</span><br><span class="line">r = Redis(connection_pool=BlockingConnectionPool(decode_responses=<span class="keyword">True</span>,</span><br><span class="line">                                                 timeout=<span class="number">5</span>,</span><br><span class="line">                                                 socket_timeout=<span class="number">15</span>,</span><br><span class="line">                                                 **conn_dict)</span><br><span class="line">         )</span><br></pre></td></tr></table></figure><ul><li><a href="https://github.com/jhao104/proxy_pool/blob/master/db/redisClient.py" target="_blank" rel="noopener">★hash使用封装</a></li></ul><blockquote><p>添加：set、zset为add、list为push、string和hash为set</p></blockquote><p>参考:<a href="https://blog.csdn.net/weixin_39576234/article/details/87603701" target="_blank" rel="noopener">Redis基本概念及使用</a>、<a href="https://www.runoob.com/redis/redis-sorted-sets.html%E3%80%81" target="_blank" rel="noopener">https://www.runoob.com/redis/redis-sorted-sets.html、</a><a href="https://www.runoob.com/w3cnote/python-redis-intro.html" target="_blank" rel="noopener">★Python redis 使用介绍</a></p><p><img src="/2022/03/29/开源项目代码阅读记录/E:%5Chexo%5Csource_posts%5C%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE%E4%BB%A3%E7%A0%81%E9%98%85%E8%AF%BB%E8%AE%B0%E5%BD%95%5Credis.jpg" alt="redis"></p></li><li><p>继承重写logging.logger，可选参数为<code>name, level=DEBUG, stream=True, file=True</code>，让每个功能函数都能生成单独的日志文件，并进行了可选控制。</p><p>相比单例，日志精度更细，但也使用起来也更麻烦，需要考虑什么地方需要。</p></li><li><p>提供&quot;扩展代理&quot;接口</p><ol><li><p>在<a href="https://github.com/jhao104/proxy_pool/blob/1a3666283806a22ef287fba1a8efab7b94e94bac/fetcher/proxyFetcher.py#L21" target="_blank" rel="noopener">ProxyFetcher</a>类中添加自定义的获取代理的静态方法， 该方法需要以生成器(yield)形式返回<code>host:ip</code>格式的代理</p></li><li><p>添加好方法后，修改<a href="https://github.com/jhao104/proxy_pool/blob/1a3666283806a22ef287fba1a8efab7b94e94bac/setting.py#L47" target="_blank" rel="noopener">setting.py</a>文件中的<code>PROXY_FETCHER</code>项下添加自定义方法的名字:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">PROXY_FETCHER = [</span><br><span class="line">    <span class="string">"freeProxy01"</span>,    </span><br><span class="line">    <span class="string">"freeProxy02"</span>,</span><br><span class="line">    <span class="comment"># ....</span></span><br><span class="line">    <span class="string">"freeProxyCustom1"</span>  <span class="comment">#  # 确保名字和你添加方法名字一致</span></span><br><span class="line">]</span><br></pre></td></tr></table></figure><p><code>schedule</code> 进程会每隔一段时间抓取一次代理，下次抓取时会自动识别调用你定义的方法。</p></li></ol><p><a href="https://github.com/jhao104/proxy_pool/blob/master/helper/fetch.py" target="_blank" rel="noopener">实现方式</a>：</p></li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">self.log.info(<span class="string">"ProxyFetch : start"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 从配置中拿执行函数</span></span><br><span class="line"><span class="keyword">for</span> fetch_source <span class="keyword">in</span> self.conf.fetchers:</span><br><span class="line">    <span class="comment"># 判断ProxyFetcher中是否有定义、是否可调用</span></span><br><span class="line">fetcher = getattr(ProxyFetcher, fetch_source, <span class="keyword">None</span>)</span><br><span class="line">    <span class="keyword">if</span> <span class="keyword">not</span> fetcher:</span><br><span class="line">self.log.error(<span class="string">"ProxyFetch - &#123;func&#125;: class method not exists!"</span>.format(func=fetch_source))</span><br><span class="line"><span class="keyword">continue</span></span><br><span class="line"><span class="keyword">if</span> <span class="keyword">not</span> callable(fetcher):</span><br><span class="line">self.log.error(<span class="string">"ProxyFetch - &#123;func&#125;: must be class method"</span>.format(func=fetch_source))</span><br><span class="line"><span class="keyword">continue</span></span><br><span class="line">thread_list.append(_ThreadFetcher(fetch_source, proxy_dict))</span><br><span class="line">    </span><br><span class="line"><span class="keyword">for</span> thread <span class="keyword">in</span> thread_list:</span><br><span class="line">    thread.setDaemon(<span class="keyword">True</span>)</span><br><span class="line">    thread.start()</span><br><span class="line"><span class="keyword">for</span> thread <span class="keyword">in</span> thread_list:</span><br><span class="line">    thread.join()</span><br><span class="line">self.log.info(<span class="string">"ProxyFetch - all complete!"</span>)</span><br></pre></td></tr></table></figure><ul><li><p>Cpython(默认安装的都是Cpython)中Dict和list、tuple都是<strong>线程安全</strong>的</p><ul><li><p>以装饰器的形式将过滤器将入到容器中</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ProxyValidator</span><span class="params">(withMetaclass<span class="params">(Singleton)</span>)</span>:</span></span><br><span class="line">    pre_validator = []</span><br><span class="line">    http_validator = []</span><br><span class="line">    https_validator = []</span><br><span class="line"></span><br><span class="line"><span class="meta">    @classmethod</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">addPreValidator</span><span class="params">(cls, func)</span>:</span></span><br><span class="line">        cls.pre_validator.append(func)</span><br><span class="line">        <span class="keyword">return</span> func</span><br><span class="line"></span><br><span class="line"><span class="comment"># 实际上执行了 formatValidator=ProxyValidator.addPreValidator(formatValidator)</span></span><br><span class="line"><span class="comment"># 由于addPreValidator返回了func, 所以formatValidator还是原来的addPreValidator, 但在类定义的时候ProxyValidator.pre_validator添加了formatValidator方法</span></span><br><span class="line"><span class="meta">@ProxyValidator.addPreValidator</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">formatValidator</span><span class="params">(proxy)</span>:</span></span><br><span class="line">    <span class="string">"""检查代理格式"""</span></span><br><span class="line">    verify_regex = <span class="string">r"\d&#123;1,3&#125;\.\d&#123;1,3&#125;\.\d&#123;1,3&#125;\.\d&#123;1,3&#125;:\d&#123;1,5&#125;"</span></span><br><span class="line">    _proxy = findall(verify_regex, proxy)</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">True</span> <span class="keyword">if</span> len(_proxy) == <span class="number">1</span> <span class="keyword">and</span> _proxy[<span class="number">0</span>] == proxy <span class="keyword">else</span> <span class="keyword">False</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DoValidator</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="string">""" 校验执行器 """</span></span><br><span class="line"><span class="meta">    @classmethod</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">validator</span><span class="params">(cls, proxy)</span>:</span></span><br><span class="line">        <span class="string">"""</span></span><br><span class="line"><span class="string">        校验入口</span></span><br><span class="line"><span class="string">        Args:</span></span><br><span class="line"><span class="string">            proxy: Proxy Object</span></span><br><span class="line"><span class="string">        Returns:</span></span><br><span class="line"><span class="string">            Proxy Object</span></span><br><span class="line"><span class="string">        """</span></span><br><span class="line">        http_r = cls.httpValidator(proxy)</span><br><span class="line">        https_r = <span class="keyword">False</span> <span class="keyword">if</span> <span class="keyword">not</span> http_r <span class="keyword">else</span> cls.httpsValidator(proxy)</span><br><span class="line"></span><br><span class="line">        proxy.check_count += <span class="number">1</span></span><br><span class="line">        proxy.last_time = datetime.now().strftime(<span class="string">"%Y-%m-%d %H:%M:%S"</span>)</span><br><span class="line">        proxy.last_status = <span class="keyword">True</span> <span class="keyword">if</span> http_r <span class="keyword">else</span> <span class="keyword">False</span></span><br><span class="line">        <span class="keyword">if</span> http_r:</span><br><span class="line">            <span class="keyword">if</span> proxy.fail_count &gt; <span class="number">0</span>:</span><br><span class="line">                proxy.fail_count -= <span class="number">1</span></span><br><span class="line">            proxy.https = <span class="keyword">True</span> <span class="keyword">if</span> https_r <span class="keyword">else</span> <span class="keyword">False</span></span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            proxy.fail_count += <span class="number">1</span></span><br><span class="line">        <span class="keyword">return</span> proxy</span><br><span class="line">    </span><br><span class="line"><span class="meta">    @classmethod</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">preValidator</span><span class="params">(cls, proxy)</span>:</span></span><br><span class="line">        <span class="keyword">for</span> func <span class="keyword">in</span> ProxyValidator.pre_validator:</span><br><span class="line">            <span class="keyword">if</span> <span class="keyword">not</span> func(proxy):</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">False</span></span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">True</span></span><br></pre></td></tr></table></figure></li></ul></li><li><p><a href="http://www.network-science.de/ascii/" target="_blank" rel="noopener">starter-banner</a>：启动横幅</p><ul><li>reflection: No、adjustment: cewnter、Stretch: Yes、width: 80</li><li>还不错的font：<ul><li>5lineoblique——好看</li><li>banner3——清楚</li><li>bell——抽象</li><li>big——清晰</li><li>bigchief——等高线版本、艺术</li><li>block——块状</li><li>bulbhead——可爱</li><li>larry3d——立体3d</li><li>ogre——清晰</li><li>puffy——清晰+一点可爱</li><li>slant——清晰+斜体</li></ul></li></ul></li><li><p>定时器框架apschedule配置:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">scheduler = BlockingScheduler(logger=scheduler_log, timezone=timezone)</span><br><span class="line"></span><br><span class="line">scheduler.add_job(__runProxyCheck, <span class="string">'interval'</span>, minutes=<span class="number">2</span>, id=<span class="string">"proxy_check"</span>, name=<span class="string">"proxy检查"</span>)</span><br><span class="line"></span><br><span class="line">executors = &#123;</span><br><span class="line">    <span class="comment"># job_defaults中的max_instances也受限于max_workers, 所以要大于max_instances；此外max_workers也决定了同时能处理几个同时发生的task</span></span><br><span class="line">    <span class="string">'default'</span>: &#123;<span class="string">'type'</span>: <span class="string">'threadpool'</span>, <span class="string">'max_workers'</span>: <span class="number">20</span>&#125;,</span><br><span class="line">    <span class="string">'processpool'</span>: ProcessPoolExecutor(max_workers=<span class="number">5</span>)</span><br><span class="line">&#125;</span><br><span class="line">job_defaults = &#123;</span><br><span class="line">    <span class="comment"># 合并将所有这些错过的执行合并为一个, 默认为True。 如果是定时的存储任务的话，参数肯定不同，不能合并所以得手动设置False</span></span><br><span class="line">    <span class="comment"># 像本项目每隔一段时间抓取到的数据也不太一样，所以无法直接当作一次错误任务合并</span></span><br><span class="line">    <span class="string">'coalesce'</span>: <span class="keyword">False</span>,</span><br><span class="line">    <span class="comment"># 默认情况下，每个作业只允许同时运行一个实例。这意味着，如果作业即将运行，但前一次运行尚未完成，则认为最近一次运行失败。通过在添加作业时使用关键字参数，可以设置调度程序允许同时运行的特定作业的最大实例数。默认为1</span></span><br><span class="line">    <span class="string">'max_instances'</span>: <span class="number">10</span>,</span><br><span class="line">    <span class="comment"># 框架会检查每个错过的执行时间，如果当前还在misfire_grace_time时间内，则会重新尝试执行任务，设高点就可以避免任务被漏掉执行。默认为1</span></span><br><span class="line">    <span class="comment"># "misfire_grace_time": 5  该项目未使用，而是采用了多任务实例来规避任务错过执行==&gt;即官方给出两种方案中的另一种。任务错过信息：Run time of job "say (trigger: interval[0:00:02])" was missed by 0:00:03.010383</span></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">scheduler.configure(executors=executors, job_defaults=job_defaults, timezone=timezone)</span><br><span class="line">scheduler.start()</span><br></pre></td></tr></table></figure><p>job_defaults参数含义见<a href="https://apscheduler.readthedocs.io/en/3.x/userguide.html?highlight=max_instances#modifying-jobs" target="_blank" rel="noopener">官方文档</a>、<a href="https://blog.csdn.net/weixin_44301439/article/details/124062178" target="_blank" rel="noopener">https://blog.csdn.net/weixin_44301439/article/details/124062178</a></p><p>注： 经过测试，在add_task中的func如果起了多个线程，其执行不受限于sche的配置</p></li><li><p>Python中如果只是使用全局变量则不需要用global声明（因为变量搜寻会由内往外），但是如果需要修改则需要用global声明，否则无法找到相应变量</p></li><li><p>生成器：使用了yield关键字的函数就是生成器，生成器是一类特殊的迭代器。</p><p>作用：</p><ul><li>处理大量数据：生成器一次返回一个结果，而不是一次返回所有结果。比如<code>sum([i for i in range(10000000000000)])</code>会卡机；<code>sum(i for i in range(10000000000000))</code>则不会</li><li><a href="https://blog.csdn.net/a200822146085/article/details/89388362" target="_blank" rel="noopener">代码更加简洁</a>：可以减少变量、空间</li><li>迭代器本身的作用</li></ul><p>yield关键字有两点作用：</p><p>保存当前运行状态（<strong>断点</strong>），然后暂停执行，即将生成器（函数）挂起；可以使用next()函数让生成器从断点处<strong>继续执行</strong>，即唤醒生成器（函数）<br>将yield关键字后面表达式的值作为<strong>返回值返回</strong>，此时可以理解为起到了return的作用</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">__runProxyFetch</span><span class="params">()</span>:</span>     </span><br><span class="line">    <span class="keyword">for</span> proxy <span class="keyword">in</span> proxy_fetcher.run():</span><br><span class="line">        proxy_queue.put(proxy)</span><br><span class="line">        </span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Fetcher</span><span class="params">(object)</span>:</span></span><br><span class="line">    name = <span class="string">"fetcher"</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="comment"># ...</span></span><br><span class="line">        <span class="comment"># 相比使用生成推导式 return [p for p in proxy_dict.values() if DoValidator.preValidator(p.proxy)]， 使用yield生成器可以节省空间</span></span><br><span class="line">        <span class="keyword">for</span> _ <span class="keyword">in</span> proxy_dict.values():</span><br><span class="line">                <span class="keyword">if</span> DoValidator.preValidator(_.proxy):</span><br><span class="line">                    <span class="keyword">yield</span> _</span><br></pre></td></tr></table></figure></li><li><p>应用部署：</p><p>①对apk换源；②设置时区</p><figure class="highlight dockerfile"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">FROM</span> python:<span class="number">3.6</span>-alpine</span><br><span class="line"><span class="comment"># ..</span></span><br><span class="line"><span class="comment"># apk repository</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> sed -i <span class="string">'s/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g'</span> /etc/apk/repositories</span></span><br><span class="line"><span class="bash"><span class="comment"># timezone</span></span></span><br><span class="line"><span class="bash">RUN apk add -U tzdata &amp;&amp; cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &amp;&amp; apk del tzdata</span></span><br><span class="line"><span class="bash"><span class="comment"># ...</span></span></span><br><span class="line"><span class="bash">ENTRYPOINT [ <span class="string">"sh"</span>, <span class="string">"start.sh"</span> ]</span></span><br></pre></td></tr></table></figure><p><code>docker-compose.yml</code>： 镜像还没编译好的情况。（如果自己改了功能并启用的话，需要用这种；或者自己发布镜像后用后一种）</p><figure class="highlight yml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">version:</span> <span class="string">'2'</span></span><br><span class="line"><span class="attr">services:</span></span><br><span class="line"><span class="attr">  proxy_pool:</span></span><br><span class="line"><span class="attr">    build:</span> <span class="string">.</span></span><br><span class="line"><span class="attr">    container_name:</span> <span class="string">proxy_pool</span></span><br><span class="line"><span class="attr">    ports:</span></span><br><span class="line"><span class="bullet">      -</span> <span class="string">"5010:5010"</span></span><br><span class="line"><span class="attr">    links:</span></span><br><span class="line"><span class="bullet">      -</span> <span class="string">proxy_redis</span></span><br><span class="line"><span class="attr">    environment:</span></span><br><span class="line"><span class="attr">      DB_CONN:</span> <span class="string">"redis://@proxy_redis:6379/0"</span></span><br><span class="line"><span class="attr">  proxy_redis:</span></span><br><span class="line"><span class="attr">    image:</span> <span class="string">"redis"</span></span><br><span class="line"><span class="attr">    container_name:</span> <span class="string">proxy_redis</span></span><br></pre></td></tr></table></figure><p><code>docker-compose.yml</code>：别人镜像已经编译好并上传</p><figure class="highlight yml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">version:</span> <span class="string">"3"</span></span><br><span class="line"><span class="attr">services:</span></span><br><span class="line"><span class="attr">  redis:</span></span><br><span class="line"><span class="attr">    image:</span> <span class="string">redis</span></span><br><span class="line"><span class="attr">    expose:</span></span><br><span class="line"><span class="bullet">      -</span> <span class="number">6379</span></span><br><span class="line"><span class="attr">  web:</span></span><br><span class="line"><span class="attr">    restart:</span> <span class="string">always</span></span><br><span class="line"><span class="attr">    image:</span> <span class="string">jhao104/proxy_pool</span></span><br><span class="line"><span class="attr">    environment:</span></span><br><span class="line"><span class="bullet">      -</span> <span class="string">DB_CONN=redis://redis:6379/0</span></span><br><span class="line"><span class="attr">    ports:</span></span><br><span class="line"><span class="bullet">      -</span> <span class="string">"5010:5010"</span></span><br><span class="line"><span class="attr">    depends_on:</span></span><br><span class="line"><span class="bullet">      -</span> <span class="string">redis</span></span><br></pre></td></tr></table></figure></li></ul><h3 id="scheduler的逻辑"><a class="markdownIt-Anchor" href="#scheduler的逻辑"></a> scheduler的逻辑</h3><p><img src="/2022/03/29/开源项目代码阅读记录/E:%5Chexo%5Csource_posts%5C%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE%E4%BB%A3%E7%A0%81%E9%98%85%E8%AF%BB%E8%AE%B0%E5%BD%95%5Cproxy_pool%E6%97%B6%E5%BA%8F%E5%9B%BE.png" alt="proxy_pool时序图"></p><h3 id="项目目录结构默写"><a class="markdownIt-Anchor" href="#项目目录结构默写"></a> 项目目录结构默写：</h3><ul><li>settings: 配置文件</li><li>main：启动文件</li><li>api：提供获取proxy数据接口</li><li>handler:<ul><li>loggerHandler：日志类</li><li>configHandler：单例的配置接口类</li><li>ProxyHandler: Proxy CRUD操作类</li></ul></li><li>fetcher: 代理数据获取类</li><li>db:<ul><li>dbClinet： 存储功能接口类</li><li>redisClient：存储功能实现类</li></ul></li><li>helper<ul><li>scheduler: 定时任务的定义与启动类</li><li>validator: proxy有效性校验类</li><li>check: 具体执行校验逻辑类</li><li>proxy: 获取的proxy数据封装类</li></ul></li><li>utils:<ul><li>lazyProperty： 懒加载描述器</li><li>singleton: 单例管理器类</li><li>six： python2与python3兼容类</li><li>webRequest: 网络请求封装类</li></ul></li></ul><h2 id="极验滑块验证码破解与研究三滑块缺口识别"><a class="markdownIt-Anchor" href="#极验滑块验证码破解与研究三滑块缺口识别"></a> 极验滑块验证码破解与研究（三）：滑块缺口识别</h2><p>首先我们需要准备两张图片：1. 带缺口的背景图；2. 与之对应的接口图</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"><span class="keyword">from</span> pathlib <span class="keyword">import</span> Path</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> PIL</span><br><span class="line"><span class="keyword">import</span> cv2</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">imshow</span><span class="params">(img, winname=<span class="string">'test'</span>, delay=<span class="number">0</span>)</span>:</span></span><br><span class="line">    <span class="string">"""cv2展示图片"""</span></span><br><span class="line">    cv2.imshow(winname, img)</span><br><span class="line">    cv2.waitKey(delay)</span><br><span class="line">    cv2.destroyAllWindows()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">pil_to_cv2</span><span class="params">(img)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    pil转cv2图片</span></span><br><span class="line"><span class="string">    :param img: pil图像, &lt;type 'PIL.JpegImagePlugin.JpegImageFile'&gt;</span></span><br><span class="line"><span class="string">    :return: cv2图像, &lt;type 'numpy.ndarray'&gt;</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    img = cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR)</span><br><span class="line">    <span class="keyword">return</span> img</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">bytes_to_cv2</span><span class="params">(img)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    二进制图片转cv2</span></span><br><span class="line"><span class="string">    :param img: 二进制图片数据, &lt;type 'bytes'&gt;</span></span><br><span class="line"><span class="string">    :return: cv2图像, &lt;type 'numpy.ndarray'&gt;</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    <span class="comment"># 将图片字节码bytes, 转换成一维的numpy数组到缓存中</span></span><br><span class="line">    img_buffer_np = np.frombuffer(img, dtype=np.uint8)</span><br><span class="line">    <span class="comment"># 从指定的内存缓存中读取一维numpy数据, 并把数据转换(解码)成图像矩阵格式</span></span><br><span class="line">    img_np = cv2.imdecode(img_buffer_np, <span class="number">1</span>)</span><br><span class="line">    <span class="keyword">return</span> img_np</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">cv2_open</span><span class="params">(img, flag=None)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    统一输出图片格式为cv2图像, &lt;type 'numpy.ndarray'&gt;</span></span><br><span class="line"><span class="string">    :param img: &lt;type 'bytes'/'numpy.ndarray'/'str'/'Path'/'PIL.JpegImagePlugin.JpegImageFile'&gt;</span></span><br><span class="line"><span class="string">    :param flag: 颜色空间转换类型, default: None</span></span><br><span class="line"><span class="string">        eg: cv2.COLOR_BGR2GRAY（灰度图）</span></span><br><span class="line"><span class="string">    :return: cv2图像, &lt;numpy.ndarray&gt;</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    <span class="keyword">if</span> isinstance(img, bytes):</span><br><span class="line">        img = bytes_to_cv2(img)</span><br><span class="line">    <span class="keyword">elif</span> isinstance(img, (str, Path)):</span><br><span class="line">        img = cv2.imread(str(img))</span><br><span class="line">    <span class="keyword">elif</span> isinstance(img, np.ndarray):</span><br><span class="line">        img = img</span><br><span class="line">    <span class="keyword">elif</span> isinstance(img, PIL.Image):</span><br><span class="line">        img = pil_to_cv2(img)</span><br><span class="line">    <span class="keyword">else</span>:</span><br><span class="line">        <span class="keyword">raise</span> ValueError(<span class="string">f'输入的图片类型无法解析: <span class="subst">&#123;type(img)&#125;</span>'</span>)</span><br><span class="line">    <span class="keyword">if</span> flag <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">None</span>:</span><br><span class="line">        img = cv2.cvtColor(img, flag)</span><br><span class="line">    <span class="keyword">return</span> img</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_distance</span><span class="params">(bg, tp, im_show=False, save_path=None)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    :param bg: 背景图路径或Path对象或图片二进制</span></span><br><span class="line"><span class="string">        eg: 'assets/bg.jpg'</span></span><br><span class="line"><span class="string">            Path('assets/bg.jpg')</span></span><br><span class="line"><span class="string">    :param tp: 缺口图路径或Path对象或图片二进制</span></span><br><span class="line"><span class="string">        eg: 'assets/tp.jpg'</span></span><br><span class="line"><span class="string">            Path('assets/tp.jpg')</span></span><br><span class="line"><span class="string">    :param im_show: 是否显示结果, &lt;type 'bool'&gt;; default: False</span></span><br><span class="line"><span class="string">    :param save_path: 保存路径, &lt;type 'str'/'Path'&gt;; default: None</span></span><br><span class="line"><span class="string">    :return: 缺口位置</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    <span class="comment"># 读取图片</span></span><br><span class="line">    bg_gray = cv2_open(bg, flag=cv2.COLOR_BGR2GRAY)</span><br><span class="line">    tp_gray = cv2_open(tp, flag=cv2.COLOR_BGR2GRAY)</span><br><span class="line">    <span class="comment"># 边缘检测</span></span><br><span class="line">    tp_gray = cv2.Canny(tp_gray, <span class="number">255</span>, <span class="number">255</span>)</span><br><span class="line">    bg_gray = cv2.Canny(bg_gray, <span class="number">255</span>, <span class="number">255</span>)</span><br><span class="line">    <span class="comment"># 目标匹配</span></span><br><span class="line">    result = cv2.matchTemplate(bg_gray, tp_gray, cv2.TM_CCOEFF_NORMED)</span><br><span class="line">    <span class="comment"># 解析匹配结果</span></span><br><span class="line">    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)</span><br><span class="line"></span><br><span class="line">    distance = max_loc[<span class="number">0</span>]</span><br><span class="line">    <span class="keyword">if</span> save_path <span class="keyword">or</span> im_show:</span><br><span class="line">        <span class="comment"># 需要绘制的方框高度和宽度</span></span><br><span class="line">        tp_height, tp_width = tp_gray.shape[:<span class="number">2</span>]</span><br><span class="line">        <span class="comment"># 矩形左上角点位置</span></span><br><span class="line">        x, y = max_loc</span><br><span class="line">        <span class="comment"># 矩形右下角点位置</span></span><br><span class="line">        _x, _y = x + tp_width, y + tp_height</span><br><span class="line">        <span class="comment"># 绘制矩形</span></span><br><span class="line">        bg_img = cv2_open(bg)</span><br><span class="line">        cv2.rectangle(bg_img, (x, y), (_x, _y), (<span class="number">0</span>, <span class="number">0</span>, <span class="number">255</span>), <span class="number">2</span>)</span><br><span class="line">        <span class="comment"># 保存缺口识别结果到背景图</span></span><br><span class="line">        <span class="keyword">if</span> save_path:</span><br><span class="line">            save_path = Path(save_path).resolve()</span><br><span class="line">            save_path = save_path.parent / <span class="string">f"<span class="subst">&#123;save_path.stem&#125;</span>.<span class="subst">&#123;distance&#125;</span><span class="subst">&#123;save_path.suffix&#125;</span>"</span></span><br><span class="line">            save_path = save_path.__str__()</span><br><span class="line">            cv2.imwrite(save_path, bg_img)</span><br><span class="line">        <span class="comment"># 显示缺口识别结果</span></span><br><span class="line">        <span class="keyword">if</span> im_show:</span><br><span class="line">            imshow(bg_img)</span><br><span class="line">    <span class="keyword">return</span> distance</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    d = get_distance(</span><br><span class="line">        bg=<span class="string">'assets/bg.jpg'</span>,</span><br><span class="line">        tp=<span class="string">'assets/tp.png'</span>,</span><br><span class="line">        im_show=<span class="keyword">True</span>,</span><br><span class="line">        save_path=<span class="string">'assets/bg.jpg'</span></span><br><span class="line">    )</span><br><span class="line">    print(d)</span><br></pre></td></tr></table></figure><p>类似代码：<strong><a href="https://github.com/lixi5338619/lxBook" target="_blank" rel="noopener">lxBook</a></strong></p><p>相关文章：</p><ol><li><a href="https://blog.csdn.net/qq_42857999/article/details/121586389" target="_blank" rel="noopener">极验滑块验证码破解与研究（一）：AST还原混淆JS</a></li><li><a href="https://blog.csdn.net/qq_42857999/article/details/121628908" target="_blank" rel="noopener">极验滑块验证码破解与研究（二）：缺口图片还原</a></li><li><a href="https://blog.csdn.net/qq_42857999/article/details/121635961" target="_blank" rel="noopener">极验滑块验证码破解与研究（三）：滑块缺口识别</a></li><li><a href="https://blog.csdn.net/qq_42857999/article/details/121659205" target="_blank" rel="noopener">极验滑块验证码破解与研究（四）：滑块轨迹构造</a></li><li><a href="https://blog.csdn.net/qq_42857999/article/details/121674134" target="_blank" rel="noopener">极验滑块验证码破解与研究（五）：请求分析及加密参数破解</a></li></ol><h1 id="机器学习"><a class="markdownIt-Anchor" href="#机器学习"></a> 机器学习</h1><h2 id="cardrecorder"><a class="markdownIt-Anchor" href="#cardrecorder"></a> <strong><a href="https://github.com/ZDZX-T/cardRecorder" target="_blank" rel="noopener">cardRecorder</a></strong></h2><blockquote><p>通过pyautoGUI实现的斗地主记牌器，提供了tkinter的界面</p></blockquote><ul><li><p>使用pyAutoGui找到类似的图片</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">   <span class="comment"># 进行截图</span></span><br><span class="line">   img = pyautogui.screenshot(region=myPos)</span><br><span class="line"><span class="comment"># 遍历所有排类型    </span></span><br><span class="line">   <span class="keyword">for</span> i <span class="keyword">in</span> myCardsNum.keys():</span><br><span class="line">       <span class="comment"># 进行定位, 找到pics\\m1.png在当前图片img里的位置</span></span><br><span class="line">       result = pyautogui.locateAll(needleImage=<span class="string">'pics\\m'</span> + i + <span class="string">'.png'</span>, haystackImage=img, confidence=myConfidence)</span><br><span class="line">       </span><br><span class="line">       myCardsNum[i] = cardsFilter(list(result), myFilter)</span><br></pre></td></tr></table></figure><p>pyautoGUI图片识别功能介绍：</p><blockquote><p>PyAutoGUI可以截取屏幕截图（截图），将其保存到文件中，并在屏幕中查找图像(定位)。</p></blockquote><ul><li>截图</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 截全屏</span></span><br><span class="line">pg.screenshot()  <span class="comment"># 只会保存到内存，不会缓存到本地</span></span><br><span class="line">pg.screenshot(<span class="string">'data/screenshot.jpg'</span>)  <span class="comment"># 指定路径，则会保存到本地</span></span><br><span class="line"><span class="comment"># 按照指定区域截屏, region(x坐标,y坐标,宽度,高度)，坐标为指定截图的左上角坐标</span></span><br><span class="line">pg.screenshot(imageFilename=<span class="string">'data/screenshot.jpeg'</span>, region=(<span class="number">300</span>, <span class="number">300</span>, <span class="number">500</span>, <span class="number">500</span>))</span><br></pre></td></tr></table></figure><ul><li>图片定位</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 在屏幕中匹配图片，并返回位置相关数据，匹配不到则返回 None</span></span><br><span class="line">locate_1 = pg.locateOnScreen(image=<span class="string">'data/test.png'</span>)</span><br><span class="line">print(locate_1)  <span class="comment"># Box(left=177, top=614, width=43, height=37)</span></span><br><span class="line">print(locate_1.left)  <span class="comment"># 177  可以通过 .left 等属性获取相应的值</span></span><br><span class="line">print(locate_1[<span class="number">0</span>])  <span class="comment"># 177  也可以通过下标获取相应的值</span></span><br><span class="line"><span class="comment"># “模糊”匹配  --  可以使用参数 confidence 来指定模糊程度，默认是 1，范围是 0 - 1（当然，太低就没意义了）, 使用该参数要求已经安装好 openCV</span></span><br><span class="line"><span class="comment"># 这里的路径关键字是 image，和上面的 imageFilename 不同，所以建议不要写关键字，直接写路径就好，防止写错</span></span><br><span class="line">locate_2 = pg.locateOnScreen(image=<span class="string">'data/test.png'</span>, confidence=<span class="number">0.9</span>)</span><br><span class="line"><span class="comment"># 返回匹配到的图片的中心位置坐标</span></span><br><span class="line">center_point_1 = pg.center(locate_1)</span><br><span class="line">print(center_point_1)  <span class="comment"># Point(x=198, y=632)</span></span><br></pre></td></tr></table></figure><p>注：屏幕截图功能需要Pillow模块、置信度需要cv</p><p>参考：<a href="https://www.cnblogs.com/math98/p/14399644.html" target="_blank" rel="noopener">PyAutoGui 图片识别+定位+截图函数文档</a>、<a href="https://zhuanlan.zhihu.com/p/417147884" target="_blank" rel="noopener">pyautogui 的截图及图片匹配</a>、<a href="https://www.cnblogs.com/gexbooks/p/10790063.html" target="_blank" rel="noopener">pyautogui 文档（五）：截图及定位功能</a></p></li><li><p>tkinter使用:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 变量使用</span></span><br><span class="line">num_8 = tkinter.StringVar()  <span class="comment"># 8</span></span><br><span class="line">num_8.set(strCards[<span class="number">8</span>])</span><br><span class="line"></span><br><span class="line">root = tkinter.Tk()</span><br><span class="line"><span class="comment"># 显示</span></span><br><span class="line">x_start = <span class="number">20</span></span><br><span class="line">y_start = <span class="number">5</span></span><br><span class="line">x_add = <span class="number">30</span></span><br><span class="line">x_dif = <span class="number">2</span></span><br><span class="line">y_dif = <span class="number">25</span></span><br><span class="line"><span class="comment"># 标签</span></span><br><span class="line">tkinter.Label(root, text=<span class="string">'大'</span>, font=(<span class="string">''</span>, <span class="number">14</span>), width=<span class="number">1</span>).place(x=x_start, y=y_start)</span><br><span class="line"><span class="comment"># 输入框</span></span><br><span class="line">tkinter.Entry(root, textvariable=num_dw, font=(<span class="string">''</span>, <span class="number">14</span>), width=<span class="number">1</span>).place(x=x_start + x_dif, y=y_start + y_dif)</span><br><span class="line"><span class="comment"># 进行循环等待事件发生</span></span><br><span class="line">root.main_loop()</span><br></pre></td></tr></table></figure><p>布局</p><ul><li>pack()：将控件放置在父控件内之前，规划此控件在区块内的位置。</li><li>grid()：将控件放置在父控件内之前，规划此控件为一个网格类型的架构。</li><li>place()：将控件放置在父控件内的特定(指定)位置。</li></ul></li></ul><h2 id="wiki_zh_word2vec"><a class="markdownIt-Anchor" href="#wiki_zh_word2vec"></a> <strong><a href="https://github.com/AimeeLee77/wiki_zh_word2vec" target="_blank" rel="noopener">wiki_zh_word2vec</a></strong></h2><blockquote><p>利用Python构建Wiki中文语料词向量模型试验——找到给定词相似的词语</p><p>代码涉及：开发环境准备、数据的获取、数据的预处理、模型构建和模型测试四大内容，对应的是实现模型构建的五个步骤。</p></blockquote><ol><li><p>开发环境准备: Python + gensim (NLP神器库) +jieba + <a href="https://bintray.com/package/files/byvoid/opencc/OpenCC" target="_blank" rel="noopener">OpenCC</a>(中文繁体替换成简体)</p></li><li><p>数据获取: 到wiki官网下载<a href="https://dumps.wikimedia.org/zhwiki/latest/zhwiki-latest-pages-articles.xml.bz2" target="_blank" rel="noopener">中文语料</a>、将XML的Wiki数据转换为text格式，使用到了gensim.corpora中的WikiCorpus函数</p></li><li><p>Wiki数据预处理：①使用OpenCC进行繁简转换；②jieba分词</p></li><li><p>Word2Vec模型训练：分好词的文档即可进行word2vec词向量模型的训练了。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 训练skip-gram模型</span></span><br><span class="line">model = Word2Vec(LineSentence(inp), size=<span class="number">400</span>, window=<span class="number">5</span>, min_count=<span class="number">5</span>,</span><br><span class="line">                 workers=multiprocessing.cpu_count())</span><br><span class="line">   </span><br><span class="line"><span class="comment"># 保存模型</span></span><br><span class="line">model.save(outp1)</span><br><span class="line">model.wv.save_word2vec_format(outp2, binary=<span class="keyword">False</span>)</span><br></pre></td></tr></table></figure><p>代码运行完成后得到如下四个文件，其中wiki.zh.text.model是建好的模型，wiki.zh.text.vector是词向量。</p></li><li><p>模型测试：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">model = gensim.models.Word2Vec.load(fdir + <span class="string">'wiki.zh.text.model'</span>)</span><br><span class="line">word = model.most_similar(<span class="string">u"足球"</span>)</span><br><span class="line"><span class="keyword">for</span> t <span class="keyword">in</span> word:</span><br><span class="line">    <span class="keyword">print</span> (t[<span class="number">0</span>],t[<span class="number">1</span>])</span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">足球运动 0.531202495098</span></span><br><span class="line"><span class="string">国际足球 0.510423064232</span></span><br><span class="line"><span class="string">足球比赛 0.499910503626</span></span><br><span class="line"><span class="string">国家足球队 0.488919615746</span></span><br><span class="line"><span class="string">排球 0.488108128309</span></span><br><span class="line"><span class="string">男体子篮球0.482638418674</span></span><br><span class="line"><span class="string">体育 0.47221454978</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure></li></ol><p>附：什么是词向量? ——词向量技术是将词转化成为稠密向量，并且对于相似的词，其对应的词向量也相近。</p><p>在<a href="https://so.csdn.net/so/search?q=%E8%87%AA%E7%84%B6%E8%AF%AD%E8%A8%80%E5%A4%84%E7%90%86&amp;spm=1001.2101.3001.7020" target="_blank" rel="noopener">自然语言处理</a>任务中，首先需要考虑词如何在计算机中表示。通常，有两种表示方式：</p><ul><li>one-hot representation(离散表示)<ul><li>传统的基于规则或基于统计的自然语义处理方法将单词看作一个原子符号</li><li>把每个词表示为一个长向量。这个向量的维度是词表大小，向量中只有一个维度的值为1，其余维度为0，这个维度就代表了当前的词。</li><li>one-hot representation相当于给每个词分配一个id，这就导致这种表示方式不能展示词与词之间的关系。另外，one-hot representation将会导致特征空间非常大，但也带来一个好处，就是在高维空间中，很多应用任务线性可分。</li></ul></li><li>distribution representation(分布式表示)<ul><li>word embedding指的是将词转化成一种分布式表示，又称词向量。分布式表示将词表示成一个定长的连续的稠密向量。</li><li>分布式表示优点:<ol><li>词之间存在相似关系：是词之间存在“距离”概念，这对很多自然语言处理的任务非常有帮助。</li><li>包含更多信息：词向量能够包含更多信息，并且每一维都有特定的含义。在采用one-hot特征时，可以对特征向量进行删减，词向量则不能。</li></ol></li></ul></li></ul><h1 id="系统设计"><a class="markdownIt-Anchor" href="#系统设计"></a> 系统设计</h1><h2 id="python-自制屏幕翻译工具"><a class="markdownIt-Anchor" href="#python-自制屏幕翻译工具"></a> <a href="https://pythondict.com/python-work/screen-translate-tools/" target="_blank" rel="noopener">Python 自制屏幕翻译工具</a></h2><p>程序的功能主要由3个模块组成</p><ol><li><p>screenshot：截图</p><blockquote><p>使用Pillow库</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 安装PIL依赖: pip install pillow</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Step1: 从剪切板读取图片</span></span><br><span class="line">img = ImageGrab.grabclipboard()</span><br><span class="line"><span class="comment"># 保存到本地 or 内存中</span></span><br><span class="line">image_result = <span class="string">'./temp.png'</span></span><br><span class="line">img.save(image_result)</span><br></pre></td></tr></table></figure></li><li><p>recognition：将截图中的问题识别成文字</p><blockquote><p>使用easyocr或者pytesseract</p></blockquote><p>easyocr：</p><ul><li>编程使用</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> easyocr</span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line"><span class="comment"># 运行的过程中会安装所需要的模型文件, 也可以自己下载后放到对应路径下</span></span><br><span class="line"><span class="comment"># Windows：C:\Users\用户名.EasyOCR\model</span></span><br><span class="line"><span class="comment"># Linux：~/ .EasyOCR / model</span></span><br><span class="line">reader = easyocr.Reader([<span class="string">'ch_sim'</span>])</span><br><span class="line"></span><br><span class="line"><span class="comment"># 使用GPU，看是否会有提示 CUDA not available</span></span><br><span class="line"><span class="comment"># reader = easyocr.Reader(['ch_sim'])</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 支持同时多个语言, 虽然可以一次性识别许多种语言，</span></span><br><span class="line"><span class="comment"># 但并非所有语言都可以一起用，通常是公共语言和一个特殊语种可以一起识别，相互兼容，比如英语和日语。</span></span><br><span class="line"><span class="comment"># reader = easyocr.Reader(['ch_sim','en'])</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 识别</span></span><br><span class="line">resp = requests.get(<span class="string">'https://images.zsxq.com/Fsl4plilMdpd_C8gXVJT1mKJDui9?e=1656604799&amp;token=kIxbL07-8jAj8w1n4s9zv64FuZZNEATmlU_Vm6zD:Pg0990rjMMQUAlf2bJlNL9lmLSU='</span>)</span><br><span class="line">result = reader.readtext(resp.content)</span><br><span class="line">print(result)</span><br></pre></td></tr></table></figure><ul><li>命令行运行： <code>easyocr -l ch_sim en -f test.png --detail=1 --gpu=True</code></li></ul><p>图片加载函数<code>readtext</code>接收三种参数</p><ul><li>文件路径</li><li>图片的<code>np_array</code></li><li>字节流对象</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> PIL <span class="keyword">import</span> Image</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> easyocr</span><br><span class="line"><span class="keyword">from</span> PIL <span class="keyword">import</span> ImageGrab</span><br><span class="line"></span><br><span class="line"><span class="comment"># 从剪切板中获得图片</span></span><br><span class="line">r = easyocr.Reader([<span class="string">"ch_sim"</span>])</span><br><span class="line"></span><br><span class="line">img = ImageGrab.grabclipboard()</span><br><span class="line">print(img, type(img))</span><br><span class="line"></span><br><span class="line"><span class="comment"># 参数为: file path or numpy-array or a byte stream object</span></span><br><span class="line"><span class="comment"># 1: file path</span></span><br><span class="line">img_path = <span class="string">"ttt.png"</span></span><br><span class="line">img.save(img_path)</span><br><span class="line">reg_result = r.readtext(img_path)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 2: numpy-array</span></span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">np_img = np.array(img)</span><br><span class="line">reg_result = r.readtext(np_img)</span><br><span class="line">print(reg_result)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3: a byte stream object</span></span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line">resp = requests.get(<span class="string">'https://images.zsxq.com/Fsl4plilMdpd_C8gXVJT1mKJDui9?e=1656604799&amp;token=kIxbL07-8jAj8w1n4s9zv64FuZZNEATmlU_Vm6zD:Pg0990rjMMQUAlf2bJlNL9lmLSU='</span>)</span><br><span class="line">reg_result = r.readtext(resp.content)</span><br><span class="line">print(reg_result)</span><br></pre></td></tr></table></figure></li><li><p>Translation：将文字翻译</p><blockquote><p>可以使用网上的有道翻译API、或者直接使用Google提供的翻译库<code>googletrans</code></p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Google翻译</span></span><br><span class="line">translator = Translator(service_urls=[<span class="string">'translate.google.cn'</span>])</span><br><span class="line"></span><br><span class="line"><span class="comment"># 翻译成中文</span></span><br><span class="line">content_chinese = translator.translate(content_eng, src=<span class="string">'en'</span>, dest=<span class="string">'zh-cn'</span>).text</span><br></pre></td></tr></table></figure></li><li><p>GUI：将翻译后的结果展示</p><blockquote><p>使用 Python 自带的 GUI tkinker，将识别后的中文显示出来。</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 初始化</span></span><br><span class="line">root = Tk()</span><br><span class="line">root.withdraw()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 显示翻译后的结果，以对话框的形式</span></span><br><span class="line">tkinter.messagebox.showinfo(<span class="string">'翻译结果'</span>, content_chinese)</span><br></pre></td></tr></table></figure></li></ol><p>完整代码：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> easyocr</span><br><span class="line"><span class="keyword">from</span> PIL <span class="keyword">import</span> ImageGrab, Image</span><br><span class="line"></span><br><span class="line"><span class="comment"># r = easyocr.Reader(["ch_sim"])</span></span><br><span class="line">r = easyocr.Reader([<span class="string">"en"</span>])</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">screenShot</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""1. 从剪切板中获得图片"""</span></span><br><span class="line">    img = ImageGrab.grabclipboard()</span><br><span class="line">    print(img, type(img))</span><br><span class="line">    <span class="keyword">return</span> img</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">identify</span><span class="params">(img: Image)</span>:</span></span><br><span class="line">    <span class="string">"""2. 识别文字"""</span></span><br><span class="line">    <span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line">    np_img = np.array(img)</span><br><span class="line">    reg_result = r.readtext(np_img)</span><br><span class="line">    rs = sorted(reg_result, key=<span class="keyword">lambda</span> x: x[<span class="number">2</span>], reverse=<span class="keyword">True</span>)[<span class="number">0</span>][<span class="number">1</span>]</span><br><span class="line">    <span class="keyword">return</span> rs</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">translate</span><span class="params">(rs: str)</span>:</span></span><br><span class="line">    <span class="string">"""3. 翻译：googletrans 库"""</span></span><br><span class="line">    <span class="keyword">from</span> googletrans <span class="keyword">import</span> Translator</span><br><span class="line">    <span class="comment"># Google翻译</span></span><br><span class="line">    translator = Translator(service_urls=[<span class="string">'translate.google.cn'</span>])</span><br><span class="line">    <span class="comment"># 翻译成中文</span></span><br><span class="line">    translated_str = translator.translate(rs, src=<span class="string">'en'</span>, dest=<span class="string">'zh-cn'</span>).text</span><br><span class="line">    <span class="keyword">return</span> translated_str</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">showText</span><span class="params">(s: str)</span>:</span></span><br><span class="line">    <span class="string">"""4. tk展示"""</span></span><br><span class="line">    <span class="keyword">import</span> tkinter <span class="keyword">as</span> tk</span><br><span class="line">    <span class="keyword">from</span> tkinter.messagebox <span class="keyword">import</span> showinfo  <span class="comment"># 弹窗库</span></span><br><span class="line"></span><br><span class="line">    <span class="comment"># 找到主应用</span></span><br><span class="line">    root = tk.Tk()</span><br><span class="line">    <span class="comment"># 隐藏主界面</span></span><br><span class="line">    root.withdraw()</span><br><span class="line">    print(s)</span><br><span class="line">    showinfo(<span class="string">"翻译结果"</span>, s)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    img = screenShot()</span><br><span class="line">    <span class="comment"># 识别出来的文字</span></span><br><span class="line">    rs = identify(img)</span><br><span class="line">    translated_str = translate(rs)</span><br><span class="line">    showText(translated_str)</span><br></pre></td></tr></table></figure><h2 id="system-design-primer"><a class="markdownIt-Anchor" href="#system-design-primer"></a> <strong><a href="https://github.com/donnemartin/system-design-primer" target="_blank" rel="noopener">system-design-primer</a></strong></h2><h2 id="go-package-plantuml"><a class="markdownIt-Anchor" href="#go-package-plantuml"></a> <strong><a href="https://github.com/qquunn/go-package-plantuml" target="_blank" rel="noopener"> go-package-plantuml</a></strong></h2><ul><li>go中的argueparser：<code>&quot;import github.com/jessevdk/go-flags&quot;</code>。除此之外，还有flag、pflag来处理输入参数</li></ul><h2 id="id-maker"><a class="markdownIt-Anchor" href="#id-maker"></a> <a href="https://github.com/yongxinz/id-maker" target="_blank" rel="noopener">id-maker</a></h2><h2 id="health_checker"><a class="markdownIt-Anchor" href="#health_checker"></a> <a href="https://github.com/Freedomisgood/When_Coding_in_ZJU/blob/main/Health_Checkin/health_checkin_helper_cst.py" target="_blank" rel="noopener">health_checker</a></h2><blockquote><p>浙大的登陆加密</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Python转义实现</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">_rsa_encrypt</span><span class="params">(self, password_str, e_str, M_str)</span>:</span></span><br><span class="line">    <span class="comment"># 将str转化为ascii的字节</span></span><br><span class="line">    password_bytes = bytes(password_str, <span class="string">'ascii'</span>)</span><br><span class="line">    <span class="comment"># 大端填充</span></span><br><span class="line">    password_int = int.from_bytes(password_bytes, <span class="string">'big'</span>)</span><br><span class="line">    <span class="comment"># 将其按照16进制的形式转成10进制</span></span><br><span class="line">    e_int = int(e_str, <span class="number">16</span>)</span><br><span class="line">    M_int = int(M_str, <span class="number">16</span>)</span><br><span class="line">    <span class="comment"># password_int^e%M</span></span><br><span class="line">    result_int = pow(password_int, e_int, M_int)</span><br><span class="line">    <span class="comment"># 10进制转成16进制str, 并 rjust() 返回一个原字符串右对齐,并使用空格填充至长度 width 的新字符串。</span></span><br><span class="line">    <span class="comment"># "hello".rjust(8, "0") ==&gt; 000hello</span></span><br><span class="line">    <span class="keyword">return</span> hex(result_int)[<span class="number">2</span>:].rjust(<span class="number">128</span>, <span class="string">'0'</span>)</span><br></pre></td></tr></table></figure><p>Js加密</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">RSAUtils.encryptedString = <span class="function"><span class="keyword">function</span>(<span class="params">key, s</span>) </span>&#123;</span><br><span class="line"><span class="keyword">var</span> a = [];</span><br><span class="line">    <span class="comment">// s为反向的password, 如321321aaa</span></span><br><span class="line"><span class="keyword">var</span> sl = s.length;</span><br><span class="line"><span class="keyword">var</span> i = <span class="number">0</span>;</span><br><span class="line"><span class="keyword">while</span> (i &lt; sl) &#123;</span><br><span class="line">a[i] = s.charCodeAt(i);</span><br><span class="line">i++;</span><br><span class="line">&#125;</span><br><span class="line">    <span class="comment">// 补足0</span></span><br><span class="line"><span class="keyword">while</span> (a.length % key.chunkSize != <span class="number">0</span>) &#123;</span><br><span class="line">a[i++] = <span class="number">0</span>;</span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">// a ==&gt; 321312aaa000000 的 ascii字符数组</span></span><br><span class="line"><span class="keyword">var</span> al = a.length;</span><br><span class="line"><span class="keyword">var</span> result = <span class="string">""</span>;</span><br><span class="line"><span class="keyword">var</span> j, k, block;</span><br><span class="line"><span class="keyword">for</span> (i = <span class="number">0</span>; i &lt; al; i += key.chunkSize) &#123;</span><br><span class="line">block = <span class="keyword">new</span> BigInt();</span><br><span class="line">j = <span class="number">0</span>;</span><br><span class="line"><span class="keyword">for</span> (k = i; k &lt; i + key.chunkSize; ++j) &#123;</span><br><span class="line">block.digits[j] = a[k++];</span><br><span class="line">block.digits[j] += a[k++] &lt;&lt; <span class="number">8</span>;</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">var</span> crypt = key.barrett.powMod(block, key.e);</span><br><span class="line">        </span><br><span class="line"><span class="keyword">var</span> text = key.radix == <span class="number">16</span> ? RSAUtils.biToHex(crypt) : RSAUtils.biToString(crypt, key.radix);</span><br><span class="line">result += text + <span class="string">" "</span>;</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">return</span> result.substring(<span class="number">0</span>, result.length - <span class="number">1</span>); <span class="comment">// Remove last space.</span></span><br><span class="line">&#125;;</span><br></pre></td></tr></table></figure><ul><li><code>new BigInt();</code>得到的是digits.len=130的int数组</li></ul><h2 id="cookiecutter-golang"><a class="markdownIt-Anchor" href="#cookiecutter-golang"></a> <strong><a href="https://github.com/lacion/cookiecutter-golang" target="_blank" rel="noopener">cookiecutter-golang</a></strong></h2><blockquote><p>快速生成Go项目的cookiecutter模板</p></blockquote><ul><li><a href="https://github.com/lacion/cookiecutter-golang/blob/master/%7B%7Bcookiecutter.app_name%7D%7D/main.go" target="_blank" rel="noopener">cobar使用</a>与<a href="https://github.com/lacion/cookiecutter-golang/blob/master/%7B%7Bcookiecutter.app_name%7D%7D/cmd/root.go" target="_blank" rel="noopener">cobar命令定义</a>，相当于Python中的Click模块</li></ul><h2 id="dddd_trainer"><a class="markdownIt-Anchor" href="#dddd_trainer"></a> <strong><a href="https://github.com/sml2h3/dddd_trainer" target="_blank" rel="noopener"> dddd_trainer</a></strong></h2><ul><li><p>Cli命令使用fire库：fire是python中用于生成命令行界面(Command Line Interfaces, CLIs)的工具，不需要做任何额外的工作，只需要从主模块中调用fire.Fire()，它会自动将你的代码转化为CLI，<u>Fire()的参数可以说任何的python对象</u></p><p>特点：所有的<strong>子命令</strong>都能以函数、类等形式书写，流程和参数获取都比较清晰；传参可以自动被解析成Python类型：字符串、元组、列表、字典</p><p>★使用指南： <a href="https://blog.csdn.net/qq_17550379/article/details/79943740" target="_blank" rel="noopener">https://blog.csdn.net/qq_17550379/article/details/79943740</a></p></li><li><p><a href="https://www.jianshu.com/p/476478c17b8e" target="_blank" rel="noopener">ONNXRuntime</a>： 通常我们在训练模型时可以使用很多不同的框架，比如有的同学喜欢用 <code>Pytorch</code>，有的同学喜欢使用 <code>TensorFLow</code>，也有的喜欢 <code>MXNet</code>，以及深度学习最开始流行的 <code>Caffe</code>等等，这样不同的训练框架就导致了产生不同的模型结果包，在模型进行部署推理时就需要不同的依赖库，而且同一个框架比如<code>tensorflow</code> 不同的版本之间的差异较大， 为了解决这个混乱问题，<a href="https://links.jianshu.com/go?to=https%3A%2F%2Flfaidata.foundation%2F" target="_blank" rel="noopener">LF AI</a> 这个组织联合 Facebook, MicroSoft等公司制定了机器学习模型的标准，这个标准叫做<strong>ONNX, Open Neural Network Exchage</strong>，<em>所有其他框架产生的模型包 (<code>.pth</code>, <code>.pb</code>) 都可以转换成这个标准格式，转换成这个标准格式后，就可以使用统一的 ONNX Runtime等工具进行统一部署。</em></p></li><li><p><a href="https://pythondict.com/life-intelligent/tools/loguru/" target="_blank" rel="noopener">Loguru — 最强大的 Python 日志记录器</a>： 强大的全局日志对象，具有开箱即用的特性、美观的输出，支持<code>f-string</code>、支持在主进程和线程中捕获异常</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">logger.add(<span class="string">"file.log"</span>, rotation=<span class="string">"12:00"</span> , format=<span class="string">"&#123;time:YYYY-MM-DD at HH:mm:ss&#125; | &#123;level&#125; | &#123;message&#125;"</span>, level=<span class="string">"INFO"</span>)</span><br><span class="line">logger.info(<span class="string">"That's it, beautiful and simple logging!"</span>)</span><br></pre></td></tr></table></figure></li></ul><h2 id="njupt-api"><a class="markdownIt-Anchor" href="#njupt-api"></a> <strong><a href="https://github.com/gaoliang/NJUPT-API" target="_blank" rel="noopener">NJUPT-API</a></strong></h2><blockquote><p>南京邮电大学网站接口, Python模块</p></blockquote><ul><li><p><code>@login_required</code> 装饰器: 由于有些操作必须登录后才会有正确结果，所以相当于在执行前做了一层校验（至于抛出异常or转向登录就看装饰器中怎么写了）—&gt; 由于是封装成了package库，所以在这一层需要进行限制。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">login_required</span><span class="params">(func)</span>:</span></span><br><span class="line"><span class="meta">    @wraps(func)</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">wrapper</span><span class="params">(self, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> self.verified:</span><br><span class="line">            self.login_by_sso()</span><br><span class="line">        <span class="keyword">return</span> func(self, *args, **kwargs)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">return</span> wrapper</span><br></pre></td></tr></table></figure></li><li><p>setuptools使用</p><ul><li><a href="http://setup.py" target="_blank" rel="noopener">setup.py</a></li><li><a href="http://MANIFEST.in" target="_blank" rel="noopener">MANIFEST.in</a></li></ul></li><li><p>模块暴露：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># njupt/__init__.py</span></span><br><span class="line"><span class="keyword">from</span> njupt.card <span class="keyword">import</span> Card  <span class="comment"># noqa</span></span><br><span class="line"><span class="keyword">from</span> njupt.zhengfang <span class="keyword">import</span> Zhengfang  <span class="comment"># noqa</span></span><br><span class="line"><span class="keyword">from</span> njupt.library <span class="keyword">import</span> Library  <span class="comment"># noqa</span></span><br><span class="line"><span class="keyword">from</span> njupt.runningman <span class="keyword">import</span> RunningMan  <span class="comment"># noqa</span></span><br><span class="line"><span class="keyword">from</span> njupt.sso <span class="keyword">import</span> SSO  <span class="comment"># noqa</span></span><br><span class="line"><span class="comment"># 表示njupt模块提供以下5个接口</span></span><br><span class="line">__all__ = [<span class="string">'SSO'</span>, <span class="string">'Card'</span>, <span class="string">'Zhengfang'</span>, <span class="string">'Library'</span>, <span class="string">'RunningMan'</span>]</span><br></pre></td></tr></table></figure></li><li><p>验证码识别</p><blockquote><p>字符型验证码：把图片去噪、分割、归类。收集了足够多的样本，然后对新的图片也分割。将分隔后的每个字符与样本集匹配，选概率最大的字符，将组合</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CardCaptcha</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    一卡通系统的验证码工具类</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, im)</span>:</span></span><br><span class="line">        <span class="string">"""</span></span><br><span class="line"><span class="string">        :param im: PIL.image对象</span></span><br><span class="line"><span class="string">        """</span></span><br><span class="line">        self.im = im.convert(<span class="string">'L'</span>)</span><br><span class="line">        self.image_pre_process()</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">image_pre_process</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="comment"># 二值化, 去除杂色点</span></span><br><span class="line">        threshold = <span class="number">200</span></span><br><span class="line">        <span class="keyword">for</span> x <span class="keyword">in</span> range(self.im.width):</span><br><span class="line">            <span class="keyword">for</span> y <span class="keyword">in</span> range(self.im.height):</span><br><span class="line">                pix = self.im.getpixel((x, y))</span><br><span class="line">                <span class="keyword">if</span> pix &gt; threshold:</span><br><span class="line">                    self.im.putpixel((x, y), BLACK)</span><br><span class="line">                <span class="keyword">else</span>:</span><br><span class="line">                    self.im.putpixel((x, y), WHITE)</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">handle_split_image</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="comment"># 切割验证码，返回包含五个字符图像的列表</span></span><br><span class="line">        y_min, y_max = <span class="number">0</span>, <span class="number">82</span></span><br><span class="line">        <span class="comment"># 字符位置大致固定([7,49], [49, 91], ...)</span></span><br><span class="line">        split_lines = [<span class="number">7</span>, <span class="number">49</span>, <span class="number">91</span>, <span class="number">133</span>, <span class="number">175</span>, <span class="number">217</span>]</span><br><span class="line">        ims = [self.im.crop([u, y_min, v, y_max]) <span class="keyword">for</span> u, v <span class="keyword">in</span> zip(split_lines[:<span class="number">-1</span>], split_lines[<span class="number">1</span>:])]</span><br><span class="line">        <span class="keyword">return</span> ims</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">crack</span><span class="params">(self)</span>:</span></span><br><span class="line">        result = []</span><br><span class="line">        <span class="comment"># 装载训练数据集</span></span><br><span class="line">        <span class="keyword">with</span> open(os.path.join(current_dir, <span class="string">'image_data.json'</span>), <span class="string">'rb'</span>) <span class="keyword">as</span> f:</span><br><span class="line">            image_data = json.load(f)</span><br><span class="line">        <span class="keyword">for</span> letter <span class="keyword">in</span> self.handle_split_image():</span><br><span class="line">            <span class="comment"># build_vector将图片转成一维向量: for pixel in image.getdata():</span></span><br><span class="line">            letter_vector = build_vector(letter)</span><br><span class="line">            guess = []</span><br><span class="line">            <span class="comment"># 在模型数据中, 找一个汉明距离最小(最像的结果)</span></span><br><span class="line">            <span class="keyword">for</span> image <span class="keyword">in</span> image_data:</span><br><span class="line">                <span class="comment"># x为key, 当前匹配的字符; y为向量</span></span><br><span class="line">                <span class="keyword">for</span> x, y <span class="keyword">in</span> image.items():</span><br><span class="line">                    <span class="keyword">if</span> len(y) != <span class="number">0</span>:</span><br><span class="line">                        <span class="comment"># 计算两个一维向量的汉明距（向量只包含0，1时）:(只要元素值不等, 距离就+1)</span></span><br><span class="line">                        guess.append((distance_hanmming(y, letter_vector), x))</span><br><span class="line">            guess.sort()</span><br><span class="line">            <span class="comment"># 找出距离最小的, 即最匹配的字符</span></span><br><span class="line">            result.append(guess[<span class="number">0</span>][<span class="number">1</span>])</span><br><span class="line">        <span class="keyword">return</span> <span class="string">''</span>.join(result)</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__str__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> self.crack()</span><br></pre></td></tr></table></figure><p>注：PIL中image.convert()函数使用</p><ol><li>模式“1”：为二值图像，非黑即白。但是它每个像素用8个bit表示，0表示黑，255表示白。</li><li>模式“L”： 为灰色图像，它的每个像素用8个bit表示，0表示黑，255表示白，其他数字表示不同的灰度。在PIL中，从模式“RGB”转换为“L”模式是按照下面的公式转换的：<code>L = R * 299/1000 + G * 587/1000+ B * 114/1000</code></li><li>模式“P”为8位彩色图像、模式“RGBA”(alpha透明度)为32位彩色图像、模式“CMYK”为32位彩色图像</li></ol></li><li><p>自制打码工具</p><ol><li><code>captcha_getter.py</code></li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 根据图片自己输入识别结果, 将图片保存下来, 图片名为正确的打码结果</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CaptchaGUI</span>:</span></span><br><span class="line">    BASE_URL = <span class="string">"http://yktapp.njupt.edu.cn:8070"</span></span><br><span class="line">    CAPTCHA_URL = BASE_URL + <span class="string">"/Login/GetValidateCode"</span></span><br><span class="line">    LOGIN_URL = BASE_URL + <span class="string">"/Login/LoginBySnoQuery"</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        self.s = requests.session()</span><br><span class="line">        self.r = self.s.get(self.CAPTCHA_URL)</span><br><span class="line">        self.im = Image.open(BytesIO(self.r.content))</span><br><span class="line">        self.root = tkinter.Tk()</span><br><span class="line">        self.tkimg = ImageTk.PhotoImage(self.im)</span><br><span class="line">        self.imgLabel = tkinter.Label(self.root, image=self.tkimg)</span><br><span class="line">        self.imgLabel.pack()</span><br><span class="line">        self.message = tkinter.Entry(self.root)</span><br><span class="line">        self.message.pack()</span><br><span class="line">        self.root.bind(<span class="string">'&lt;Return&gt;'</span>, self.judge_and_save)</span><br><span class="line">        self.root.mainloop()</span><br><span class="line">        </span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">judge_and_save</span><span class="params">(self, event)</span>:</span></span><br><span class="line">        <span class="comment"># 输出输入框的内容</span></span><br><span class="line">        captcha_value = self.message.get()</span><br><span class="line">        print(captcha_value)</span><br><span class="line">        data = &#123;</span><br><span class="line">            <span class="string">"sno"</span>: self.account,</span><br><span class="line">            <span class="string">"pwd"</span>: base64.b64encode(self.pwd.encode(<span class="string">"utf8"</span>)),</span><br><span class="line">            <span class="string">"ValiCode"</span>: captcha_value,</span><br><span class="line">            <span class="string">"remember"</span>: <span class="number">1</span>,</span><br><span class="line">            <span class="string">"uclass"</span>: <span class="number">1</span>,</span><br><span class="line">            <span class="string">"zqcode"</span>: <span class="string">""</span>,</span><br><span class="line">            <span class="string">"json"</span>: <span class="keyword">True</span>,</span><br><span class="line">        &#125;</span><br><span class="line"><span class="comment"># 只有输入正确的才保存</span></span><br><span class="line">        r = self.s.post(self.LOGIN_URL, data=data)</span><br><span class="line">        <span class="keyword">if</span> r.json()[<span class="string">"IsSucceed"</span>]:</span><br><span class="line">            print(<span class="string">"成功！"</span>)</span><br><span class="line">            <span class="keyword">with</span> open(<span class="string">"captchas/&#123;&#125;.gif"</span>.format(captcha_value), <span class="string">'wb+'</span>) <span class="keyword">as</span> f:</span><br><span class="line">                f.write(self.r.content)</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            print(r.json())</span><br><span class="line">            print(<span class="string">"验证码输错了"</span>)</span><br><span class="line"><span class="comment"># 刷新, 进行打码下一张</span></span><br><span class="line">        self.r = self.s.get(self.CAPTCHA_URL)</span><br><span class="line">        self.im = Image.open(BytesIO(self.r.content))</span><br><span class="line">        <span class="comment"># 刷新图片框</span></span><br><span class="line">        self.tkimg = ImageTk.PhotoImage(self.im)</span><br><span class="line">        self.imgLabel.config(image=self.tkimg)</span><br><span class="line">        <span class="comment"># ▲. 将输入框清空</span></span><br><span class="line">        self.message.delete(<span class="number">0</span>, <span class="string">'end'</span>)</span><br></pre></td></tr></table></figure><ol start="2"><li><code>captcha_gen.py</code></li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">spilt2chars</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    分割已有的数据为字符并保存</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    <span class="keyword">try</span>:</span><br><span class="line">        shutil.rmtree(<span class="string">'captcha_chars'</span>)</span><br><span class="line">    <span class="keyword">except</span> FileNotFoundError:</span><br><span class="line">        <span class="keyword">pass</span></span><br><span class="line">    os.mkdir(<span class="string">"captcha_chars"</span>)</span><br><span class="line">    values = <span class="string">"1234567890"</span></span><br><span class="line">    <span class="comment"># 每个字符创建对应的文件夹</span></span><br><span class="line">    <span class="keyword">for</span> value <span class="keyword">in</span> values:</span><br><span class="line">        os.mkdir(<span class="string">'captcha_chars/&#123;&#125;'</span>.format(value))</span><br><span class="line"></span><br><span class="line">    file_names = os.listdir(<span class="string">'captchas'</span>)</span><br><span class="line">    <span class="comment"># 遍历所有字符文件夹</span></span><br><span class="line">    <span class="keyword">for</span> file_name <span class="keyword">in</span> file_names: </span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> os.path.isdir(<span class="string">'captchas/&#123;&#125;'</span>.format(file_name)) <span class="keyword">and</span> file_name != <span class="string">'.DS_Store'</span>:</span><br><span class="line">            <span class="comment"># 获得打码img的图片名称(正确结果)</span></span><br><span class="line">            values = file_name[:<span class="number">4</span>]</span><br><span class="line">            im = Image.open(<span class="string">'captchas/&#123;&#125;'</span>.format(file_name))</span><br><span class="line">            captcha = CardCaptcha(im)</span><br><span class="line">            <span class="comment"># handle_split_image为切割后的四个字符图片, values为3fgc的值</span></span><br><span class="line">            <span class="keyword">for</span> im_part, value <span class="keyword">in</span> zip(captcha.handle_split_image(), values):</span><br><span class="line">                <span class="comment"># 进行md5随机取名保存</span></span><br><span class="line">                m = hashlib.md5()</span><br><span class="line">                m.update(<span class="string">"&#123;&#125;&#123;&#125;"</span>.format(time.time(), value).encode(<span class="string">'utf8'</span>))</span><br><span class="line">                <span class="comment"># captcha_chars/3/asdz.png</span></span><br><span class="line">                im_part.save(<span class="string">"captcha_chars/&#123;&#125;/&#123;&#125;.png"</span>.format(value, m.hexdigest()))</span><br><span class="line">                </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">generate_models</span><span class="params">()</span>:</span></span><br><span class="line">    iconset = list(<span class="string">'0123456789'</span>)</span><br><span class="line">    <span class="comment"># 将图像数据转为向量数据并保存</span></span><br><span class="line">    image_data = []</span><br><span class="line">    <span class="keyword">for</span> letter <span class="keyword">in</span> iconset:</span><br><span class="line">        <span class="keyword">try</span>:</span><br><span class="line">            <span class="comment"># 遍历字符3下面所有的图片</span></span><br><span class="line">            <span class="keyword">for</span> img <span class="keyword">in</span> os.listdir(<span class="string">'captcha_chars/&#123;&#125;/'</span>.format(letter)):</span><br><span class="line">                <span class="keyword">if</span> img != <span class="string">"Thumbs.db"</span> <span class="keyword">and</span> img != <span class="string">".DS_Store"</span>:</span><br><span class="line">                    <span class="comment"># 得到其图片向量</span></span><br><span class="line">                    temp = build_vector(Image.open(<span class="string">"captcha_chars/&#123;&#125;/&#123;&#125;"</span>.format(letter, img)))</span><br><span class="line">                    <span class="comment"># models_list中添加[&#123;1: [0,1,1...]&#125;]</span></span><br><span class="line">                    image_data.append(&#123;letter: temp&#125;)</span><br><span class="line">        <span class="keyword">except</span> FileNotFoundError:</span><br><span class="line">            print(letter)</span><br><span class="line">image_data = [ &#123;<span class="number">1</span>: [<span class="number">0</span>,<span class="number">1</span>,<span class="number">1</span>,...]&#125;, &#123;<span class="number">1</span>: [<span class="number">1</span>,<span class="number">0</span>,...]&#125;, ..., &#123;<span class="number">2</span>: [<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>,...]&#125;]</span><br><span class="line">    <span class="keyword">with</span> open(<span class="string">'image_data.json'</span>, <span class="string">'w'</span>) <span class="keyword">as</span> f:</span><br><span class="line">        json.dump(image_data, f)</span><br></pre></td></tr></table></figure><p>附：找到最合适的旋转角度</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">rotate_img</span><span class="params">(image)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    根据图像在x轴方向投影大小确定字符的摆放方向</span></span><br><span class="line"><span class="string">    -45~+45循环, 当x轴上投影得到的白色像素数量最小，此时就是正确的角度</span></span><br><span class="line"><span class="string">    :param image: PIL.Image object</span></span><br><span class="line"><span class="string">    :return: rotated Image object</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    min_count = <span class="number">1000</span></span><br><span class="line">    final_angle = <span class="number">0</span></span><br><span class="line">    <span class="keyword">for</span> angle <span class="keyword">in</span> range(<span class="number">-45</span>, <span class="number">45</span>):</span><br><span class="line">        x_count = <span class="number">0</span></span><br><span class="line">        ti = image.rotate(angle, expand=<span class="keyword">True</span>)</span><br><span class="line">        <span class="keyword">for</span> x <span class="keyword">in</span> range(ti.width):</span><br><span class="line">            <span class="comment"># 当前y列只要有一个像素为white, 那么就+1结束, 进行下一列判断</span></span><br><span class="line">            <span class="keyword">for</span> y <span class="keyword">in</span> range(ti.height):</span><br><span class="line">                <span class="keyword">if</span> ti.getpixel((x, y)) == WHITE:</span><br><span class="line">                    x_count += <span class="number">1</span></span><br><span class="line">                    <span class="keyword">break</span></span><br><span class="line">        <span class="keyword">if</span> x_count &lt; min_count:</span><br><span class="line">            min_count = x_count</span><br><span class="line">            final_angle = angle</span><br><span class="line">    image = image.rotate(final_angle, expand=<span class="keyword">False</span>)</span><br><span class="line">    <span class="keyword">return</span> image</span><br></pre></td></tr></table></figure></li></ul><h2 id="rss-monitor"><a class="markdownIt-Anchor" href="#rss-monitor"></a> <strong><a href="https://github.com/Freedomisgood/Rss-Monitor" target="_blank" rel="noopener">Rss-Monitor</a></strong></h2><blockquote><p>一个通用的RSS订阅通知器，和**<a href="https://github.com/DIYgod/RSSHub" target="_blank" rel="noopener">RSSHub</a>**一起食用效果更好哦。<br>RSS(Really Simple Syndication,简易信息聚合)是一种描述和同步网站内容的格式你可以认为是一种定制个性化推送信息的服务。它能够解决你漫无目的的浏览网页的问题。</p></blockquote><ul><li><p>动态导入模块</p></li><li><p>获得Rss链接信息——<a href="https://pypi.python.org/pypi/feedparser/" target="_blank" rel="noopener">feedparser</a>库</p><blockquote><p>feedparser是一个Python的Feed解析库，可以处理RSS ，CDF，Atom 。使用它我们可从任何 RSS 或 Atom 订阅源得到标题、链接和文章的条目了。</p></blockquote><p>标准的item格式:</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="name">item</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">title</span>&gt;</span>&lt;![CDATA[厦门公交车放火案死者名单公布&lt;br/&gt;警方公布嫌犯犯罪证据]]&gt;<span class="tag">&lt;/<span class="name">title</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">link</span>&gt;</span>http://www.infzm.com/content/91404<span class="tag">&lt;/<span class="name">link</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">description</span>&gt;</span>&lt;![CDATA[6月11日下午，厦门BRT公交车放火案47名死亡者名单公布。厦门政府新闻办6月10日发布消息称，有证据表明，陈水总携带汽油上了闽DY7396公交车。且有多名幸存者指认其在车上纵火，致使整部车引起猛烈燃烧。经笔迹鉴定，陈水总6月7日致妻、女的两封绝笔书系陈水总本人所写。]]&gt;<span class="tag">&lt;/<span class="name">description</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">category</span>&gt;</span>南方周末-热点新闻<span class="tag">&lt;/<span class="name">category</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">author</span>&gt;</span>infzm<span class="tag">&lt;/<span class="name">author</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">pubDate</span>&gt;</span>2013-06-11 11:24:32<span class="tag">&lt;/<span class="name">pubDate</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">item</span>&gt;</span></span><br></pre></td></tr></table></figure><p>feedparser模块api</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> feedparser</span><br><span class="line">d=feedparser.parse(<span class="string">'http://feed.cnblogs.com/blog/sitehome/rss'</span>)</span><br><span class="line"><span class="comment"># d.feed为数据字典, 与entries相比一个是dict一个是list, 其次entries中是更加凝练点的数据</span></span><br><span class="line"><span class="comment">#   输出字典中的键值有哪些，一共有10中如下：</span></span><br><span class="line">[<span class="string">'feed'</span>, <span class="string">'status'</span>, <span class="string">'version'</span>, <span class="string">'encoding'</span>, <span class="string">'bozo'</span>, <span class="string">'headers'</span>, <span class="string">'href'</span>, <span class="string">'namespaces'</span>, <span class="string">'entries'</span>, <span class="string">'bozo_exception'</span>]</span><br><span class="line">&#123;<span class="string">'feed'</span>: &#123;&#125;, <span class="string">'encoding'</span>: <span class="string">u'utf-8'</span>, <span class="string">'bozo'</span>: <span class="number">1</span>, <span class="string">'version'</span>: <span class="string">u''</span>, <span class="string">'namespaces'</span>: &#123;&#125;, <span class="string">'entries'</span>: [], <span class="string">'bozo_exception'</span>: SAXParseException(<span class="string">'no element found'</span>,)&#125;</span><br><span class="line"><span class="comment"># d.channel下的属性为channel中中的字段，比如title、link、description、generator、webMaster、lastBuildDate</span></span><br><span class="line">&gt;&gt;&gt;d[<span class="string">'feed'</span>][<span class="string">'title'</span>]</span><br><span class="line"><span class="string">'博客园_首页'</span></span><br><span class="line">&gt;&gt;&gt;d.feed.title    <span class="comment">#通过属性的方式访问</span></span><br><span class="line"><span class="string">'博客园_首页'</span></span><br><span class="line">&gt;&gt;&gt;d.feed.subtitle</span><br><span class="line"><span class="string">'代码改变世界'</span></span><br><span class="line">&gt;&gt;&gt;d.feed.link</span><br><span class="line"><span class="string">'uuid:1b90fd0c-6080-4ea5-86b1-b87c64b95d69;id=4466'</span></span><br><span class="line">d.entries <span class="comment">#该属性类型为列表，表示一组文章的条目</span></span><br><span class="line">&gt;&gt;&gt;type(d.entries)    <span class="comment">#类型为列表, 为item标签中的内容</span></span><br><span class="line">&lt;<span class="class"><span class="keyword">class</span> '<span class="title">list</span>'&gt;</span></span><br><span class="line"><span class="class">&gt;&gt;&gt;<span class="title">len</span><span class="params">(d.entries)</span>   #一共20篇文章</span></span><br><span class="line"><span class="class">20</span></span><br></pre></td></tr></table></figure><p>每个RSS和Atom订阅源都包含一个标题（d.feed.title）和一组文章条目(d.entries)</p><p><strong>通常</strong>每个文章条目都有一段摘要（d.entries[i].summary）,或者是包含了条目中实际文本的描述性标签（d.entries[i].description）</p></li><li><p>自定义添加Rss链接-&gt;<a href="https://docs.rsshub.app/joinus/quick-start.html%E3%80%81" target="_blank" rel="noopener">https://docs.rsshub.app/joinus/quick-start.html、</a><a href="https://github.com/DIYgod/RSSHub/pull/1768/files#" target="_blank" rel="noopener"><br>添加浙江工商大学教务处和学校首页通知</a></p><blockquote><p>除了按照Rsshub来自定义Rss订阅源以外，还可以使用 <a href="https://github.com/huginn/huginn" target="_blank" rel="noopener">Huginn</a>、Feed43等工具，如<a href="https://www.jianshu.com/p/1024956c8396" target="_blank" rel="noopener">用Feed43为教务处网站制作RSS</a></p></blockquote><ol><li><p>暴露路由链接：在 <a href="https://github.com/DIYgod/RSSHub/blob/master/lib/router.js" target="_blank" rel="noopener">/lib/router.js (opens new window)</a>里添加路由</p></li><li><p>编写数据源脚本：在 <a href="https://github.com/DIYgod/RSSHub/tree/master/lib/routes" target="_blank" rel="noopener">/lib/routes/ (opens new window)</a>中的路由对应路径下创建新的 js 脚本：</p><ol><li>使用 got 请求 HTML 数据：</li><li>使用 cheerio 解析返回的 HTML:</li><li>使用 map 遍历数组，解析出每一个 item 的结果 ——对数据进行进一步处理，生成符合 RSS 规范的对象，把获取的标题、链接、描述、发布时间等数据赋值给 ctx.state.data, <a href="https://docs.rsshub.app/joinus/quick-start.html#%E7%94%9F%E6%88%90-rss-%E6%BA%90" target="_blank" rel="noopener">生成 RSS 源</a>：</li></ol><blockquote><p>生成 RSS 源原理：获取到的数据赋给 ctx.state.data, 然后数据会经过 <a href="https://github.com/DIYgod/RSSHub/blob/master/lib/middleware/template.js" target="_blank" rel="noopener">template.js (opens new window)</a>中间件处理，最后传到 <a href="https://github.com/DIYgod/RSSHub/blob/master/lib/views/rss.art" target="_blank" rel="noopener">/lib/views/rss.art (opens new window)</a>来生成最后的 RSS 结果，每个字段的含义如下：<a href="https://docs.rsshub.app/joinus/quick-start.html#ti-jiao-xin-de-rsshub-gui-ze-bian-xie-jiao-ben" target="_blank" rel="noopener">https://docs.rsshub.app/joinus/quick-start.html#ti-jiao-xin-de-rsshub-gui-ze-bian-xie-jiao-ben</a></p></blockquote></li></ol><p>3.添加脚本文档: <code>&lt;Route author=&quot;HenryQW&quot; example=&quot;/github/issue/DIYgod/RSSHub&quot; path=&quot;/github/issue/:user/:repo&quot; :paramsDesc=&quot;['用户名', '仓库名']&quot; /&gt;</code></p></li></ul><h2 id="ddt-sharp-shooter"><a class="markdownIt-Anchor" href="#ddt-sharp-shooter"></a> <strong><a href="https://github.com/boring-plans/ddt-sharp-shooter" target="_blank" rel="noopener"> ddt-sharp-shooter</a></strong></h2><blockquote><p>使用到的库: <code>screeninfo、pillow、ddddocr、pynput</code></p><ul><li>pynput: 控制和监视输入设备；类似的有PyHook3（监视键鼠）、pywin32 （模拟键鼠）</li><li>ddddocr：识别验证码，这边用来识别数字</li></ul></blockquote><ol><li><p>进程间通信。 Tkinter界面开启mainloop进程，其中又开辟出一个子线程来侦听其他进程发送的数据消息，然后通过<code>tk.Text</code>控件来展示</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> multiprocessing</span><br><span class="line"><span class="keyword">import</span> threading</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> tkinter</span><br><span class="line"><span class="comment"># 声明全局对象及类型</span></span><br><span class="line">_tk: tkinter.Tk</span><br><span class="line">_text: tkinter.Text</span><br><span class="line">_terminate = <span class="keyword">False</span></span><br><span class="line">_queue: multiprocessing.Queue</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">update_text</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""子线程侦听进程消息"""</span></span><br><span class="line">    <span class="keyword">while</span> <span class="keyword">not</span> _terminate:</span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> _queue.empty():</span><br><span class="line">            text = _queue.get(<span class="keyword">False</span>)</span><br><span class="line">            append_text(text)</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            time.sleep(<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">append_text</span><span class="params">(text)</span>:</span></span><br><span class="line">    _text.config(state=<span class="string">'normal'</span>)</span><br><span class="line">    _text.insert(<span class="string">'end'</span>, <span class="string">f'\n<span class="subst">&#123;text&#125;</span>'</span>)</span><br><span class="line">    _text.see(<span class="string">'end'</span>)</span><br><span class="line">    _text.config(state=<span class="string">'disabled'</span>)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(gui_queue)</span>:</span></span><br><span class="line">    <span class="keyword">global</span> _tk, _text, _queue, _screen_size</span><br><span class="line">    _queue = gui_queue</span><br><span class="line">    <span class="comment"># ...</span></span><br><span class="line">    threading.Thread(target=update_text).start()</span><br><span class="line">    </span><br><span class="line">    _tk.mainloop()</span><br></pre></td></tr></table></figure></li><li><p><code>py2app setup</code>，在macos下创建python应用， <code>python setup.py py2app</code></p></li><li><p>对纯数字识别结果进行清洗</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">recognize_digits</span><span class="params">(image: bytes)</span>:</span></span><br><span class="line">    ocr = ddddocr.DdddOcr(show_ad=<span class="keyword">False</span>)</span><br><span class="line">    result = ocr.classification(image)</span><br><span class="line">    <span class="keyword">return</span> wash_digits(result)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">wash_digits</span><span class="params">(digits: str)</span>:</span></span><br><span class="line">    <span class="string">"""由于不会出现非数字, 所以对易识别错误的字符进行替换"""</span></span><br><span class="line">    washed = digits \</span><br><span class="line">        .replace(<span class="string">'g'</span>, <span class="string">'9'</span>).replace(<span class="string">'q'</span>, <span class="string">'9'</span>) \</span><br><span class="line">        .replace(<span class="string">'l'</span>, <span class="string">'1'</span>).replace(<span class="string">'i'</span>, <span class="string">'1'</span>) \</span><br><span class="line">        .replace(<span class="string">'z'</span>, <span class="string">'2'</span>) \</span><br><span class="line">        .replace(<span class="string">'o'</span>, <span class="string">'0'</span>)</span><br><span class="line">    <span class="keyword">return</span> re.sub(<span class="string">r'\D'</span>, <span class="string">'0'</span>, washed)</span><br></pre></td></tr></table></figure></li></ol><h1 id="文章记录"><a class="markdownIt-Anchor" href="#文章记录"></a> 文章记录</h1><h2 id="github优秀的文章项目"><a class="markdownIt-Anchor" href="#github优秀的文章项目"></a> Github优秀的文章项目</h2><ul><li><strong><a href="https://github.com/danistefanovic/build-your-own-x" target="_blank" rel="noopener">build-your-own-x</a></strong>——自己造轮子</li><li><strong><a href="https://github.com/donnemartin/system-design-primer" target="_blank" rel="noopener">system-design-primer</a></strong>——系统架构设计</li><li><strong><a href="https://github.com/doocs/advanced-java" target="_blank" rel="noopener">advanced-java</a></strong>——互联网 Java 工程师进阶知识完全扫盲</li><li><a href="https://juejin.cn/post/6844903624062009352" target="_blank" rel="noopener">https://juejin.cn/post/6844903624062009352</a></li><li></li></ul><h2 id="go好用的三方库"><a class="markdownIt-Anchor" href="#go好用的三方库"></a> Go好用的三方库</h2><ul><li><p>丰富原生数据结构map、slice的工具库：<a href="https://github.com/samber/lo" target="_blank" rel="noopener">https://github.com/samber/lo</a></p></li><li><p>输入参数解析库: <a href="https://github.com/jessevdk/go-flags" target="_blank" rel="noopener">go-flags</a></p></li><li><p>操作excel的库：<a href="https://github.com/qax-os/excelize" target="_blank" rel="noopener">excelize</a></p></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h1 id=&quot;爬虫系列&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#爬虫系列&quot;&gt;&lt;/a&gt; 爬虫系列&lt;/h1&gt;
&lt;h2 id=&quot;抢购-秒杀&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#抢购-秒杀&quot;&gt;&lt;/a&gt; 抢购、秒杀&lt;
      
    
    </summary>
    
    
      <category term="代码阅读" scheme="https://nymrli.top/tags/%E4%BB%A3%E7%A0%81%E9%98%85%E8%AF%BB/"/>
    
  </entry>
  
  <entry>
    <title>Python进阶</title>
    <link href="https://nymrli.top/2022/03/27/Python%E8%BF%9B%E9%98%B6/"/>
    <id>https://nymrli.top/2022/03/27/Python进阶/</id>
    <published>2022-03-27T14:30:39.000Z</published>
    <updated>2022-05-25T16:05:49.058Z</updated>
    
    <content type="html"><![CDATA[<h2 id="xxxatrr家族"><a class="markdownIt-Anchor" href="#xxxatrr家族"></a> xxxatrr家族</h2><h3 id="hasattr"><a class="markdownIt-Anchor" href="#hasattr"></a> hasattr()</h3><blockquote><p>用于判断对象是否包含对应的属性。</p></blockquote><p>hasattr 语法：<code>hasattr(object, name)</code></p><h3 id="setattr"><a class="markdownIt-Anchor" href="#setattr"></a> setattr()</h3><blockquote><p><strong>setattr()</strong> 函数对应函数 <a href="https://www.runoob.com/python/python-func-getattr.html" target="_blank" rel="noopener">getattr()</a>，用于设置属性值，该属性不一定是存在的。如果属性不存在会创建一个新的对象属性，并对属性进行赋值。</p></blockquote><p>setattr() 语法：<code>setattr(object, name, value)</code></p><h3 id="getattr"><a class="markdownIt-Anchor" href="#getattr"></a> getattr()</h3><blockquote><p><strong>getattr()</strong> 函数用于返回一个对象<strong>属性值</strong>。</p></blockquote><p><code>getattr(object, name[, default])</code></p><ul><li>object – 对象。</li><li>name – 字符串，对象属性。</li><li>default – 默认返回值，如果不提供该参数，在没有对应属性时，将触发 <u>AttributeError</u>。</li></ul><h2 id="__get____getattr____getattribute__"><a class="markdownIt-Anchor" href="#__get____getattr____getattribute__"></a> <a href="https://www.cnblogs.com/pyhai/p/9243551.html" target="_blank" rel="noopener"><code>__get__,__getattr__,__getattribute__</code>及区别</a></h2><h3 id="1object__getattribute__self-name"><a class="markdownIt-Anchor" href="#1object__getattribute__self-name"></a> <strong>1.<code>object.__getattribute__(self, name)</code></strong></h3><p>无条件被调用，通过实例访问属性、函数（点和getattr函数都会触发）。如果class中定义了<code>__getattribute__()</code>和<code>__getattr__()</code>，则只有在显式调用或引发AttributeError异常才会调用<code>__getattr__()</code></p><p>注：</p><ul><li>只要定义了<code>__getattribute__</code>方法，不管你访问一个存在的还是不存在的属性，都由这个方法返回，比如访问<code>t.a</code>，虽然a存在，但是只要定义了这个访问，那么就不是访问最开始的a了</li><li>如果<code>__getattribute__</code>抛出了<code>AttributeError</code>异常，并且定了了<code><strong>getattr</strong></code>函数，那么会调用<code><strong>getattr</strong></code>这个函数并返回getattr函数的返回值</li><li>属性访问的一个大致优先级是：<code>__getattribute__</code> &gt; <code>__getattr__</code> &gt; <code>__dict__</code></li></ul><h3 id="2object__getattr__self-name"><a class="markdownIt-Anchor" href="#2object__getattr__self-name"></a> <strong>2.<code>object.__getattr__(self, name)</code></strong></h3><p>为内置方法，当使用点号获取实例属性时，如果<u>属性不存在</u>(找不到attribute)的时候，会调用<code>__getattr__</code>，返回一个值或AttributeError异常。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">p</span><span class="params">()</span>:</span></span><br><span class="line">    s = <span class="number">2</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__getattr__</span><span class="params">(self, item)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> <span class="string">"hello"</span></span><br><span class="line">d = p()</span><br><span class="line">print(d.s)<span class="comment"># ==&gt; 2</span></span><br><span class="line">print(d.p)  <span class="comment"># ==&gt; hello</span></span><br></pre></td></tr></table></figure><p>注：如果属性不存在，则不管是否有<code>__getattribute__</code>，都会调用<code>__getattr__</code></p><h3 id="3object__get__self-instance-owner"><a class="markdownIt-Anchor" href="#3object__get__self-instance-owner"></a> <strong>3.<code>object.__get__(self, instance, owner)</code></strong></h3><blockquote><p>Python 内置的 <code>property</code> 函数可以说是最著名的描述器之一，几乎所有讲述描述器的文章都会拿它做例子。而我们可以通过实现<code>__set__</code>、<code>__get__</code>、<code>__delete__</code>来实现自己的描述器。</p><ul><li>描述器只对新式类起作用；</li></ul><p>方法的第一个参数instance是实际拥有者的descriptor<strong>实例</strong>，如果是Descriptor类(定义了描述器协议的描述器类)则为<code>None</code>，第二个参数owner是实际所属的<strong>类本身</strong>(所有者类)。</p></blockquote><p>一个类只要实现了<code>__get__、 __set__,__delete__</code>中任意一个方法，我们就可以叫它描述器（descriptor），他是一个可以<strong>描述一个属性操作</strong>的<strong>对象</strong>。如果只定义了<code>__get__</code>我们叫非资料描述器（non-data descriptor），如果<code>__set__、 __delete__</code>：任意一个或者同时出现，叫资料描述器（data descriptor）。</p><p>首先明确一点，拥有<code>__get__</code>的类，应该**（也可以说是必须）产生一个实例**，并且这个实例是另外一个类的<strong>类属性</strong>（注意一定是类属性，通过self的方式产生就不属于<code>__get__</code>范畴了）。也就是说拥有这个方法的类，那么它的实例应该属于另外一个类/对象的一个属性。</p><p>owner是所有者的类，instance是访问descriptor的实例，如果不是通过实例访问，而是通过类访问的话，instance则为None。（descriptor的实例自己访问自己是不会触发<code>__get__</code>，而会触发<code>__call__</code>，只有descriptor作为其它类的属性才有意义。）（所以下文的d是作为C2的一个属性被调用）</p><p>将描述器作为一个独立对象，并不能展现出描述器的魔力，只有在描述器作为另一个对象的属性的时候，描述器的魔力才能真正展现出来。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestDes</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__get__</span><span class="params">(self, instance, owner)</span>:</span></span><br><span class="line">        print(instance, owner)</span><br><span class="line">        <span class="keyword">return</span> <span class="string">'TestDes:__get__'</span></span><br><span class="line"> </span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestMain</span>:</span></span><br><span class="line">    des = TestDes()</span><br><span class="line"> </span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    t = TestMain()</span><br><span class="line">    <span class="comment"># des虽然被定义成类变量，但t.des还是跟实例相关的，对其增删改都会走到其描述器对象的__get__、__set__、__delete__中处理</span></span><br><span class="line">    print(t.des)</span><br><span class="line">    print(TestMain.des)</span><br><span class="line"><span class="comment"># 其中TestDes定义了__get__方法，在TestMain中，定义了一个类属性des，是TestDes的一个实例，我们访问t.des或者TestMain.des的时候访问的就是访问了TestDes的__get__方法。</span></span><br></pre></td></tr></table></figure><p>▲描述器往往以装饰器的方式被使用，导致二者常被混淆。描述器类和不带参数的装饰器类一样，都传入函数对象作为参数，并返回一个<strong>类实例</strong>，所不同的是，装饰器类返回 <u>callable 的实例</u>，描述器则返回<u>描述器实例</u>。</p><p>Q:如果实例中有和描述器重名的属性 <code>x</code> 怎么办？</p><p>A:资料和非资料描述器的区别在于，相对于实例字典的优先级不同。当描述器和实例字典中的某个属性重名，按访问优先级，资料描述器 &gt; 同名实例字典中的属性 &gt; 非资料描述器 or (数据描述符 &gt; 实例变量 &gt; 非数据描述符)，优先级小的会被大的覆盖。==&gt;类的方法实际就是一个仅实现了 <code>__get__()</code> 的非资料描述器，所以如果实例 <code>c</code> 中同时定义了名为 <code>foo</code> 的方法和属性，那么 <code>c.foo</code> 访问的是属性而非方法。</p><p>描述器more：<a href="https://www.jianshu.com/p/58f1df955bda" target="_blank" rel="noopener">Python 描述器解析</a></p><p>针对描述器的说明： 描述器是被<code>__getattribute__</code>调用的，如果重写了这个方法，将会阻止自动调用描述器，资料描述器总是覆盖了实例的<code>__dict__</code>， 非资料描述器可能覆盖实例的<code>__dict__</code>。</p><p>小结：访问存在的属性，如果是描述器，描述器生效</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestDes</span>:</span></span><br><span class="line">    <span class="comment"># 方法的第一个参数是实际拥有者的实例，如果没有则为None，第二个参数是实际所属的类。</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__get__</span><span class="params">(self, instance, owner)</span>:</span></span><br><span class="line">        print(instance, owner)</span><br><span class="line">        <span class="keyword">return</span> <span class="string">'TestDes:__get__'</span></span><br><span class="line"> </span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestMain</span>:</span></span><br><span class="line">    des = TestDes()</span><br><span class="line"> </span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    t = TestMain()</span><br><span class="line">    print(t.des)</span><br><span class="line">    print(TestMain.des)<span class="comment"># 调用时instance输出None</span></span><br></pre></td></tr></table></figure><ul><li>非资料描述器，也就是只有<code>__get__</code>，不管是类还是实例去访问，默认都获得的是<code>__get__</code>的返回值，但是，如果中间有任何一次重新赋值<code>(t.des = 1)</code>，那么，这个实例获得的是新的值(对象)，已经和原来的描述器完全脱离了关系(描述器<code>__get__</code>函数失效)</li><li>资料描述器，比如有<code>__set__</code>方法，后期通过实例对描述器进行赋值，那么访问的是<code>__set__</code>，并且永远关联起来==&gt;针对上述问题的修复。但是如果通过修改类属性的方式复制(TestMain.des = 1)，那么也会被重新获取新的值(对象)，即<code>__set__</code>函数失效。</li></ul><p><strong>总结：</strong></p><ul><li>可以看出，每次通过实例访问属性，都会经过<code>__getattribute__</code>函数。而当属性不存在时，仍然需要访问<code>__getattribute__</code>，不过接着要访问<code>__getattr__</code>，就好像是一个异常处理函数。</li><li>每次访问descriptor（即实现了<code>__get__</code>的类），都会先经过<code>__get__</code>函数。</li><li>需要注意的是，当使用类访问不存在的变量是，不会经过<code>__getattr__</code>函数。而descriptor不存在此问题，只是把instance标识为none而已。</li></ul><p>至于三者区别，首先关注：a.x时发生了什么?=&gt;属性的lookup顺序如下:</p><ul><li>如果重载了<code>__getattribute__</code>,则调用.</li><li><code>a.__dict__</code>, 实例中是不允许有descriptor的,所以不会遇到descriptor</li><li><code>A.__dict__</code>, 也即<code>a.__class__.__dict__</code>.如果遇到了descriptor,优先调用descriptor.</li><li>沿着继承链搜索父类.搜索<code>a.__class__.__bases__</code>中的所有<code>__dict__</code>. 如果有多重继承且是菱形继承的情况,按MRO(Method Resolution Order)顺序搜索.</li></ul><p>参考：</p><ul><li><a href="https://www.cnblogs.com/andy1031/p/10923834.html" target="_blank" rel="noopener">python的__get__方法看这一篇就足够了</a></li><li>more：<a href="https://www.cnblogs.com/Meanwey/p/9898222.html" target="_blank" rel="noopener">Python进阶-----描述符(<strong>get</strong>(),<strong>set</strong>(),<strong>delete</strong>())</a></li></ul><h3 id="__getitem__"><a class="markdownIt-Anchor" href="#__getitem__"></a> <code>__getitem__</code></h3><blockquote><p>当实例对象做p[key] 运算时，会调用类中的方法<code>__getitem__</code>，得到<code>__getitem__</code>的返回值</p></blockquote><ul><li><strong>点运算符</strong> 用于访问任何对象的属性；</li><li><strong>方括号运算符</strong> 表示法用于访问集合的成员；</li></ul><p>区别：点运算符，用来获取语言内置的属性、方法等等；而方括号运算符则是用来获取用户定义的数据，一般是字典、列表、元组以及字符串的成员。</p><h2 id="__dict__"><a class="markdownIt-Anchor" href="#__dict__"></a> <code>__dict__</code></h2><blockquote><p>Python中大多以对象的形式存在，而对象的<strong>属性</strong>则是存在<code>__dict__</code>属性中</p></blockquote><ul><li>类的静态函数、类函数、普通函数、全局变量以及一些内置的属性都是放在**类<code>__dict__</code>**里的</li><li><strong>对象</strong>的<code>__dict__</code>中存储了一些self.xxx的一些东西</li></ul><p>注意</p><ul><li>内置的数据类型没有<code>__dict__</code>属性，如list、dict、int</li><li>每个类有自己的<code>__dict__</code>属性，就算存着继承关系，父类的<code>__dict__</code> 并不会影响子类的<code>__dict__</code></li><li>对象也有自己的<code>__dict__</code>属性， <a href="http://xn--self-p07f497c.xxx" target="_blank" rel="noopener">存储self.xxx</a> 信息，父子类对象公用<code>__dict__</code></li><li>对象的<code>__dict__</code>优先于类的<code>__dict__</code>调用，如<code>self.__dict__['a']=1</code>，而<code>Class.__dict__['a']=&quot;func&quot;</code>，则c.a输出的是1，而不是func</li></ul><h2 id="可变参数与unpack"><a class="markdownIt-Anchor" href="#可变参数与unpack"></a> 可变参数与unpack</h2><ol><li><p>函数的可变参数</p><p>当函数的参数前面有一个星号<code>*</code>的时候表示这是一个可变的位置参数，两个星号<code>**</code>表示是可变的关键字参数。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">saySomething</span><span class="params">(word, *args, **kwargs)</span>:</span></span><br><span class="line"><span class="keyword">pass</span></span><br><span class="line"><span class="comment"># word会被赋值成hello word, </span></span><br><span class="line"><span class="comment"># mrli和mrdu被打包进args数组中</span></span><br><span class="line"><span class="comment"># size=15; color=white会被作为kv传到kwargs的字典中</span></span><br><span class="line">saySomething(<span class="string">"hello word"</span>, <span class="string">"mrli"</span>, <span class="string">"mrdu"</span>, size=<span class="number">15</span>, color=<span class="string">"white"</span>)</span><br></pre></td></tr></table></figure></li><li><p>unpack参数</p><p>星号<code>*</code>把序列/集合解包（unpack）成位置参数，两个星号<code>**</code>把字典解包成关键字参数。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">txtOptions = &#123;</span><br><span class="line"><span class="string">"fontSize"</span>: <span class="number">15</span>,</span><br><span class="line"><span class="string">"color"</span>: <span class="string">"white"</span></span><br><span class="line">&#125;</span><br><span class="line">addtional_text = (<span class="string">"mrli"</span>, <span class="string">"mrdu"</span>)</span><br><span class="line"><span class="comment"># unpack的形式传参数</span></span><br><span class="line">saySomething(<span class="string">"hello word"</span>, *addtional_text, **txtOptions)</span><br><span class="line"><span class="comment"># 两句话是等价的</span></span><br><span class="line">saySomething(<span class="string">"hello word"</span>, <span class="string">"mrli"</span>, <span class="string">"mrdu"</span>, size=<span class="number">15</span>, color=<span class="string">"white"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 拓展-我们经常在开源库的doc中看到某个函数能接收xxx参数，但是点进去一看，最底层的参数传的是**kwargs。按道理我们是什么参数都可以传的，大不了函数不解析这些无用参数罢了，但是实际上当我们传API中没有写到的参数时会报错。 ==&gt; 这个是由于kwargs能传递的参数受限于最底层的参数需求，以requests.get为例</span></span><br><span class="line"><span class="comment"># requests.py</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get</span><span class="params">(url, params=None, **kwargs)</span>:</span></span><br><span class="line">    <span class="keyword">return</span> request(<span class="string">'get'</span>, url, params=params, **kwargs)</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">request</span><span class="params">(method, url, **kwargs)</span>:</span></span><br><span class="line">    <span class="keyword">return</span> session.request(method=method, url=url, **kwargs)</span><br><span class="line"><span class="comment"># get中能传的**关键字参数**实际上只能是这些关键字参数</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">request</span><span class="params">(self, method, url,</span></span></span><br><span class="line"><span class="function"><span class="params">            params=None, data=None, headers=None, cookies=None, files=None,</span></span></span><br><span class="line"><span class="function"><span class="params">            auth=None, timeout=None, allow_redirects=True, proxies=None,</span></span></span><br><span class="line"><span class="function"><span class="params">            hooks=None, stream=None, verify=None, cert=None, json=None)</span>:</span></span><br><span class="line"><span class="comment"># Python也是通过这两个特性从而实现了**重写overrided**</span></span><br></pre></td></tr></table></figure><ul><li>至于星号(<code>*</code>)还有一个用途是来明确写明位置参数的截断位置，<code>*</code>后的参数也不能再以位置参数的形式给出，必须以关键字参数的形式给出——见<a href="https://blog.csdn.net/weixin_40796925/article/details/107654347" target="_blank" rel="noopener">Python3 函数参数列表单独一个星号 * 的作用</a></li></ul></li></ol><p>类比Go中的<code>...</code>语法</p><h2 id="装饰器"><a class="markdownIt-Anchor" href="#装饰器"></a> 装饰器</h2><blockquote><p>装饰器(Decorators)是 Python 的一个重要部分，通过他们可以在不用更改原函数的代码前提下，修改、拓展其他函数的功能的函数，他们有助于让我们的代码更简洁。==&gt;切面编程的方式</p></blockquote><p><strong>闭包</strong></p><blockquote><p>Python查找变量会一层层地向外层查找，直到global全局也没有时raise <code>NameError: name 'xxx' is not defined</code></p></blockquote><p>在装饰器中运用到了闭包的思想。闭包，一句话说就是，在函数中再嵌套一个函数，并且引用<strong>外部函数</strong>的变量，这就是一个闭包了。(从而使得内部函数和对其周围状态（<strong>lexical environment，词法环境</strong>）的引用捆绑在一起)</p><p>执行闭包后，闭包实例将会维持了一个词法环境，其中包含对局部变量的引用，使得原本就应该失效的局部变量仍然存活。</p><p><strong>闭包的应用</strong></p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">makeAdder</span>(<span class="params">x</span>) </span>&#123;</span><br><span class="line">  <span class="keyword">return</span> <span class="function"><span class="keyword">function</span>(<span class="params">y</span>) </span>&#123;</span><br><span class="line">    <span class="keyword">return</span> x + y;</span><br><span class="line">  &#125;;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> add5 = makeAdder(<span class="number">5</span>);</span><br><span class="line"><span class="keyword">var</span> add10 = makeAdder(<span class="number">10</span>);</span><br><span class="line"></span><br><span class="line"><span class="built_in">console</span>.log(add5(<span class="number">2</span>));  <span class="comment">// 7</span></span><br><span class="line"><span class="built_in">console</span>.log(add10(<span class="number">2</span>)); <span class="comment">// 12</span></span><br></pre></td></tr></table></figure><p>从本质上讲，<code>makeAdder</code> 是一个函数工厂 — 他创建了将指定的值和它的参数相加求和的函数。在上面的示例中，我们使用函数工厂创建了两个新函数 — 一个将其参数和 5 求和，另一个和 10 求和。</p><p><code>add5</code> 和 <code>add10</code> 都是闭包。它们共享相同的函数定义，但是保存了不同的词法环境。在 <code>add5</code> 的环境中，<code>x</code> 为 5。而在 <code>add10</code> 中，<code>x</code> 则为 10。</p><p>more： <a href="https://www.bilibili.com/video/BV1ZJ411y7Te?from=search&amp;seid=15415737541054747822&amp;spm_id_from=333.337.0.0" target="_blank" rel="noopener">5句口诀理解记忆Python闭包和装饰器</a></p><p>[<strong>装饰器类型</strong>](<a href="https://python3-cookbook.readthedocs.io/zh_CN/latest/c09/p05_define_decorator_with_user_adjustable_attributes.html" target="_blank" rel="noopener">9.5 可自定义属性的装饰器 — python3-cookbook 3.0.0 文档</a>)</p><p>闭包很有用，因为它允许将函数与其所操作的某些数据（环境）关联起来。<strong>外部函数相当于给内部函数提供了一个额外数据的执行环境</strong>，使得内部函数（被装饰函数）拥有了获得外层函数的数据or执行了外层函数。根据参数[或是叫环境不同]，可以分成如下几种装饰器：</p><ul><li><p>无参数</p><ul><li><p>常见的有输出日志</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> functools <span class="keyword">import</span> wraps</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">do_log</span><span class="params">(func)</span>:</span></span><br><span class="line"><span class="meta">    @wraps(func)</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">wrapper</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line">        print(<span class="string">"do before"</span>)</span><br><span class="line">        func(*args, **kwargs)</span><br><span class="line">        print(<span class="string">"do after"</span>)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">return</span> wrapper</span><br><span class="line"></span><br><span class="line"><span class="meta">@do_log</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">say</span><span class="params">(word)</span>:</span></span><br><span class="line">    print(word)</span><br><span class="line">say(<span class="string">"hello world"</span>)</span><br><span class="line"><span class="comment"># 等价于下面两局</span></span><br><span class="line"><span class="comment"># say = do_log(say)</span></span><br><span class="line"><span class="comment"># say("hello world") ===&gt; do_log(say)("hello world")</span></span><br></pre></td></tr></table></figure><p>这个可以类比JS</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">say</span>(<span class="params">word</span>)</span>&#123;</span><br><span class="line"><span class="built_in">console</span>.log(word);</span><br><span class="line">&#125;</span><br><span class="line">play = <span class="function"><span class="keyword">function</span>(<span class="params">w</span>)</span>&#123;</span><br><span class="line">    <span class="built_in">console</span>.log(<span class="string">"我要开始笑啦"</span>);</span><br><span class="line">    say(w);</span><br><span class="line">    <span class="built_in">console</span>.log(<span class="string">"我不笑了"</span>);</span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">// play相当于装饰了一层say, 同比上面的do_log也是。在之后的执行中play和do_log才是真正的执行函数, 只不过Python语法糖@，让do_log又赋值给了say, 所以我们又可以直接使用say，从而增加了函数原本的功能</span></span><br></pre></td></tr></table></figure></li><li><p>@语法糖：它放在函数开始定义的地方，这样就可以省略最后一步再次赋值的操作</p></li></ul></li><li><p>默认参数</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">createLogger</span><span class="params">()</span>:</span></span><br><span class="line">    logger = logging.getLogger(<span class="string">"decorate"</span>)</span><br><span class="line">    formatter = logging.Formatter(<span class="string">'%(asctime)s-%(name)s-%(levelname)s-%(message)s'</span>)</span><br><span class="line">    sh = logging.StreamHandler()</span><br><span class="line">    sh.setLevel(logging.DEBUG)</span><br><span class="line">    sh.setFormatter(formatter)</span><br><span class="line">    logger.addHandler(sh)</span><br><span class="line">    <span class="keyword">return</span> logger</span><br><span class="line">logger = createLogger()</span><br><span class="line">logger.warning(<span class="string">"hello"</span>)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">log_by_level</span><span class="params">(level)</span>:</span></span><br><span class="line">    <span class="comment"># 创建新环境：多提供了一个level变量</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">do_log</span><span class="params">(func)</span>:</span></span><br><span class="line"><span class="meta">        @wraps(func)</span></span><br><span class="line">        <span class="function"><span class="keyword">def</span> <span class="title">wrapper</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line">            getattr(logger, level)(<span class="string">"do before"</span>)</span><br><span class="line">            func(*args, **kwargs)</span><br><span class="line">            getattr(logger, level)(<span class="string">"do after"</span>)</span><br><span class="line">        <span class="keyword">return</span> wrapper</span><br><span class="line">    <span class="keyword">return</span> do_log</span><br><span class="line"></span><br><span class="line"><span class="meta">@log_by_level("warning")</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">say</span><span class="params">(word)</span>:</span></span><br><span class="line">    print(word)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 等价于</span></span><br><span class="line">do_log = log_by_level(<span class="string">"warning"</span>) <span class="comment"># 作用是提供了一个level=warning的环境</span></span><br><span class="line">say = do_log(say)</span><br><span class="line">say(<span class="string">"hello"</span>)</span><br></pre></td></tr></table></figure></li><li><p>可自定义属性的装饰器：</p><blockquote><p>允许用户提供参数在<strong>运行时</strong>控制装饰器行为。比如前端设置一个单选，可以控制日志输出级别，此时则需要在运行时修改装饰器行为</p></blockquote></li><li><p>可选参数</p><blockquote><p>既可以不传参数给它，比如 <code>@decorator</code> ， 也可以传递可选参数给它，比如 <code>@decorator(x,y,z)</code> 。</p></blockquote><p>实现见：<a href="https://python3-cookbook.readthedocs.io/zh_CN/latest/c09/p06_define_decorator_that_takes_optional_argument.html" target="_blank" rel="noopener">9.6 带可选参数的装饰器 — python3-cookbook 3.0.0 文档</a></p></li><li><p>类装饰器：</p><blockquote><p>使用一个装饰器去包装函数，返回的是一个可调用的<strong>实例</strong></p></blockquote><p>类装饰器主要依靠类的<code>__call__</code>方法</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ClassDecorator</span>:</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(func)</span>:</span></span><br><span class="line">        self.func = func</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">__call__</span><span class="params">(self)</span>:</span></span><br><span class="line">        print(<span class="string">"enter"</span>)</span><br><span class="line">        self.func()</span><br><span class="line">        print(<span class="string">"end"</span>)</span><br><span class="line"><span class="meta">@ClassDecorator</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">bar</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="keyword">print</span> (<span class="string">'bar'</span>)</span><br><span class="line"><span class="comment"># bar = ClassDecorator(bar)</span></span><br><span class="line"><span class="comment"># bar() ==&gt; ClassDecorator(bar)()  调用实例==&gt;即触发__call__方法</span></span><br></pre></td></tr></table></figure></li><li><p>类中装饰器:</p><blockquote><p>在类中定义装饰器，并将其作用在其他函数或方法上</p></blockquote><p>跟无参数的装饰器一样写法，只不过定义在类中；并且在使用的时候是以<code>@Decorator.decortae</code>的形式</p></li><li><p>类方法、静态方法装饰器：跟无参数的装饰器一样写法，但是需要在@staticmethod、@classmethod之前标注</p></li></ul><p>▲<strong>注意：</strong></p><ul><li><p>装饰器只会在<strong>函数定义</strong>时被调用一次</p></li><li><p>可以看到的是，在装饰器代码中总有一行<code>@wraps(func)</code>的代码，其目的：是你写了一个装饰器作用在某个函数上，但是这个函数的重要的元信息比如名字、文档字符串、注解和参数签名都丢失了----&gt;任何时候你定义装饰器的时候，都应该使用functools库中的@wraps装饰器来注解底层包装函数</p></li><li><p>装饰器的执行是依次的，离函数签名越近，则越先被定义</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@a</span></span><br><span class="line"><span class="meta">@b</span></span><br><span class="line"><span class="meta">@c</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">f</span> <span class="params">()</span>:</span></span><br><span class="line">    <span class="keyword">pass</span></span><br><span class="line"><span class="comment"># 等价于 f = a(b(c(f)))</span></span><br></pre></td></tr></table></figure></li></ul><p>总结：</p><p>理解装饰器应该从①<code>@</code>语法糖到底做了什么；②闭包是什么，有什么作用；③装饰器如何等价表示；来理解</p><h2 id="懒加载属性"><a class="markdownIt-Anchor" href="#懒加载属性"></a> <a href="https://www.jianshu.com/p/708dc26f9b92" target="_blank" rel="noopener">懒加载属性</a></h2><p><strong>描述符+<a href="#%E8%A3%85%E9%A5%B0%E5%99%A8">类装饰器</a>实现</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">LazyProperty</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, func)</span>:</span></span><br><span class="line">        <span class="comment"># 初始化func为serverHost函数</span></span><br><span class="line">        self.func = func</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__get__</span><span class="params">(self, instance, owner)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> instance <span class="keyword">is</span> <span class="keyword">None</span>:</span><br><span class="line">            <span class="keyword">return</span> self</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            value = self.func(instance)</span><br><span class="line">            <span class="comment"># 会将结果值通过setattr方法存入到instance实例的__dict__中</span></span><br><span class="line">            setattr(instance, self.func.__name__, value)</span><br><span class="line">            <span class="keyword">return</span> value</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">pass</span></span><br><span class="line"><span class="comment"># 返回一个LazyProperty实例 </span></span><br><span class="line"><span class="meta">    @LazyProperty</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">serverHost</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> os.environ.get(<span class="string">"HOST"</span>, setting.HOST)</span><br><span class="line"></span><br><span class="line">setting = namedtuple(<span class="string">"setting"</span>, [<span class="string">"HOST"</span>])</span><br><span class="line">setting.HOST = <span class="string">"g"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 测试一</span></span><br><span class="line">a = ConfigHandler()</span><br><span class="line">print(a.__dict__)</span><br><span class="line"><span class="comment"># 1. 注解先是创建了LazyProperty(serverHost)的实例</span></span><br><span class="line"><span class="comment"># 2. 再是语法糖进行了赋值serverHost = LazyProperty(serverHost)</span></span><br><span class="line"><span class="comment"># 3. 当第一次进行调用的时候, instance = configHandler**实例**, self.func(instance实例) == 调用serverHost(instance实例)从而获得了真正值value。而之后的 setattr处将self实例的__dict__中添加了serverHost-value，再次访问self.serverHost时, 已经不再是函数, 而是value值(serverHost不再从ConfigHandler.__dict__中取, 而是实例a.__dict__中取)</span></span><br><span class="line">print(a.serverHost)</span><br><span class="line">print(<span class="string">"say"</span>)</span><br><span class="line">print(a.__dict__)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 测试二</span></span><br><span class="line"><span class="comment"># 如果先执行类的__dict__能看到类的serverHost是一个**描述器对象实例**=&gt; 'serverHost': &lt;__main__.LazyProperty object at 0x0000020A1AEB7FD0&gt;</span></span><br><span class="line">print(ConfigHandler.__dict__)</span><br><span class="line"><span class="comment"># 通过__dir__能见到serverHost为实例的一个方法</span></span><br><span class="line">print(a.__dir__())</span><br><span class="line"><span class="comment"># 此时a.__dict__为空</span></span><br><span class="line">print(a.serverHost)</span><br><span class="line"><span class="comment"># 等到调用过a.serverHost后可以发现a.__dict__中多了serverHost</span></span><br><span class="line">print(a.__dict__)</span><br><span class="line"><span class="comment"># 由于实例__dict__会优先于类的__dict__使用，所以直接返回了value值</span></span><br></pre></td></tr></table></figure><p>重点：</p><ol><li>跟[<code>print(t.des)</code>](#3.<code>object.__get__(self, instance, owner)</code>)会触发t.des指向的descriptor实例的<code>__get__</code>一样，通过类<code>__dict__[&quot;serverHost&quot;]</code>，其也是个描述器实例，因此也会触发LazyProperty object的<code>__get__</code></li><li>实例<a href="#%60__dict__%60"><code>__dict__</code></a>会优先于类的<code>__dict__</code>使用，如果实例<code>__dict__</code>找不到，会往上类<code>__dict__</code>找</li></ol><p><strong>修饰符(方法装饰器)</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">lazy_property</span><span class="params">(func)</span>:</span></span><br><span class="line">    <span class="comment"># 创建protected属性</span></span><br><span class="line">    attr_name = <span class="string">"_lazy_"</span> + func.__name__</span><br><span class="line"><span class="meta">    @property</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">_lazy_property</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="comment"># print("done")</span></span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> hasattr(self, attr_name):</span><br><span class="line">            <span class="comment"># print("set")</span></span><br><span class="line">            setattr(self, attr_name, func(self))</span><br><span class="line">        <span class="keyword">return</span> getattr(self, attr_name)</span><br><span class="line">    <span class="keyword">return</span> _lazy_property</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Circle</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, radius)</span>:</span></span><br><span class="line">        self.radius = radius</span><br><span class="line"></span><br><span class="line"><span class="meta">    @lazy_property</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">area</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> <span class="number">3.14</span> * self.radius ** <span class="number">2</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 当解析Circle类、定义area方法的时候，会将Circle.area = @property修饰的_lazy_property函数</span></span><br><span class="line">c = Circle(<span class="number">4</span>)</span><br><span class="line">print(<span class="string">'before calculate area'</span>)</span><br><span class="line">print(c.__dict__)</span><br><span class="line"><span class="comment"># 当调用c.area时，会输出done, 此时会执行_lazy_property内的具体函数, 此次会进行setattr</span></span><br><span class="line">print(c.area)</span><br><span class="line"><span class="comment"># 此次不会调用setattr</span></span><br><span class="line">print(c.area)</span><br><span class="line">print(<span class="string">'after calculate area'</span>)</span><br><span class="line">print(c.__dict__)</span><br><span class="line">c.radius = <span class="number">5</span></span><br></pre></td></tr></table></figure><p>借鉴：<a href="https://www.jianshu.com/p/708dc26f9b92" target="_blank" rel="noopener">Python中的lazy property</a></p><h2 id="由单例元类引发的知识点"><a class="markdownIt-Anchor" href="#由单例元类引发的知识点"></a> 由单例元类引发的知识点</h2><p>看开源代码时，看到了下面一段代码，于是对withMetaclass产生了好奇，经过了解发现其作用是six对python2和python3使用元类兼容的写法。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Python2和3兼容使用元类写法</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span><span class="params">(withMetaclass<span class="params">(Singleton)</span>)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">pass</span></span><br></pre></td></tr></table></figure><p>因此，上述代码在Python3中相当于</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Python3元类使用写法</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span><span class="params">(metaclass=Singleton)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">pass</span></span><br></pre></td></tr></table></figure><p>那么，问题来了，withMetaclass到底是怎么实现兼容的呢？下面是其实现代码</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">withMetaclass</span><span class="params">(meta, *bases)</span>:</span></span><br><span class="line">    <span class="class"><span class="keyword">class</span> <span class="title">MetaClass</span><span class="params">(meta)</span>:</span></span><br><span class="line">        <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, this_bases, d)</span>:</span></span><br><span class="line">            <span class="keyword">return</span> meta(name, bases, d)</span><br><span class="line">    <span class="keyword">return</span> type.__new__(MetaClass, <span class="string">'temporary_class'</span>, (), &#123;&#125;)</span><br></pre></td></tr></table></figure><p>可以看到其中出现了不少我们很少看到的使用方法。接下来我们就仔细的学习上述写法为什么可以成功。</p><p>元类使用可以参考：<a href="https://zhuanlan.zhihu.com/p/49158035" target="_blank" rel="noopener">Python3 元类（metaclass）</a></p><h3 id="预置知识type和object"><a class="markdownIt-Anchor" href="#预置知识type和object"></a> 预置知识：type和object</h3><blockquote><p>object 和 type的关系很像鸡和蛋的关系，先有object还是先有type没法说，obejct和type是共生的关系，必须同时出现的。</p><p>记住一点：<strong>在Python里面，所有的东西都是对象的概念</strong>，即包括类（类是type的实例对象）</p></blockquote><p>最重要的两点</p><ul><li>object类是所有类的超类（也是type类的父类）</li><li>type是所有类的类（类型，所有类都是type的实例对象，object<u>类型</u>也是type的实例对象；type 创建的对象拥有创建对象的能力(也就是类)）–&gt;是所有类的元类</li></ul><p>此外：</p><ul><li>type是所有<strong>元类</strong>的父亲。我们可以通过<strong>继承type来创建元类</strong>（通过重写<code>type.__new__</code>和<code>type.__call__</code>来拦截自定义类的创建过程）。</li><li>object是所有类的父亲。</li><li>实例是对象关系链的末端，不能再被子类化和实例化。</li></ul><p>了解到这些关键的点后，我们继续看代码中出现的一些内容：</p><h3 id="__new__"><a class="markdownIt-Anchor" href="#__new__"></a> <code>__new__</code></h3><blockquote><p><code>__new__()</code> 是一种负责<strong>创建类实例</strong>的<strong>静态方法</strong>，它无需使用 staticmethod 装饰器修饰，且该方法会优先 <code>__init__()</code> 初始化方法被调用。</p></blockquote><p><code>__new__()</code>通常会返回该类的一个实例，但有时也可能会返回其他类的实例，其<code>super().__new__(cls)</code>中会调用<code>object.__init__</code>来<code>Create and return a new object.</code>，因此我们可以通过改写子类的<code>__new__</code>可以添加一些逻辑来控制实例的产生，然后再通过<code>super().__new__(cls)</code>来生成一个instance并返回。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">demoClass</span>:</span></span><br><span class="line">    instances_created = <span class="number">0</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="comment"># __new__(): &lt;class '__main__.demoClass'&gt; ('abc',) &#123;&#125;</span></span><br><span class="line">        print(<span class="string">"__new__():"</span>, cls, args, kwargs)</span><br><span class="line">        <span class="comment"># 1. 通过父类__new__生成一个实例: 调用父类object.__new__生成实例(Create and return a new object.) </span></span><br><span class="line">        instance = super().__new__(cls)</span><br><span class="line">        <span class="comment"># 2. 自己重写要实现的逻辑</span></span><br><span class="line">        instance.number = cls.instances_created</span><br><span class="line">        cls.instances_created += <span class="number">1</span></span><br><span class="line">        <span class="comment"># 3. 将父类生成的实例返回</span></span><br><span class="line">        <span class="keyword">return</span> instance</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, attribute)</span>:</span></span><br><span class="line">        <span class="comment"># __init__(): &lt;__main__.demoClass object at 0x00000185A6466EB0&gt; abc</span></span><br><span class="line">        print(<span class="string">"__init__():"</span>, self, attribute)</span><br><span class="line">        self.attribute = attribute</span><br><span class="line"></span><br><span class="line">test1 = demoClass(<span class="string">"abc"</span>)</span><br><span class="line">test2 = demoClass(<span class="string">"xyz"</span>)</span><br><span class="line">print(test1.number, test1.instances_created)</span><br><span class="line">print(test2.number, test2.instances_created)</span><br></pre></td></tr></table></figure><p>Q:什么情况下重写类的<code>__new__()</code>呢？答案很简单，在<code>__init__()</code>不够用的时候。</p><p><code>__new__()</code>通常会返回该类的一个实例，但有时也可能会返回其他类的实例，如果发生了这种情况，则会跳过对 <code>__init__()</code>方法的调用。而在某些情况下（比如需要修改<strong>不可变类实例</strong>（<a href="http://c.biancheng.net/python/" target="_blank" rel="noopener">Python</a> 的某些内置类型）的创建行为），利用这一点会事半功倍。比如：<a href="http://c.biancheng.net/view/5484.html%EF%BC%8C%E5%AF%B9" target="_blank" rel="noopener">http://c.biancheng.net/view/5484.html，对</a> Python 不可变的内置类型（如 int、str、float 等）进行了子类化，这是因为一旦创建了这样不可变的对象实例，就无法在<code>__init__()</code>方法中对其进行修改。</p><p>注：由于 <code>__new__()</code> 不限于返回同一个类的实例，所以很容易被滥用，不负责任地使用这种方法可能会对代码有害，所以要谨慎使用。</p><h3 id="metaclass元类"><a class="markdownIt-Anchor" href="#metaclass元类"></a> <a href="http://c.biancheng.net/view/2293.html" target="_blank" rel="noopener">MetaClass元类</a></h3><blockquote><p>承接上文<code>__new__</code>，Python中大量使用<code>__new__()</code>方法且合理的地方，就是 MetaClass 元类。</p><p>MetaClass元类，并不是某一个类的名字，它是一个概念，是一种Python的思想。当然其本质也是一个类，但和普通类的用法不同，它可以对<strong>类内部的定义</strong>（包括类属性和类方法）进行<strong>动态的修改</strong>。可以这么说，使用元类的主要目的就是为了实现在创建类时，能够动态地改变类中定义的属性或者方法。其可以将<strong>创建对象的过程拦截</strong>下来，从而对这个对象进行自定义（这个需要<strong>类继承type</strong>，与前文继承object的做区别）。</p><p>明确一点：元类可以理解成是自定义类继承的父类（从兼容写法中也能看出），但元类的特点是<strong>不会出现在自定义类的继承关系(<code>__mro__</code>)之中</strong></p></blockquote><p>举个例子，根据实际场景的需要，我们要为多个类添加一个 name 属性和一个 say() 方法。显然有多种方法可以实现，但其中一种方法就是使用 MetaClass 元类。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 定义一个元类，继承type。因为只有继承type才能通过重写__new__来拦截创建过程</span></span><br><span class="line"><span class="comment"># ▲注意，继承type后__new__能拿到的参数信息跟不继承type的有天壤之别，原因继续看下去</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">FirstMetaClass</span><span class="params">(type)</span>:</span></span><br><span class="line">    <span class="comment"># cls代表元类的类: FirstMetaClass</span></span><br><span class="line">    <span class="comment"># name代表自定义类的类名: CLanguage</span></span><br><span class="line">    <span class="comment"># bases代表被动态修改的类的所有父类</span></span><br><span class="line">    <span class="comment"># attr代表被动态修改的类的所有属性、方法组成的字典</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, bases, attrs)</span>:</span></span><br><span class="line">        <span class="comment"># 动态为该类添加一个name属性</span></span><br><span class="line">        attrs[<span class="string">'name'</span>] = <span class="string">"C语言中文网"</span></span><br><span class="line">        attrs[<span class="string">'say'</span>] = <span class="keyword">lambda</span> self: print(<span class="string">"调用 say() 实例方法"</span>)</span><br><span class="line">        <span class="keyword">return</span> super().__new__(cls,name,bases,attrs)</span><br><span class="line">        </span><br><span class="line"><span class="comment"># 定义类时，指定元类</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CLanguage</span><span class="params">(object, metaclass=FirstMetaClass)</span>:</span></span><br><span class="line">    <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line">clangs = CLanguage()</span><br><span class="line">print(clangs.name)</span><br><span class="line">clangs.say()</span><br></pre></td></tr></table></figure><p>可以看到，在创建类时，通过在标注父类的同时指定元类（格式为<code>metaclass=元类名</code>），则当 <a href="http://c.biancheng.net/python/" target="_blank" rel="noopener">Python</a> 解释器在创建该类实例时，<code>FirstMetaClass(type)</code>元类中的<code>__new__</code>方法就会被调用，其中bases和attrs能拿到自定义类的参数，从而实现动态修改类属性或者类方法的目的。</p><p><strong>元类和父类的区别：</strong></p><p>在定义子类的时候，我们有两个选择：①是传需要继承的父类；②自定义的元类。</p><ul><li>父类是子类的模板，子类的功能是跟父类紧耦合的，子类和父类一般是一一对应的</li><li>元类是子类的修饰器，可以为该子类和其他子类都添加自定义功能，并且不在继承关系中(<code>Class.__mro__</code>查看)，子类和元类是一对多的关系。元类并不是特地为某个子类服务的</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TestMeta3</span><span class="params">(type)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, bases, attrs)</span>:</span></span><br><span class="line">        print(cls)    <span class="comment"># 当前类 </span></span><br><span class="line">        print(<span class="string">"name"</span>, name)<span class="comment"># 如果是通过metaclass触发的，此处为调用metaclass的类的类型</span></span><br><span class="line">        print(<span class="string">"bases"</span>, bases)<span class="comment"># 如果是通过metaclass触发的，此处为调用metaclass的类的父类</span></span><br><span class="line">        print(<span class="string">"attrs"</span>, attrs)<span class="comment"># 如果是通过metaclass触发的，此处为调用metaclass的类的属性</span></span><br><span class="line">        <span class="keyword">return</span> type.__new__(cls, name, bases, attrs)</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Pa3</span>:</span></span><br><span class="line">    <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># python3中可以直接通过metaclass关键字参数来指定类的元类</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Eg3</span><span class="params">(Pa3, metaclass=TestMeta3)</span>:</span></span><br><span class="line"><span class="meta">    @classmethod</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">get</span><span class="params">(self)</span>:</span></span><br><span class="line">        kkk = []</span><br><span class="line">        kkk.append(self.__skiless__)</span><br><span class="line">        <span class="keyword">return</span> kkk</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">acc2</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> <span class="string">'a2'</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">输出</span></span><br><span class="line"><span class="string">&lt;class '__main__.TestMeta3'&gt;</span></span><br><span class="line"><span class="string">name Eg3</span></span><br><span class="line"><span class="string">bases (&lt;class '__main__.Pa3'&gt;,)</span></span><br><span class="line"><span class="string">attrs &#123;'__module__': '__main__', '__qualname__': 'Eg3', 'get': &lt;classmethod object at 0x00000263511C6FA0&gt;, 'acc2': &lt;function Eg3.acc2 at 0x00000263511C5310&gt;&#125;</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure><p>在定义的时候，发现竟然有输出。因为定义的时候，python解释器会在当前类中查找metaclass[3]，如果找到了，就<strong>使用该metaclass创建Eg3类</strong>。所以打印出来的name、bases、attrs都和Eg3有关。</p><p><strong>with_metaclass</strong></p><blockquote><p>由于python2和python3中元类使用方法的不同，我们需要使用一种兼容的方式[1]，如下所示：</p></blockquote><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">withMetaclass</span><span class="params">(meta, *bases)</span>:</span></span><br><span class="line">    <span class="string">"""Create a base class with a metaclass."""</span></span><br><span class="line">    <span class="class"><span class="keyword">class</span> <span class="title">MetaClass</span><span class="params">(meta)</span>:</span></span><br><span class="line"><span class="comment"># 如果删除__new__，则类.__mro__中能看到meta类</span></span><br><span class="line">        <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, this_bases, d)</span>:</span></span><br><span class="line">            <span class="comment"># 因为meta是类，所以这边是在调用meta的__call__()。▲bases，d为ConfigHandler的父类和属性</span></span><br><span class="line">            <span class="keyword">return</span> meta(name, bases, d)</span><br><span class="line"><span class="comment"># 返回一个新类型, type.__new__()要求第一个必须是type的子类</span></span><br><span class="line">    <span class="keyword">return</span> type.__new__(MetaClass, <span class="string">'temporary_class'</span>, (), &#123;&#125;)</span><br><span class="line"><span class="comment"># 下面两句话等价，withMetaclass是为了兼容python2和python3,2中没有metaclass关键字</span></span><br><span class="line"><span class="comment"># 其在创建实例时，__new__方法会被MetaClass拦截(其实就是子类没定义__new__，走了父类的__new__)</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span><span class="params">(withMetaclass<span class="params">(Singleton)</span>)</span></span></span><br><span class="line"><span class="class"># <span class="title">python3</span>写法</span></span><br><span class="line"><span class="class"><span class="title">class</span> <span class="title">ConfigHandler</span><span class="params">(metaclass=Singleton)</span></span></span><br><span class="line"><span class="class"></span></span><br><span class="line"># 因为ConfigHandler相当于继承了 type.__new__返回的类MetaClass，所以在c = ConfigHandler()实例化的时候，会触发Metaclass的__new__然后调用meta.__call__从而返回一个对象</span><br></pre></td></tr></table></figure><p><code>with_metaclass</code>返回的临时类中，本身无任何属性，但包含了元类和基类的所有信息，并在<strong>下一步定义类时</strong>将所有信息解包出来[1]。</p><p>见：<a href="https://zhuanlan.zhihu.com/p/354828950" target="_blank" rel="noopener">★Python 元类及with_metaclass</a></p><h3 id="type"><a class="markdownIt-Anchor" href="#type"></a> <a href="http://c.biancheng.net/view/2292.html" target="_blank" rel="noopener"><code>type</code>动态创建类</a></h3><ul><li>type() 函数属于 <a href="http://c.biancheng.net/python/" target="_blank" rel="noopener">Python</a> 内置函数，通常用来查看某个变量的具体类型。<ul><li><code>type(obj)</code></li></ul></li><li>其实，type() 函数还有一个更高级的用法，即<strong>创建</strong>一个自定义<strong>类型</strong>（也就是创建一个类）。<ul><li><code>type(name, bases, dict)</code>：其中 name 表示类的名称；bases 表示一个元组，其中存储的是该类的父类；dict 表示一个字典，用于表示类内定义的属性或者方法。</li></ul></li></ul><p>实际上<code>type(name, bases, dict)</code>是调用了type类的<code>type.__init__(cls, what, bases=None, dict=None)</code>方法，创建了一个type的实例（类类型就是一个type实例），类型是<code>&lt;class 'type'&gt;</code></p><p><strong>&lt;class ‘type’&gt;是所有类型的类型。&lt;class ‘object’&gt;也是所有对象的超类（除了它自己，包括type）</strong></p><p>▲. 此外type还有<code>type.__new__(*args, **kwargs)</code>，其作用是<code>Create and return a new object.</code>，可以写成<code>type.__new__(ClassTpye, name, base, dicts)</code>，但ClassType必须是type的子类。会返回一个跟ClassType有关系的新类型</p><h3 id="通过元类创建单例类"><a class="markdownIt-Anchor" href="#通过元类创建单例类"></a> 通过元类创建单例类</h3><blockquote><p>现在让我们正式看，我在开源代码里看到的内容：</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 注意这边继承了type, 所以下面的__call__是重写type的__call__，即创建实例的方法</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Singleton</span><span class="params">(type)</span>:</span></span><br><span class="line">    <span class="string">"""</span></span><br><span class="line"><span class="string">    Singleton Metaclass</span></span><br><span class="line"><span class="string">    """</span></span><br><span class="line">    _inst = &#123;&#125;</span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__call__</span><span class="params">(cls, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> cls <span class="keyword">not</span> <span class="keyword">in</span> cls._inst:</span><br><span class="line">            cls._inst[cls] = super(Singleton, cls).__call__(*args, **kwargs)</span><br><span class="line">        <span class="keyword">return</span> cls._inst[cls]</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">withMetaclass</span><span class="params">(meta, *bases)</span>:</span></span><br><span class="line">    <span class="string">"""Create a base class with a metaclass."""</span></span><br><span class="line">    <span class="comment"># 这需要一点解释：基本思想是为一个级别的类实例化创建一个虚拟元类，用实际的元类替换它自己。</span></span><br><span class="line">    </span><br><span class="line">    <span class="comment"># KeyPoint1. 继承meta类</span></span><br><span class="line">    <span class="class"><span class="keyword">class</span> <span class="title">MetaClass</span><span class="params">(meta)</span>:</span></span><br><span class="line">        <span class="comment"># 实际上下面的__new__不影响</span></span><br><span class="line">        <span class="function"><span class="keyword">def</span> <span class="title">__new__</span><span class="params">(cls, name, this_bases, d)</span>:</span></span><br><span class="line">            <span class="comment"># cls为withMetaclass; name为使用者的类型; this_bases为使用者的父类们; d为使用者的属性</span></span><br><span class="line">            <span class="keyword">return</span> meta(name, bases, d)</span><br><span class="line">    <span class="comment"># KeyPoint2. type.__new__创建一个名称叫temporary_class，类型为MetaClass的类</span></span><br><span class="line">    <span class="comment"># ▲注意type.__new__中的类类型必须是type的子类</span></span><br><span class="line">    <span class="keyword">return</span> type.__new__(MetaClass, <span class="string">'temporary_class'</span>, (), &#123;&#125;)</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ConfigHandler</span><span class="params">(withMetaclass<span class="params">(Singleton)</span>)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        print(<span class="string">"__init__"</span>)</span><br><span class="line">        <span class="keyword">pass</span></span><br><span class="line"><span class="meta">    @LazyProperty</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">serverHost</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">return</span> os.environ.get(<span class="string">"HOST"</span>, setting.HOST)</span><br><span class="line"><span class="comment"># res=withMetaclass(Singleton)的类型为&lt;class 'util.six.withMetaclass.&lt;locals&gt;.MetaClass'&gt;</span></span><br><span class="line"><span class="comment"># 将其传给ConfigHandler作为父类，在定义 ConfigHandler 时会触发MetaClass.__new__, 于是调用meta(name, bases, d)，此处的meta为Singleton，而name为ConfigHandler类, bases为空, d为ConfigHandler的属性和方法。</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># print(type(ConfigHandler)) ==&gt;  &lt;class 'util.singleton.Singleton'&gt;</span></span><br><span class="line"><span class="comment"># 因此 c = ConfigHandler() ==&gt; Singleton的__call__方法，</span></span><br><span class="line">当c = ConfigHandler()时会因为type.__new__(MetaClass, <span class="string">'temporary_class'</span>, (), &#123;&#125;)去找MetaClass的__call__进行调用，MetaClass没有__call__则找到了其父类meta(Singleton)的__call__</span><br></pre></td></tr></table></figure><p>注：类也是对象，是元类的对象，即我们实例化一个类时，调用其元类的<code>__call__(cls, *args, **kwargs)</code>方法进行创建对象。</p><h2 id="__call__"><a class="markdownIt-Anchor" href="#__call__"></a> <code>__call__</code></h2><blockquote><p>一个非常特殊的实例方法，即<code>__call__()</code>。该方法的功能是在类中<strong>重载了对象的 () 运算符</strong>，使得类实例对象可以像调用普通函数那样，以“<strong>对象名()</strong>”的形式使用。</p></blockquote><p>实际上，如果不重写<code>__call__</code>的话，<code>Class.__call__(*args, **kwargs)</code>还承担着产生类实例的功能（会调用父类（可以通过<code>Class.__class__</code>来查看父类）的<code>type.__call__</code>其会返回一个实例）</p><p>案例一:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 默认继承的是object, 而不是type</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Meta</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name)</span>:</span></span><br><span class="line">        print(<span class="string">"init"</span>)</span><br><span class="line">        self.name = name</span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__call__</span><span class="params">(self, *args, **kwargs)</span>:</span></span><br><span class="line">        print(<span class="string">"call"</span>)</span><br><span class="line"><span class="comment"># 当没有重写__call__时，无论是显式的调用__call__，还是通过()运算符调用，都会调用type.__call__返回一个实例</span></span><br><span class="line">res = Meta.__call__(<span class="string">"asd"</span>)</span><br><span class="line">print(res, type(res))</span><br><span class="line">res = Meta(<span class="string">"asd"</span>)</span><br><span class="line">print(res, type(res))</span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">init</span></span><br><span class="line"><span class="string">&lt;__main__.Meta object at 0x0000014FB5115EE0&gt; &lt;class '__main__.Meta'&gt;</span></span><br><span class="line"><span class="string">init</span></span><br><span class="line"><span class="string">&lt;__main__.Meta object at 0x0000014FB5115A90&gt; &lt;class '__main__.Meta'&gt;</span></span><br><span class="line"><span class="string">上述两种都能创建对象</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="comment"># 当重写__call__以后, __call__()返回实例的效果就失效了==&gt;因为上述代码没有return</span></span><br><span class="line"><span class="comment"># 此时 Meta()与Meta.__call__()不再等价</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">call</span></span><br><span class="line"><span class="string">None &lt;class 'NoneType'&gt;</span></span><br><span class="line"><span class="string">init</span></span><br><span class="line"><span class="string">&lt;__main__.Meta object at 0x0000016CC2745EE0&gt; &lt;class '__main__.Meta'&gt;</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure><p>Q:我们在实例化一个对象的时候<code>f = Foo(1, y=2)</code>，可以发现在<code>__init__()</code>中并没有返回实例，但调用<code>Foo(1, y=2)</code><strong>确实</strong>返回了一个对象，而且，<code>__init__</code>预期一个<code>self</code>参数，但是当我们调用<code>Foo(1, y=2)</code>时这里并没有这个参数。那么类实例化的过程到底是怎么样的呢？</p><p>A:构造顺序——<a href="https://www.jianshu.com/p/f63ad9d550f1" target="_blank" rel="noopener">理解python的类实例化</a></p><p>首先明确一点，Python中的<strong>类也是对象</strong>！类、函数、方法以及实例都是对象——<strong>类类型是type的对象</strong>，并且无论何时你将一对括号放在它们的名字后面时，就会调用<code>type.__call__()</code>方法。为什么呢？因为type是类型的父类</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>Foo.__class__</span><br><span class="line">&lt;<span class="class"><span class="keyword">class</span> '<span class="title">type</span>'&gt;</span></span><br></pre></td></tr></table></figure><p>所以<code>Foo</code>是类型<code>type</code>的一个对象，并且调用type类的<code>__call__(self, *args, **kwargs)</code>返回一个<code>Foo</code>类的对象。让我们看下<code>type</code>中的<code>__call__</code>方法是什么样的。这个方法相当的复杂，但是我们将其C代码转成Python代码，并尝试尽量简化它，结果如下。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">type</span><span class="params">(object)</span>:</span></span><br><span class="line"><span class="comment"># 这边的obj_type跟cls一样</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__call__</span><span class="params">(obj_type, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="comment"># 通过__new__创建一个空的类实例，如果obj_type没有__new__则使用type.__new__</span></span><br><span class="line">        obj = obj_type.__new__(*args, **kwargs)</span><br><span class="line">        <span class="comment"># 进行类型检查</span></span><br><span class="line">        <span class="keyword">if</span> obj <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">None</span> <span class="keyword">and</span> issubclass(obj, obj_type):</span><br><span class="line">            <span class="comment"># 对类进行__init__初始化</span></span><br><span class="line">            obj.__init__(*args, **kwargs)</span><br><span class="line">        <span class="comment"># 返回类实例</span></span><br><span class="line">        <span class="keyword">return</span> obj</span><br></pre></td></tr></table></figure><p>可见<code>__new__</code>方法为对象分配了内存空间，构建它为一个“空&quot;对象然后<code>__init__</code>方法被调用来初始化它。</p><p>那我们定义了一个具体类来讲解这个过程。首先明确一点：<strong>Foo相对于产生了一个type实例化对象</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Foo</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, x, y=<span class="number">0</span>)</span>:</span></span><br><span class="line">        self.x = x</span><br><span class="line">        self.y = y</span><br></pre></td></tr></table></figure><p>获得实例化对象**<code>Foo(*args, **kwargs)</code><strong>也可以看作是<code>type对象()</code>即</strong>调用了type中()运算符的触发的函数<code>type.__call__</code>从而创建一个Foo的实例**</p><ul><li>至于<code>type.__call__</code>发生了什么就是上面抽象代码中介绍的那般，调用<code>type.__new__(Foo, *args, **kwargs)</code>然后返回一个对象实例obj。</li><li><code>obj</code>随后通过调用<code>obj.__init__(*args, **kwargs)</code>被初始化。</li><li><code>obj</code>被<code>type.__call__</code>中返回。</li></ul><p>▲注意：<code>Foo.__call__</code>重载的是<code>foo对象</code>的()运算符，而<code>Foo()</code>实例化foo对象，则执行的是<code>type对象</code>的()运算符。</p><p><strong>小总结</strong>：</p><ul><li>现在我们能知道为什么元类必须继承type了：因为我们实例化对象<code>Foo(xxx)</code>时调用了<code>type.__call__</code>,而<code>type.__call__</code>又会调用<code>type.__new__</code>因此如果type子类重写实现了<code>__new__</code>（返回的类实例对象的类型作控制）、<code>__call__</code>（对实例化的流程做控制），则可以对<strong>类对象的类型和类属性</strong>起到自定义的功能，而重写就必须继承type=&gt;需要元类必须继承type</li><li>所以按照上述的逻辑，如果定义了一个元类让自定义类用的话<code>class Foo(metaclass=MyMetaClass)</code>，在其实例化过程中<code>Foo()</code>会直接调用重写后的<code>MyMetaClass.__call__</code>，而只要记住在<code>MyMetaClass.__call__</code>中使用到<code>return super(Singleton, cls).__call__(*args, **kwargs)</code>就可以把<code>type.__call__</code>生成的实例返回啦。所以这也是为什么<u>编写元类，一般都是继承了type，然后根据想控制实例化流程就重写<code>__call__</code>方法，想添加属性就重写<code>__new__</code>方法就行了。</u></li><li><strong>★元类产生影响的时间点是在实例化的时候</strong></li></ul><p>注意点：元类继承了type，所以实例化元类是在产生一个类类型，就要以type创建类类型的参数去产生。而元类的使用一般都是自定义类<code>class MyClass(metaclass=元类)</code>，然后实例化自定义类<code>MyClass(xxx)</code></p><p><strong>总结</strong>：看完上述知识点后，我们能知道为什么withclass能起到metaclass的作用(类的<code>__mro__</code>中不出现指定的元类)了：</p><ul><li>首先分析流程：<code>return type.__new__(Metaclass)</code>返回了一个类型供自定义类继承，由于MetaClass继承的是真正的元类(元类都继承type)，所以在自定义类实例化的时候会被Metaclass的<code>__new__</code>方法拦截，在<code>MetaClass.__new__</code>里return了一个自定义实例，并把对象加入到了Singleton字典中了。</li><li>其次讲解为什么MetaClass中没有MetaClass：因为根据<a href="#"><code>__new__</code></a>知识点中讲到的，<code>__new__</code>控制了实例产生，<code>return type.__new__(Metaclass)</code>中创建了<code>Metaclass</code>，但其在<code>__new__</code>中返回的并不是MetaClass，因此<code>__mro__</code>中不会出现<code>Metaclass</code></li><li>最后还要讲讲Singleton中的执行逻辑：</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Singleton</span><span class="params">(type)</span>:</span></span><br><span class="line">    _inst = &#123;&#125;</span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__call__</span><span class="params">(cls, *args, **kwargs)</span>:</span></span><br><span class="line">        <span class="keyword">if</span> cls <span class="keyword">not</span> <span class="keyword">in</span> cls._inst:</span><br><span class="line">            <span class="comment"># super(Singleton, cls).__call__调用的是type.__call__(自定义类类名name, 自定义类父类bases, 自定义类属性attrs)</span></span><br><span class="line">            cls._inst[cls] = super(Singleton, cls).__call__(*args, **kwargs)</span><br><span class="line">        <span class="keyword">return</span> cls._inst[cls]</span><br></pre></td></tr></table></figure><h2 id="函数和模块的特殊属性"><a class="markdownIt-Anchor" href="#函数和模块的特殊属性"></a> <a href="https://cloud.tencent.com/developer/article/1607611" target="_blank" rel="noopener">函数和模块的特殊属性<code>__annotations__</code></a></h2><blockquote><p>众所周知，Python是一种动态类型语言，也是强类型语言。在Python语言中，使用变量之前不需要声明其类型，直接赋值即可创建变量，变量初始类型取决于等号右侧表达式的值的类型。创建之后，变量的类型可以随时发生变化，<u>但在任何时刻，每个变量都有确定的类型</u>。</p></blockquote><p>同理，在定义函数和类的方法时，也不需要声明形参类型，完全取决于实参类型.</p><p>因此很多从其他语言转过来的朋友很不习惯这样的方式，还是习惯于声明变量和参数的类型。虽然Python不支持声明，但是允许在定义函数时使用“注解”的形式来标注形参和返回值的类型，<strong>但这种注解的形式在运行时并不会对形参进行任何约束和检查，在实际调用函数时，即使实参不符合形参的类型标注，一样能够正常传递</strong>（IDE会进行检查，在编译之前提醒传参不一致，但也不会报错）</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 标注了形参s的类型为str或者dict; 也标注了返回值为str类型</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">extract_value</span><span class="params">(s: Union[str, dict])</span> -&gt; str:</span></span><br><span class="line"><span class="keyword">pass</span></span><br></pre></td></tr></table></figure><p>在Python中，函数会维护一个特殊属性<code>__annotations__</code>，这是一个字典，其中的“键”是被注解的形参名，“值”为注解的内容。使用时并不要求注解的内容是Python中的类型，可以是任意内容。例如，</p><p>从官方文档来看，函数的__annotations__属性只包含形参和返回值的注解，即使在函数体中有类似的注解，但这并不等价于C语言中的变量声明，①这样的注解不会创建变量，②也不会被收集到这个特殊属性<code>__annotations__</code>中。例如</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">demo</span><span class="params">(param1: int, param2: int)</span>-&gt;int:</span></span><br><span class="line"><span class="keyword">return</span> param1 * param2</span><br><span class="line">print(demo.__annotations__)</span><br><span class="line"><span class="comment"># &#123;'param1': &lt;class 'int'&gt;, 'param2': &lt;class 'int'&gt;, 'return': &lt;class 'int'&gt;&#125;</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">demo</span><span class="params">(a: <span class="number">3</span>, b: int)</span> -&gt; int:</span></span><br><span class="line">    c: <span class="number">5</span></span><br><span class="line">    d: <span class="number">8</span></span><br><span class="line">    <span class="keyword">return</span> a * b * c * d</span><br><span class="line"></span><br><span class="line">print(demo.__annotations__)</span><br><span class="line"><span class="comment"># &#123;'a': 3, 'b': &lt;class 'int'&gt;, 'return': &lt;class 'int'&gt;&#125;</span></span><br><span class="line">demo(<span class="number">3</span>, <span class="number">8</span>)</span><br><span class="line"><span class="comment"># Traceback (most recent call last):</span></span><br><span class="line"><span class="comment">#    return a * b * c * d</span></span><br><span class="line"><span class="comment"># UnboundLocalError: local variable 'c' referenced before assignment</span></span><br></pre></td></tr></table></figure><p>另外，在<u>模块</u>中也有个特殊属性<code>__annotations__</code>用于收集模块中变量的注解，但这些注解同样也不会创建对应的变量。例如，在下面的代码中，并没有创建变量e、f、g。</p><h3 id="用途修改dataclass构造"><a class="markdownIt-Anchor" href="#用途修改dataclass构造"></a> 用途：修改dataclass构造</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> dataclasses <span class="keyword">import</span> dataclass, field</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@dataclass</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Goods</span>:</span></span><br><span class="line">    <span class="comment"># 商品上架时间</span></span><br><span class="line">    addTime: str</span><br><span class="line">    <span class="comment"># 商品上架创建者</span></span><br><span class="line">    addUserId: str</span><br><span class="line">    <span class="comment"># 商品更新操作者</span></span><br><span class="line">    updateUserId: str</span><br><span class="line">    <span class="comment"># 商品id</span></span><br><span class="line">    goodsId: str</span><br><span class="line">    <span class="comment"># 商品详情描述</span></span><br><span class="line">    content: str</span><br><span class="line">    <span class="comment"># 商品名称</span></span><br><span class="line">    goodsName: str</span><br><span class="line">    <span class="comment"># 尺寸</span></span><br><span class="line">    specification: str</span><br><span class="line">    <span class="comment"># 作者</span></span><br><span class="line">    author: str</span><br><span class="line">    <span class="comment"># 展示图片链接</span></span><br><span class="line">    imgUrl: str</span><br><span class="line">    <span class="comment"># 钻石价格</span></span><br><span class="line">    price: int</span><br><span class="line">    <span class="comment"># 现金价格</span></span><br><span class="line">    cashPrice: int</span><br><span class="line">    <span class="comment"># 绑定的对应订单ID</span></span><br><span class="line">    bindingOrderId: int</span><br><span class="line">    <span class="comment"># 拍卖者用户id</span></span><br><span class="line">    userId: int</span><br><span class="line">    <span class="comment">#</span></span><br><span class="line">    alrHandNum: int</span><br><span class="line">    <span class="comment">#</span></span><br><span class="line">    deliveryMoney: float</span><br><span class="line">    <span class="comment">#</span></span><br><span class="line">    silver: int</span><br><span class="line">    <span class="comment">#</span></span><br><span class="line">    startNum: int</span><br><span class="line">    <span class="comment">#</span></span><br><span class="line">    endNum: int</span><br><span class="line"></span><br><span class="line"><span class="meta">    @classmethod</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">from_kwargs</span><span class="params">(cls, **kwargs)</span>:</span></span><br><span class="line">        <span class="comment"># split the kwargs into native ones and new ones</span></span><br><span class="line">        native_args, new_args = &#123;&#125;, &#123;&#125;</span><br><span class="line">        <span class="keyword">for</span> name, val <span class="keyword">in</span> kwargs.items():</span><br><span class="line">            <span class="comment"># __annotations__是上述标注的属性</span></span><br><span class="line">            <span class="keyword">if</span> name <span class="keyword">in</span> cls.__annotations__:</span><br><span class="line">                native_args[name] = val</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                new_args[name] = val</span><br><span class="line">        <span class="comment"># use the native ones to create the class ...</span></span><br><span class="line">        ret = cls(**native_args)</span><br><span class="line">        <span class="comment"># ... and add the new ones by hand</span></span><br><span class="line">        <span class="keyword">for</span> new_name, new_val <span class="keyword">in</span> new_args.items():</span><br><span class="line">            <span class="comment"># 这些额外的属性会被存放在self.__dict__中， 但是dataclass只会根据标注的属性来定义__repr__, 因此这些new_agrs参数不会在__repr__中出现</span></span><br><span class="line">            setattr(ret, new_name, new_val)</span><br><span class="line">        <span class="keyword">return</span> ret</span><br></pre></td></tr></table></figure><h2 id="dataclass数据类"><a class="markdownIt-Anchor" href="#dataclass数据类"></a> Dataclass(数据类)</h2><blockquote><p>python3.7的新特性dataclass，其之前类似的实现是<code>@attr.s</code>，主要用来描述数据对象，其含义为“一个带有默认值的可变的namedtuple”，广义的定义就是有一个类，它的属性均可公开访问，可以带有默认值并能被修改，而且类中含有与这些属性相关的类方法，那么这个类就可以称为dataclass，再通俗点讲，dataclass就是一个含有数据及操作数据方法的容器。</p><p>乍一看可能会觉得这个概念不就是普通的class么，然而还是有几处不同：</p><ol><li>相比普通class，dataclass通常不包含私有属性，数据可以直接访问</li><li>dataclass的repr方法通常有固定格式，会打印出类型名以及属性名和它的值</li><li>dataclass拥有<code>__eq__</code>和<code>__hash__</code>魔法方法</li><li>dataclass有着模式单一固定的构造方式，或是需要重载运算符，而普通class通常无需这些工作</li></ol><p>因为数据对象类有些固定的行为，也正适合程序为我们自动生成，因此<code>dataclasses</code>模块诞生了。</p></blockquote><p><code>dataclasses</code>模块中提供了一些常用函数供我们处理数据类。</p><ul><li>使用<code>dataclasses.asdict</code>和<code>dataclasses.astuple</code>我们可以把<u>数据类实例</u>中的数据转换成字典或者元组</li><li>使用<code>dataclasses.is_dataclass</code>可以判断一个类or实例对象是否是<u>数据类</u></li></ul><h3 id="数据类的基石dataclassesfield"><a class="markdownIt-Anchor" href="#数据类的基石dataclassesfield"></a> 数据类的基石——dataclasses.field</h3><p>函数原型: <code>dataclasses.field(*, default=MISSING, default_factory=MISSING, repr=True, hash=None, init=True, compare=True, metadata=None)</code>， 通常我们无需直接使用，装饰器会根据我们给出的类型注解自动生成field，但有时候我们也需要定制这一过程，这时<code>dataclasses.field</code>就显得格外有用了。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@dataclass</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">C</span>:</span></span><br><span class="line">    <span class="comment"># 实际上等于 age: int = field()</span></span><br><span class="line">    age: int</span><br><span class="line">       </span><br><span class="line">    mylist: List[int] = field(default_factory=list)</span><br></pre></td></tr></table></figure><ul><li>default和default_factory参数将会影响默认值的产生，它们的默认值都是None，意思是调用时如果为指定则产生一个为None的值。<ul><li>其中default是field的默认值，比如<code>age: int = field(default=2)</code>，那么<code>c=C()</code>得到的c实例中的age就为2；（等价于<code>age: int = 2</code>）</li><li>default_factory控制如何产生值，它接收一个无参数或者全是默认参数的<code>callable</code>对象，然后用调用这个对象获得field的初始值，之后再将default（如果值不是MISSING）复制给<code>callable</code>返回的这个对象。</li></ul></li></ul><p>▲举个例子，对于list，当复制它时只是<strong>复制了一份引用</strong>，所以像dataclass里那样直接复制给实例的做法的<strong>危险而错误</strong>的，为了保证使用list时的安全性，应该这样做：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@dataclass</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">C</span>:</span></span><br><span class="line">    mylist: List[int] = field(default_factory=list)</span><br></pre></td></tr></table></figure><ul><li>**<code>field.init</code>**参数如果设置为False，表示不为这个field生成初始化操作，dataclass提供了hook——<code>__post_init__</code>供我们利用这一特性：<code>__post_init__</code>在<code>__init__</code>后被调用，我们可以在这里初始化那些需要前置条件的field。</li><li><strong><code>dataclasses.InitVar</code></strong>: 如果指定一个field的类型注解为<code>dataclasses.InitVar</code>，那么这个field将只会在初始化过程中（<code>__init__</code>和<code>__post_init__</code>）可以被使用，当初始化完成后访问该field会返回一个<code>dataclasses.Field</code>对象而不是field原本的值，也就是该field不再是一个可访问的数据对象。</li></ul><p>参考：<a href="https://www.cnblogs.com/apocelipes/p/10284346.html%E3%80%81https://docs.python.org/zh-cn/3/library/dataclasses.html" target="_blank" rel="noopener">https://www.cnblogs.com/apocelipes/p/10284346.html、https://docs.python.org/zh-cn/3/library/dataclasses.html</a></p><h4 id="dataclasses嵌套"><a class="markdownIt-Anchor" href="#dataclasses嵌套"></a> dataclasses嵌套</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">nested_dataclass</span><span class="params">(*args, **kwargs)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">wrapper</span><span class="params">(cls)</span>:</span></span><br><span class="line">        cls = dataclass(cls, **kwargs)</span><br><span class="line">        original_init = cls.__init__</span><br><span class="line"></span><br><span class="line">        <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, *args, **kwargs)</span>:</span></span><br><span class="line">            <span class="keyword">for</span> name, value <span class="keyword">in</span> kwargs.items():</span><br><span class="line">                field_type = cls.__annotations__.get(name, <span class="keyword">None</span>)</span><br><span class="line">                <span class="keyword">if</span> is_dataclass(field_type) <span class="keyword">and</span> isinstance(value, dict):</span><br><span class="line">                    new_obj = field_type(**value)</span><br><span class="line">                    kwargs[name] = new_obj</span><br><span class="line">                <span class="keyword">if</span> isinstance(value, list) <span class="keyword">and</span> is_dataclass(field_type[<span class="number">0</span>]):</span><br><span class="line">                    res = []</span><br><span class="line">                    <span class="keyword">for</span> c <span class="keyword">in</span> value:</span><br><span class="line">                        new_obj = field_type[<span class="number">0</span>](**c)</span><br><span class="line">                        res.append(new_obj)</span><br><span class="line">                    kwargs[name] = res</span><br><span class="line">            original_init(self, *args, **kwargs)</span><br><span class="line"></span><br><span class="line">        cls.__init__ = __init__</span><br><span class="line">        <span class="keyword">return</span> cls</span><br><span class="line"></span><br><span class="line">    <span class="keyword">return</span> wrapper(args[<span class="number">0</span>]) <span class="keyword">if</span> args <span class="keyword">else</span> wrapper</span><br></pre></td></tr></table></figure><p><code>**</code>解包得到的额外信息</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 额外信息</span></span><br><span class="line"><span class="meta">@classmethod</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">from_kwargs</span><span class="params">(cls, **kwargs)</span>:</span></span><br><span class="line">    <span class="comment"># split the kwargs into native ones and new ones</span></span><br><span class="line">    native_args, new_args = &#123;&#125;, &#123;&#125;</span><br><span class="line">    <span class="keyword">for</span> name, val <span class="keyword">in</span> kwargs.items():</span><br><span class="line">        <span class="keyword">if</span> name <span class="keyword">in</span> cls.__annotations__:</span><br><span class="line">            native_args[name] = val</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            new_args[name] = val</span><br><span class="line"></span><br><span class="line">    <span class="comment"># use the native ones to create the class ...</span></span><br><span class="line">    ret = cls(**native_args)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># ... and add the new ones by hand</span></span><br><span class="line">    <span class="keyword">for</span> new_name, new_val <span class="keyword">in</span> new_args.items():</span><br><span class="line">        setattr(ret, new_name, new_val)</span><br><span class="line">    <span class="keyword">return</span> ret</span><br></pre></td></tr></table></figure><h2 id="生成器-迭代器-容器"><a class="markdownIt-Anchor" href="#生成器-迭代器-容器"></a> 生成器、迭代器、容器</h2><p>在使用Python的过程中，经常会和列表/元组/字典（list/tuple/dict）、容器（container）、可迭代对象（iterable）、迭代器（iterator）、生成器（generator）等这些名词打交道，众多的概念掺杂到一起难免会让人一头雾水，这里我们用一张图来展现它们之间的关系——from： <a href="https://zhuanlan.zhihu.com/p/341439647" target="_blank" rel="noopener">Python迭代器和生成器详解</a></p><p><img src="/2022/03/27/Python进阶/E:%5Chexo%5Csource_posts%5CPython%E8%BF%9B%E9%98%B6%5C%E8%BF%AD%E4%BB%A3%E5%99%A8-%E7%94%9F%E6%88%90%E5%99%A8.png" alt="迭代器-生成器"></p><h3 id="python中的容器"><a class="markdownIt-Anchor" href="#python中的容器"></a> Python中的容器</h3><p>容器是一种把多个元素组织在一起的数据结构，容器中的元素可以逐个迭代获取，可以用 in，not in 关键字判断元素是否包含在容器中。</p><p>我们常用的 string、set、list、tuple、dict 都属于容器对象。</p><p>尽管大多数容器都提供了某种方式获取其中的每一个元素，但这并不是容器本身提供的能力，而是可迭代对象赋予了容器这种能力，当然并不是所有容器都是可迭代的。</p><h3 id="可迭代对象与iterator迭代器"><a class="markdownIt-Anchor" href="#可迭代对象与iterator迭代器"></a> 可迭代对象与Iterator（迭代器）</h3><p><strong>可迭代对象</strong></p><p>可以返回一个迭代器的对象都可以称之为可迭代对象。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>x = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>a = iter(x)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>b = iter(x)</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>next(a)</span><br><span class="line"><span class="number">1</span></span><br></pre></td></tr></table></figure><p>a 和 b 是两个独立的迭代器，迭代器内部有一个状态，该状态用于记录当前迭代所在的位置，以方便下次迭代时获取正确的元素。迭代器有一种具体的迭代器类型，比如 <code>list_iterator</code>，<code>set_iterator</code>。可迭代对象实现了 <code>__iter__()</code> 方法，该方法返回一个迭代器对象。</p><p>在<strong>循环遍历</strong>自定义容器对象时，会使用python内置函数iter()调用遍历对象的**<code>__iter__()</code>获得一个迭代器**，之后再循环对这个迭代器使用<code>next()</code>调用迭代器对象的<code>__next__()</code>。<code>__iter__()</code>只会被调用一次，而<code>__next__()</code>会被调用 n 次。</p><p><img src="/2022/03/27/Python进阶/E:%5Chexo%5Csource_posts%5CPython%E8%BF%9B%E9%98%B6%5C%E8%BF%AD%E4%BB%A3%E8%BF%87%E7%A8%8B.png" alt="迭代过程"></p><p><strong>Iterator（迭代器）</strong></p><p>迭代器是一个带状态的对象，它能在你调用<code>next()</code>方法时返回容器中的下一个值，任何实现了<code>__iter__()</code>和<code>__next__()</code>方法的对象都是迭代器，<code>__iter__()</code>返回迭代器自身，<code>__next__()</code>返回容器中的下一个值，如果容器中没有更多元素了，则抛出<code>StopIteration</code>异常。</p><p>▲迭代器与列表的区别在于，构建迭代器的时候，不像列表把所有元素一次性加载到内存，而是以一种延迟计算（lazy evaluation）方式返回元素，这正是它的优点：节省空间。因为它并没有把<em>所有</em>元素装载到内存中，而是等到调用<code>next()</code>方法的时候才返回该元素（按需调用 call by need 的方式，本质上 for 循环就是不断地调用迭代器的<code>next()</code>方法）。</p><h3 id="generator生成器"><a class="markdownIt-Anchor" href="#generator生成器"></a> Generator（生成器）</h3><p>普通函数用<code>return</code>返回一个值,还有一种函数用<code>yield</code>返回值(yield是每次“惰性返回”一个值)，这种函数叫生成器函数。</p><p>函数被调用时会返回一个生成器对象。<strong>生成器其实是一种特殊的迭代器</strong>，不过这种迭代器更加优雅，它不需要像普通迭代器一样实现<code>__iter__()</code>和<code>__next__()</code>方法了，只需要一个<code>yield</code>关键字。生成器一定是迭代器（反之不成立），因此任何生成器也是一种懒加载的模式生成值。</p><p><code>yield</code>就是<code>return</code>返回一个值，并且记住这个返回的位置，下次迭代就从这个位置后（下一行）开始。<code>next</code>方法和<code>send</code>方法都可以返回下一个元素，区别在于<code>send</code>可以传递参数给<code>yield</code>表达式，这时<u>传递的参数会作为yield表达式的值</u>，而<u>yield的参数是返回给调用者的值</u>。</p><p><strong><code>send()</code>方法</strong>:</p><ul><li>send方法有一个参数value ，该参数value指定的是<strong>上一次被挂起的yield语句的返回值</strong>。send(value) 会把 value作为<code>yield express</code>的返回值赋值给接收者(即<code>xxx =</code>yield yyy`将value赋值给等号左边xxx)。</li><li>在使用send（）方法前，程序必须被<a href="https://so.csdn.net/so/search?q=%E6%8C%82%E8%B5%B7&amp;spm=1001.2101.3001.7020" target="_blank" rel="noopener">挂起</a>，不然会报错——所以生成器未启动之前send()中不能传非None的值</li></ul><p>注：</p><ul><li>在用生成器时，第一次要用<code>next</code>启动、或者是<code>send(None)</code>, 注意，虽然 send(None) 的功能是 next() 完全相同，但更推荐使用 next()，不推荐使用 send(None)。</li><li>send和next的执行很像，只是send可以和生成器互动，传入一个值。两者的执行也都是从上一个<code>yield exp</code>等号左边的位置再往后执行到下一个<code>yield</code>的位置</li></ul><p><strong><code>throw(</code>)方法</strong></p><p>throw() 方法的功能是，<strong>在生成器函数执行暂停处，抛出一个指定的异常</strong>，之后程序会继续执行生成器函数中后续的代码，直到遇到下一个 yield 语句。需要注意的是，如果到剩余代码执行完毕没有遇到下一个 yield 语句，则程序会抛出 StopIteration 异常。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">foo</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="keyword">try</span>:</span><br><span class="line">        <span class="keyword">yield</span> <span class="number">1</span></span><br><span class="line">    <span class="keyword">except</span> ValueError:</span><br><span class="line">        print(<span class="string">'捕获到 ValueError'</span>)</span><br><span class="line">f = foo()</span><br><span class="line">print(next(f))</span><br><span class="line">f.throw(ValueError)</span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">1</span></span><br><span class="line"><span class="string">捕获到 ValueError</span></span><br><span class="line"><span class="string">Traceback (most recent call last):</span></span><br><span class="line"><span class="string">  File "D:\python3.6\1.py", line 9, in &lt;module&gt;</span></span><br><span class="line"><span class="string">    f.throw(ValueError)</span></span><br><span class="line"><span class="string">StopIteration</span></span><br><span class="line"><span class="string">"""</span></span><br></pre></td></tr></table></figure><p>显然，一开始生成器函数在 yield 1 处暂停执行，当执行 throw() 方法时，它会先抛出 ValueError 异常，然后继续执行后续代码直至找到下一个 yield 语句，在执行过程中ValueError被try捕捉，而之后由于程序后续不再有 yield 语句，因此执行到最后会抛出一个 StopIteration 异常。</p><p>案例见：<a href="http://c.biancheng.net/view/7090.html" target="_blank" rel="noopener">http://c.biancheng.net/view/7090.html</a></p><h4 id="分析一个最简单的生成器"><a class="markdownIt-Anchor" href="#分析一个最简单的生成器"></a> 分析一个最简单的生成器：</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">foo</span><span class="params">()</span>:</span></span><br><span class="line">    bar_a = <span class="keyword">yield</span> <span class="string">"hello"</span></span><br><span class="line">    bar_b = <span class="keyword">yield</span> bar_a</span><br><span class="line">    <span class="keyword">yield</span> bar_b</span><br><span class="line">f = foo()</span><br><span class="line">print(f.send(<span class="keyword">None</span>))</span><br><span class="line">print(f.send(<span class="string">"C语言中文网"</span>))</span><br><span class="line">print(f.send(<span class="string">"http://c.biancheng.net"</span>))</span><br></pre></td></tr></table></figure><p>分析一下此程序的执行流程：</p><ol><li>首先，构建生成器函数，并利用器创建生成器（对象）f 。</li><li>使用生成器 f 调用无参的 send() 函数，其功能和 next() 函数完全相同，因此开始执行生成器函数，即执行到第一个 yield “hello” 语句，该语句会返回 “hello” 字符串，然后程序停止到此处（注意，此时还未执行对 bar_a 的赋值操作）。</li><li>下面开始使用生成器 f 调用有参的 send() 函数，首先它会将暂停的程序开启，同时还会将其参数“C语言中文网”赋值给当前 yield 语句的接收者，也就是 bar_a 变量。程序一直执行完 yield bar_a 再次暂停，因此会输出“C语言中文网”。</li><li>最后依旧是调用有参的 send() 函数，同样它会启动餐厅的程序，同时将参数“<a href="http://c.biancheng.net" target="_blank" rel="noopener">http://c.biancheng.net</a>”传给 bar_b，然后执行完 yield bar_b 后（输出 <a href="http://c.biancheng.net" target="_blank" rel="noopener">http://c.biancheng.net</a>），程序执行再次暂停。</li></ol><h2 id="yield和yield-from"><a class="markdownIt-Anchor" href="#yield和yield-from"></a> <a href="https://blog.csdn.net/qq_27825451/article/details/85244237" target="_blank" rel="noopener">yield和yield from</a></h2><blockquote><p>在进入协程之前，我们需要明白<code>yield</code>和<code>yield from</code>，其是官方实现协程的关键.</p><p><code>yield from</code> 是在Python3.3才出现的语法</p></blockquote><ul><li><code>yield</code>是每次“惰性返回”一个值</li><li><code>yield from</code>是“从什么（生成器）里面返回，实际上就是委托给了另一个生成器：<ul><li>yield from 后面可以跟的可以是“ 生成器 、元组、 列表、range()函数产生的序列等<strong>可迭代对象</strong>”</li><li>从本质上讲，委托给了另一个生成器的意思是: <code>yield from iterable</code>本质上等于 <code>for item in iterable: yield item</code></li></ul></li></ul><p>当然除了最显而易见的使用区别以外，yield from还对yield的不足进行了改进。</p><ol><li><p><strong>针对yield无法获取生成器return的返回值</strong>：在使用yield生成器的时候，如果使用for语句去迭代生成器，则不会显式的出发StopIteration异常，而是自动捕获StopIteration异常，所以如果遇到return，只是会终止迭代，而不会触发异常，故而也就没办法获取return的值。</p><p>for迭代语句不会显式触发异常，故而无法获取到return的值，迭代到2的时候遇到<strong>return语句，隐式的触发了StopIteration异常</strong>，就终止迭代了，但是在程序中不会显示出来。</p><p>yield from具有以下几个特点：</p><p>（1）上面的my_generator是原始的生成器，main是调用方，使用yield的时候，只涉及到这两个函数，即“调用方”与“生成器（协程函数）”是直接进行交互的，不涉及其他方法，即“调用方——&gt;生成器函数(协程函数)”；</p><p>（2）在使用yield from的时候，多了一个对原始my_generator的包装函数，然后调用方是通过这个包装函数（后面会讲到它专有的名词）来与生成器进行交互的，即“调用方——&gt;生成器包装函数——&gt;生成器函数(协程函数)”；</p><p>（3）yield from iteration结构会在内部自动捕获 iteration生成器的StopIteration 异常。这种处理方式与 for 循环处理 StopIteration 异常的方式一样。而且对 yield from 结构来说，解释器不仅会捕获 StopIteration 异常，还会把return返回的值或者是StopIteration的value 属性的值变成 yield from 表达式的值，即上面的result。</p></li><li><p><strong>yield from所实现的数据传输通道</strong>：yield涉及到“调用方与生成器两者”的交互，生成器通过next()的调用将值返回给调用者，而调用者通过send()方法向生成器发送数据；</p><p>PEP 380 使用了一些yield from使用的专门术语：</p><ul><li>委派生成器：<strong>包含</strong><code>yield from &lt;iterable&gt;</code>表达式的生成器函数；即上面的wrap_my_generator生成器函数.在调用方与子生成器之间建立一个<strong>双向通道</strong>。</li><li>子生成器：从<code>yield from 表达式中 &lt;iterable&gt;</code>部分<strong>被获取</strong>的生成器函数；即上面的my_generator生成器函数</li><li>调用方：<strong>调用委派生成器的客户端</strong>代码；即上面的main生成器函数</li></ul><p>（1）yield from主要设计用来向子生成器委派操作任务，但yield from可以向任意的可迭代对象委派操作；</p><p>（2）委派生成器（group）相当于管道，所以可以把任意数量的委派生成器连接在一起—一个委派生成器使用yield from 调用一个子生成器，而那个子生成器本身也是委派生成器，使用yield from调用另一个生成器。</p></li><li><p>可以帮助处理StopIteration异常： 我们在调用生成器的时候，一般要么for，要么next。在next的时候我们就得自己去关注生成器什么时候会爆出StopIteration异常，而将其yield from委托给委派生成器后，调用者直接对委派生成器操作，其会自动帮我们处理<code>thorw()</code>、<code>send()</code>、<code>close()</code></p><p>任何使用send()方法发给委派生产器（即外部生产器）的值被直接传递给迭代器。如果send值是None，则调用迭代器next()方法；如果不为None，则调用迭代器的send()方法。如果对迭代器的调用产生StopIteration异常，委派生产器恢复继续执行yield from后面的语句；若迭代器产生其他任何异常，则都传递给委派生产器</p></li><li><p>yield 返回一个、yield from 返回一堆</p></li></ol><p>其实yield from最重要的作用就是提供了一个“数据传输的管道”，打开双向通道，把最外层的调用方法与最内层的子生成器连接起来，这两者就可以进行发送值和返回值了。例子见：<a href="https://blog.csdn.net/qq_27825451/article/details/85244237" target="_blank" rel="noopener">https://blog.csdn.net/qq_27825451/article/details/85244237</a></p><p>yield from使用案例: <a href="https://www.v2ex.com/t/516739" target="_blank" rel="noopener">使用 yeild from 写一个异步爬虫</a></p><h2 id="python协程"><a class="markdownIt-Anchor" href="#python协程"></a> Python协程</h2><p>随着Go的普及，越来越多人听到了协程的概念，也就是用户级线程，相比于线程而言，由于不涉及用户态和内核态的切换，因此性能更好，此外也不会出现竞态条件，对锁的要求也<strong>少</strong>很多。对于受限于GIL的Python而言更是如此，对此，Python也不断在优化并发，比如比较熟为人知的thread 模块多线程和 multiprocessing 多进程，后来慢慢引入基于 yield 关键字，逐渐引入了<strong>协程</strong>的概念，但之后这样的协程写法也不太被官方推荐，而是在Python3.5后使用特地提供的关键字： <strong>async/await</strong></p><blockquote><p>并发指的是 一个 CPU 同时处理多个程序，但是在同一时间<strong>点</strong>只会处理其中一个。虽然一个时刻只能处理一个任务，但是因为程序切换的速度非常快，1 秒钟内可以完全很多次程序切换，肉眼无法感知，所以对于用户而言就相当于是同时运行的。最典型的就是操作系统上运行N个应用程序，其实也是多进程并行的，但给人的感觉却是同时的。</p><p>所以，我们也能看出并发的核心在于，如何让程序更快的进行切换。</p></blockquote><p>在此顺便普及<strong>并发、并行、同步和异步</strong>概念区别：</p><ul><li>并发指的是 一个 CPU 同时处理多个程序，但是在同一时间<strong>点</strong>只会处理其中一个。</li><li>并行指的是多个 CPU 同时处理多个程序，同一时间点可以处理多个。</li><li>同步：执行 IO 操作时，必须等待任务执行完成得到返回结果后，再执行后一个任务。</li><li>异步：执行 IO 操作时，不必等待执行完成的返回结果，就执行下一个任务。</li></ul><h3 id="协程线程和进程的区别"><a class="markdownIt-Anchor" href="#协程线程和进程的区别"></a> 协程，线程和进程的区别</h3><ul><li>多进程通常利用的是多核 CPU 的优势，同时执行多个计算任务。每个进程有自己独立的内存管理，所以不同进程之间要进行数据通信比较麻烦。</li><li>多线程是在一个 cpu 上创建多个子任务，当某一个子任务休息的时候其他任务接着执行。多线程的控制是由 python 自己控制的。 子线程之间的内存是共享的，并不需要额外的数据通信机制。但是线程存在数据同步问题，所以要有锁机制。</li><li>协程的实现是在一个线程内实现的，相当于流水线作业。由于线程切换的消耗比较大，所以对于并发编程，可以优先使用协程。</li></ul><p>写一个简单的协程:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="comment"># 1. 在普通的函数前面加 async 关键字；</span></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">visit_url</span><span class="params">(url, response_time)</span>:</span></span><br><span class="line">    <span class="string">"""访问 url"""</span></span><br><span class="line">    <span class="comment"># 2. await 表示在这个地方会等待子函数执行完成，再往下执行。（在并发操作中，把程序控制权教给主程序，让他分配其他协程执行。实际就是让出CPU时间片，挂载起任务。） await 只能在带有 async 关键字的函数中声明。</span></span><br><span class="line">    <span class="keyword">await</span> asyncio.sleep(response_time)</span><br><span class="line">    <span class="keyword">return</span> <span class="string">f"访问<span class="subst">&#123;url&#125;</span>, 已得到返回结果"</span></span><br><span class="line"></span><br><span class="line">start_time = time.perf_counter()</span><br><span class="line">task = visit_url(<span class="string">'http://wangzhen.com'</span>, <span class="number">2</span>)</span><br><span class="line"><span class="comment"># 3， asynico.run() 运行程序</span></span><br><span class="line">asyncio.run(task)</span><br><span class="line">print(<span class="string">f"消耗时间：<span class="subst">&#123;time.perf_counter() - start_time&#125;</span>"</span>)</span><br></pre></td></tr></table></figure><p>由于上面只有一个任务，不涉及任务切换，因此写一个可以比较的实验代码</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">visit_url</span><span class="params">(url, response_time)</span>:</span></span><br><span class="line">    <span class="string">"""访问 url"""</span></span><br><span class="line">    <span class="keyword">await</span> asyncio.sleep(response_time)</span><br><span class="line">    <span class="keyword">return</span> <span class="string">f"访问<span class="subst">&#123;url&#125;</span>, 已得到返回结果"</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">run_task</span><span class="params">()</span>:</span></span><br><span class="line">    <span class="string">"""收集子任务"""</span></span><br><span class="line">    task_1 = visit_url(<span class="string">'http://wangzhen.com'</span>, <span class="number">2</span>)</span><br><span class="line">    task_2 = visit_url(<span class="string">'http://another'</span>, <span class="number">3</span>)</span><br><span class="line">    <span class="comment"># 当运行到 task_1 中的sleep时会挂载起来，让task2执行</span></span><br><span class="line">    <span class="keyword">await</span> asyncio.run(task)</span><br><span class="line">    <span class="comment"># 由于 task_1 被挂起了，因此就运行task2，呆执行到task2中sleep时task2也会被挂起，此时线程只好不断检查是否有可用的协程，直至 task_1 的sleep结束, 继续运行task_1之后的逻辑</span></span><br><span class="line">    <span class="keyword">await</span> asyncio.run(task_2)</span><br><span class="line"></span><br><span class="line">asyncio.run(run_task())</span><br><span class="line">print(<span class="string">f"消耗时间：<span class="subst">&#123;time.perf_counter() - start_time&#125;</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># asyncio.run可以改成如下</span></span><br><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line">loop = asyncio.get_event_loop(visit_url(<span class="string">'http://wangzhen.com'</span>, <span class="number">2</span>))</span><br><span class="line"><span class="comment"># 多个任务的话, 通过asyncio.wait来等待同时完成</span></span><br><span class="line">res = loop.run_until_complete(asyncio.wait([visit_url(<span class="string">'http://wangzhen.com'</span>, <span class="number">2</span>), visit_url(<span class="string">'http://another'</span>, <span class="number">3</span>)]))</span><br><span class="line"><span class="comment"># 注： run_until_complete传的是 types.CoroutineType 类型, 不能是 types.AsyncGeneratorType</span></span><br><span class="line"><span class="comment"># asyncio.run是 3.7 的新内容</span></span><br></pre></td></tr></table></figure><p>明确的点：</p><ol><li><p>增加并发，实际上就是减少一些无效的等待，由于一些IO任务会比较耗时，如果一直让CPU等待则大大减低了执行效率，因此await就是碰到耗时任务时，主动告诉cpu让其去做其他事，所以await后面都是耗时函数</p></li><li><p>由于await的设计：后面都必须跟的是Awaitable对象，或者是实现了其协议的对象，因此跟一般写法有些许出入，比如time.sleep改成了async.sleep，比如requests.get改成了await <code>aiohttp.ClientSession().get</code></p><p>awaitable对象必须满足如下条件中其中之一</p><ul><li>原生协程对象<ul><li>通过async def定义的函数是原生的协程对象，<code>async def download()</code>，但如果download内部没有调用异步代码，即使用的是requests.get，阻塞后实际上还是串行的，所以要用异步的aiohttp代替requests</li></ul></li><li><code>types.coroutine()</code>修饰的<strong>基于生成器的协程对象</strong>，注意不是Python3.4中asyncio.coroutine</li><li>实现了<code>__await__</code> method，并在其中返回了iterator的<strong>对象</strong></li></ul></li><li><p>await只能用在async标明的函数中</p></li></ol><p><strong>参考：</strong></p><ul><li>★. <a href="https://zhuanlan.zhihu.com/p/27258289" target="_blank" rel="noopener">Python Async/Await入门指南</a>——写法，对比，Awaitable具体实现</li><li><a href="https://www.cnblogs.com/heniu/p/12740400.html" target="_blank" rel="noopener">python教程：使用 async 和 await 协程进行并发编程</a>——概念</li><li>怎么掌握asyncio? - 灵剑的回答 - 知乎 <a href="https://www.zhihu.com/question/294188439/answer/555273313%E2%80%94%E2%80%94%E2%98%85%E5%8D%8F%E7%A8%8B%E6%A1%86%E6%9E%B6%E8%AE%BE%E8%AE%A1" target="_blank" rel="noopener">https://www.zhihu.com/question/294188439/answer/555273313——★协程框架设计</a></li></ul><p>注：</p><ul><li>多线程主要用于IO密集型任务、多进程主要用于CPU密集型任务。因此协程的主要应用场景是 IO 密集型任务：①网络请求，比如吧虫，大量使用aiohttp②文件读取，aiofile ③web框架，aiohttp,fastapi④数据库查询，asyncpg、databases</li><li>协程也是需要上锁的：<a href="https://zhuanlan.zhihu.com/p/40279108" target="_blank" rel="noopener">深入探讨，协程是否是线程安全(协程安全)的？</a>——协程的安全指的是，在你交出控制权之前是安全的，交出以后当然不安全啊。因此针对改变公共资源中间切换协程的情况需要上锁。例子中，self.a的加和减都必须在一起，中间不能await让出(如果run里面不用await交出控制权，或者统一在await之后进行变量的操作，可以实现协程安全)----自己的话理解一下就是：用户在await让出控制权之前操作都是事务安全的，但是如果在await前后都对公共资源进行修改了（比如self.a)，那么肯定会出错：因为执行顺序不可控，所以执行快（不可控）的协程可能会在任意时间进行await后的self.a-=1，导致每次的结果不一致</li></ul><h3 id="网络请求速度对比"><a class="markdownIt-Anchor" href="#网络请求速度对比"></a> 网络请求速度对比</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">from</span> concurrent.futures <span class="keyword">import</span> wait, ALL_COMPLETED</span><br><span class="line"><span class="keyword">from</span> concurrent.futures.thread <span class="keyword">import</span> ThreadPoolExecutor</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> aiohttp</span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line">times = <span class="number">100</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">tt</span><span class="params">()</span>:</span></span><br><span class="line">    r = requests.get(<span class="string">"http://httpbin.org/get"</span>)</span><br><span class="line">    <span class="keyword">return</span> r.status_code</span><br><span class="line"><span class="comment"># 单线程</span></span><br><span class="line">    start = time.time_ns()</span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(times):</span><br><span class="line">        tt()</span><br><span class="line">    print((time.time_ns() - start)/<span class="number">1e6</span>)</span><br><span class="line">    <span class="comment"># 46204.7239</span></span><br><span class="line"></span><br><span class="line">    <span class="comment"># 多线程</span></span><br><span class="line">    start = time.time_ns()</span><br><span class="line">    <span class="keyword">with</span> ThreadPoolExecutor() <span class="keyword">as</span> f:</span><br><span class="line">        task = [f.submit(tt) <span class="keyword">for</span> i <span class="keyword">in</span> range(times)]</span><br><span class="line">    wait(task, <span class="number">120</span>, return_when=ALL_COMPLETED)</span><br><span class="line">    print((time.time_ns() - start)/<span class="number">1e6</span>)</span><br><span class="line">    <span class="comment"># 6078.0416</span></span><br><span class="line"></span><br><span class="line">    <span class="comment"># 协程</span></span><br><span class="line">    start = time.time_ns()</span><br><span class="line">    <span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">asyncGet</span><span class="params">()</span>:</span></span><br><span class="line">        <span class="keyword">async</span> <span class="keyword">with</span> aiohttp.ClientSession() <span class="keyword">as</span> sess:</span><br><span class="line">            <span class="keyword">async</span> <span class="keyword">with</span> sess.get(<span class="string">"http://httpbin.org/get"</span>) <span class="keyword">as</span> resp:</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">await</span> resp.json()</span><br><span class="line">    <span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">userth</span><span class="params">(t)</span>:</span></span><br><span class="line">        res = <span class="keyword">await</span> asyncGet()</span><br><span class="line">        print(t)</span><br><span class="line">    loop = asyncio.get_event_loop()</span><br><span class="line">    loop.run_until_complete(asyncio.wait([userth(i) <span class="keyword">for</span> i <span class="keyword">in</span> range(times)]))</span><br><span class="line">    loop.close()</span><br><span class="line">    print((time.time_ns() - start)/<span class="number">1e6</span>)</span><br><span class="line"><span class="comment"># 997.2545</span></span><br></pre></td></tr></table></figure><h3 id="gevent和asyncio区别"><a class="markdownIt-Anchor" href="#gevent和asyncio区别"></a> gevent和asyncio区别</h3><ul><li><p>asyncio是Python 3.4版本引入的标准库，直接内置了对异步IO的支持，不需要第三方的支持，我们从asyncio模块中直接获取一个EventLoop的引用，然后把需要执行的协程扔到EventLoop中执行，就实现了异步IO。</p><p>asycio 需要自己在代码中让出CPU，控制权在自己手上</p></li><li><p>gevent是第三方库，通过greenlet实现协程，其基本思路是：当一个greenlet遇到IO操作时，就自动切换到其他的greenlet，等到IO操作完成，再在适当的时候切换回来继续执行。</p><p>gevent 用会替换标准库，你以为调用的是标准库的方法实际已经被替换成gevent自己的实现，遇到阻塞调用，gevent会自动让出CPU</p></li></ul><h3 id="使用uvloop加速"><a class="markdownIt-Anchor" href="#使用uvloop加速"></a> 使用uvloop加速</h3><p>uvloop基于libuv，libuv是一个使用C语言实现的高性能异步I/O库，uvloop用来代替asyncio默认事件循环，可以进一步加快异步I/O操作的速度。</p><p>uvloop的使用非常简单，只要在asnycio获取事件循环前设置<code>asnycio.set_event_loop_policy(uvloop.EventLoopPolicy())</code>，就将asyncio的事件循环策略设置为uvloop的事件循环策略。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">from</span> concurrent.futures <span class="keyword">import</span> wait, ALL_COMPLETED</span><br><span class="line"><span class="keyword">from</span> concurrent.futures.thread <span class="keyword">import</span> ThreadPoolExecutor</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> aiohttp</span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">times = <span class="number">100</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line">    start = time.time_ns()</span><br><span class="line">    <span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">asyncGet</span><span class="params">()</span>:</span></span><br><span class="line">        <span class="keyword">async</span> <span class="keyword">with</span> aiohttp.ClientSession() <span class="keyword">as</span> sess:</span><br><span class="line">            <span class="keyword">async</span> <span class="keyword">with</span> sess.get(<span class="string">"http://httpbin.org/get"</span>) <span class="keyword">as</span> resp:</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">await</span> resp.json()</span><br><span class="line">    <span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">userth</span><span class="params">(t)</span>:</span></span><br><span class="line">        res = <span class="keyword">await</span> asyncGet()</span><br><span class="line">        <span class="comment"># print(t)</span></span><br><span class="line">    loop = asyncio.get_event_loop()</span><br><span class="line">    loop.run_until_complete(asyncio.wait([userth(i) <span class="keyword">for</span> i <span class="keyword">in</span> range(times)]))</span><br><span class="line">    loop.close()</span><br><span class="line">    print((time.time_ns() - start)/<span class="number">1e6</span>)</span><br><span class="line"><span class="comment"># 7488.398822</span></span><br><span class="line">    </span><br><span class="line">    <span class="comment"># 使用uvloop</span></span><br><span class="line">    <span class="keyword">try</span>:</span><br><span class="line">        <span class="keyword">import</span> uvloop</span><br><span class="line">    <span class="keyword">except</span> ModuleNotFoundError:</span><br><span class="line">        print(<span class="string">"无法启用uvloop"</span>)</span><br><span class="line">    <span class="keyword">else</span>:</span><br><span class="line">        print(<span class="string">"启用uvloop"</span>)</span><br><span class="line">        asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())</span><br><span class="line">    start = time.time_ns()</span><br><span class="line">    <span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">asyncGet</span><span class="params">()</span>:</span></span><br><span class="line">        <span class="keyword">async</span> <span class="keyword">with</span> aiohttp.ClientSession() <span class="keyword">as</span> sess:</span><br><span class="line">            <span class="keyword">async</span> <span class="keyword">with</span> sess.get(<span class="string">"http://httpbin.org/get"</span>) <span class="keyword">as</span> resp:</span><br><span class="line">                <span class="keyword">return</span> <span class="keyword">await</span> resp.json()</span><br><span class="line">    <span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">userth</span><span class="params">(t)</span>:</span></span><br><span class="line">        res = <span class="keyword">await</span> asyncGet()</span><br><span class="line">        <span class="comment"># print(t)</span></span><br><span class="line">    loop = asyncio.get_event_loop()</span><br><span class="line">    done, _ = loop.run_until_complete(asyncio.wait([userth(i) <span class="keyword">for</span> i <span class="keyword">in</span> range(times)]))</span><br><span class="line">    <span class="comment"># print([t.result() for t in done])  # 从request-html源码中看到</span></span><br><span class="line">    loop.close()</span><br><span class="line">    print(<span class="string">"uvloop"</span>, (time.time_ns() - start)/<span class="number">1e6</span>)</span><br><span class="line"><span class="comment"># uvloop 4617.166035</span></span><br></pre></td></tr></table></figure><p>注： 2022年4月26日目前不支持在windows上安装uvloop</p><p>附： aiohttp利用协程批量下载图片: <a href="https://pythondict.com/scrapy/python-file-download/%E2%80%94%E2%80%94%E5%8F%AF%E4%BB%A5%E9%80%9A%E8%BF%87asyncio%E5%88%9B%E5%BB%BA%E4%BA%8B%E4%BB%B6%E5%BE%AA%E7%8E%AF%EF%BC%8C%E8%80%8C%E4%B8%8D%E6%98%AFsession.run(%E8%BF%99%E4%B8%AA%E4%B9%9F%E4%BC%9A%E5%88%9B%E5%BB%BA%E4%BA%8B%E4%BB%B6%E5%BE%AA%E7%8E%AF)" target="_blank" rel="noopener">https://pythondict.com/scrapy/python-file-download/——可以通过asyncio创建事件循环，而不是session.run(这个也会创建事件循环)</a></p><h3 id="api说明与对比"><a class="markdownIt-Anchor" href="#api说明与对比"></a> API说明与对比</h3><ul><li><code>asyncio.run</code>:<code>asyncio.run()</code>在 Python3.7才提出,可以省去显式的定义事件循环的步骤</li><li><code>asyncio.get_event_loop</code>和<code>loop.run_util_complete</code>：创建时间循环并执行协程任务<ul><li><code>task.add_done_callback(callback)</code>，callback为一个函数</li></ul></li></ul><p>见：<a href="https://cloud.tencent.com/developer/article/1194415#:~:text=asyncio.run%20%28%29%20%E5%87%BD%E6%95%B0%E7%9A%84%E5%AE%98%E6%96%B9%E6%96%87%E6%A1%A3%E6%98%AF%E8%BF%99%E6%A0%B7%E5%AD%90%E7%9A%84%3A%20Signature%3A%20asyncio.run%28main%2C%20%2A%2C%20debug%20%3DFalse%29,the%20asyncio%20event%20loop%20and%20finalizing%20asynchronous%20generators." target="_blank" rel="noopener">Python3.7的新API:asyncio.run()</a></p><h4 id="wait和gather"><a class="markdownIt-Anchor" href="#wait和gather"></a> wait和gather</h4><ul><li><code>asyncio.wait</code></li><li><code>asyncio.gather</code>:先把所有的协程任务事先创建好，然后一次性交给 <code>asyncio.gather(*tasks)</code>，gather会将协程任务再都加入到事件循环中</li></ul><p><strong>asyncio.gather 和asyncio.wait区别:</strong></p><p>共同点：</p><ul><li>两个方法的参数都是接受多个future或coro组成的列表，其作用都是吧把所有 Task 任务结果收集起来</li></ul><p>不同点</p><ul><li><p>wait：</p><p>在内部wait()使用一个<strong>set</strong>保存它创建的Task实例。因为set是<strong>无序</strong>的所以这也就是我们的任务不是顺序执行的原因。</p><ol><li>wait的返回值是一个元组，包括两个集合：<code>done</code> 和 <code>pending</code>，分别表示已完成的协程和超时未完成的task，想知道done的结果需要通过<code>.result</code>来获得resultg结果</li><li>wait第二个参数为一个超时值，达到这个超时时间后，未完成的任务状态变为pending，当程序退出时还有任务没有完成此时就会看到错误提示。</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">  <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">5</span>):</span><br><span class="line">       task = asyncio.create_task(async_func(i))</span><br><span class="line">       task_list.append(task)</span><br><span class="line"><span class="comment"># 不需要拆包, 当设置return_when=asyncio.tasks.FIRST_COMPLETED, pending有值</span></span><br><span class="line">   done, pending = <span class="keyword">await</span> asyncio.wait(task_list, timeout=<span class="keyword">None</span>)</span><br><span class="line">   <span class="keyword">for</span> done_task <span class="keyword">in</span> done:</span><br><span class="line">       print((<span class="string">f"[<span class="subst">&#123;current_time()&#125;</span>] 得到执行结果 <span class="subst">&#123;done_task.result()&#125;</span>"</span>))</span><br></pre></td></tr></table></figure></li><li><p>gather的作用和wait类似不同的是：</p><ol><li><p>gather任务无法取消。</p></li><li><p><u>返回值是一个结果列表</u></p></li><li><p>按照传入协程的<strong>顺序</strong>，会顺序保存的对应协程的执行结果输出。</p><p>实验输出可见: <a href="https://www.jianshu.com/p/6872bf356af7" target="_blank" rel="noopener">Python：asyncio.wait 和 asyncio.gather 的异同</a></p></li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">   <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">5</span>):</span><br><span class="line">       task = asyncio.create_task(func(i))</span><br><span class="line">       task_list.append(task)</span><br><span class="line"><span class="comment"># 这边需要拆包</span></span><br><span class="line">   results = <span class="keyword">await</span> asyncio.gather(*task_list)</span><br><span class="line">   <span class="comment"># ▲可以看到跟上述不同的是，这里不需要通过`.result()`来获得结果值</span></span><br><span class="line">   <span class="keyword">for</span> result <span class="keyword">in</span> results:</span><br><span class="line">       print((<span class="string">f"[<span class="subst">&#123;current_time()&#125;</span>] 得到执行结果 <span class="subst">&#123;result&#125;</span>"</span>))</span><br></pre></td></tr></table></figure><p>★在大部分情况下，用asyncio.gather是足够的，如果你有特殊需求，可以选择asyncio.wait，举2个例子：</p><ol><li>需要拿到封装好的Task，以便取消或者添加成功回调等</li><li>业务上需要<code>FIRST_COMPLETED / FIRST_EXCEPTION</code>即返回的</li></ol></li></ul><p>详细实现：<a href="https://www.cnblogs.com/MrVolleyball/p/15812407.html" target="_blank" rel="noopener">从头造轮子：python3 asyncio之 gather （3）</a>、<a href="https://blog.csdn.net/weixin_30444111/article/details/113558620" target="_blank" rel="noopener">ensure_future（确保参数为future对象）和create_task（创建Task对象）</a></p><p>注：使用搭配：<code>loop.run_until_complete(asyncio.wait([userth(i) for i in range(times)]))</code>或者<code>asyncio.run(main()、task1 = asyncio.create_task(func())</code></p><h4 id="ensure_future和create_task"><a class="markdownIt-Anchor" href="#ensure_future和create_task"></a> ensure_future和create_task</h4><ul><li><p><code>asyncio.ensure_future</code>： <code>future = asyncio.Future()、future.set_result('data')、await future</code>证明future对象都是awaitable，这也是协程中主要操作的对象coroutine，另一个是Task（实际上是Future的子类）</p><blockquote><p>作用为: <code>Wrap a coroutine or an awaitable in a future.</code>，即将一个协程或者是Awaitable对象转换成Future对象（确保这个是一个Future对象）==&gt;参数需要为: <code>TypeError: An asyncio.Future, a coroutine or an awaitable is required</code></p></blockquote><ul><li><code>asyncio.wait</code>中就调用了，实际上在执行协程的过程中都得转成Future对象</li></ul><p>📖Future执行逻辑讲解：<a href="https://zhuanlan.zhihu.com/p/27258289" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/27258289</a></p></li></ul><p><code>asyncio.create_task</code>：Python 3.7提出的统一的、更高阶的<em>创建Task</em>方法</p><p><code>asyncio.create_task</code>与<code>asyncio.ensure_future</code></p><ul><li><code>loop.create_task</code>接受的参数需要是一个协程，</li><li>但是<code>asyncio.ensure_future</code>除了接受协程，还可以是Future对象或者awaitable对象:<ul><li>如果参数是协程，其实底层还是用的<code>loop.create_task</code>，返回Task对象</li><li>如果是Future对象会直接返回</li><li>如果是一个awaitable对象会await这个对象的__await__方法，再执行一次<code>ensure_future</code>，最后返回Task或者Future</li></ul></li></ul><p>使用async修饰的函数调用时会返回一个协程对象，await只能放在async修饰的函数里面使用，await后面必须要跟着一个协程对象或Awaitable，await的目的是等待协程控制流的返回，而实现暂停并挂起函数的操作是yield。</p><h3 id="异步库"><a class="markdownIt-Anchor" href="#异步库"></a> 异步库</h3><ul><li>asyncio ： 标准库<ul><li>除了asyncio之外，curio和trio是更加轻量级的替代物，而且也更容易使用</li></ul></li><li>aiohttp： http库</li><li>aiofiles： 文件库</li><li>requests-html：支持异步访问</li><li>aiomysql：mysql异步库</li></ul><p>注意：</p><ul><li>判别函数是否使用了协程： 可能针对整个程序是异步的。但是对于 <code>main()</code>，它的 <code>for</code>循环还是阻塞的——for循环的任务并没有被添加到事件循环中，事件循环中实际上就只有main一个协程任务。具体案例见：<a href="https://zhuanlan.zhihu.com/p/65212327" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/65212327</a></li></ul><p>▲. 可以对比看看 Go的协程 —— <a href="https://juejin.cn/post/6844903624062009352" target="_blank" rel="noopener">[译] 通过插图学习 Go 的并发</a></p><h2 id="项目结构"><a class="markdownIt-Anchor" href="#项目结构"></a> 项目结构</h2><blockquote><p>关于项目结构组织。由于Python在执行程序时，会自动将<code>.</code>即当前路径加入到库搜索路径，所以当下路径下的包（文件夹下有<code>__init__.py</code>）都能被检测到使用，如下面的handler、helper都可以直接在其他文件里通过<code>helper.xxx</code>来调用，但是顶层的<code>settings.py</code>不能通过<code>feiyu_shoot.settings</code>来调用，即使feiyu_shoot工程下也有<code>__init__.py</code>，因为项目工程的上层目录并没有加入到Python环境搜索中，所以他实际不知道谁是<code>feiyu_shoot</code></p><p>因此，根据这种设计，可以将项目结构组织成两种</p></blockquote><ol><li><p>不依赖子模块的放在root目录的top接口：需要被其他文件引用的列入文件夹中作为库使用，不需要被引用的可以直接放顶层</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># helper/runner.py</span></span><br><span class="line"><span class="comment"># handler/ 在.下作为包导入</span></span><br><span class="line"><span class="keyword">from</span> handler imoprt ConfigHandler</span><br><span class="line"><span class="comment"># settings.py 在.下作为模块导入</span></span><br><span class="line"><span class="keyword">from</span> settings <span class="keyword">import</span> USER_AGENT</span><br></pre></td></tr></table></figure><p><img src="/2022/03/27/Python进阶/F:%5CaDevelopment%5CPython%5CpythonGitProject%5Cfeiyu_shoot%5Cdebugs%5CtopFile.jpg" alt="topFile"></p></li><li><p>★将入口放在最外层，核心内容作为单独一个文件夹（模块）：由于feiyu是个整体的包，所以在feiyu下的任意Python文件中都可以通过<code>from feiyu.xxx import yyy</code>来导入。</p></li></ol><p><img src="/2022/03/27/Python进阶/F:%5CaDevelopment%5CPython%5CpythonGitProject%5Cfeiyu_shoot%5Cdebugs%5ConeEntry.jpg" alt="oneEntry"></p><p>根据<a href="https://www.jianshu.com/p/e33b4f0373bc" target="_blank" rel="noopener">Python打包利器：auto-py-to-exe</a>中打包计算机程序的方式，更推荐第二种，这样代码资源文件位置更统一，也更加清晰一些。</p><p>more： <a href="https://zhuanlan.zhihu.com/p/403558690#:~:text=%E8%A7%82%E7%82%B91%EF%BC%9A%20%E4%B8%A5%E6%A0%BC%E6%9D%A5%E8%AF%B4Python%E4%B8%AD%E6%98%AF%E6%B2%A1%E6%9C%89%E5%BA%93,%28library%29%E7%9A%84%E6%A6%82%E5%BF%B5%EF%BC%8C%E6%A8%A1%E5%9D%97%20%28module%29%E5%92%8C%E5%8C%85%20%28package%29%E9%83%BD%E6%98%AFPython%E8%AF%AD%E6%B3%95%E4%B8%AD%E6%9C%89%E7%9A%84%E6%A6%82%E5%BF%B5%EF%BC%8CPython%E4%B8%AD%E7%9A%84%E5%BA%93%E6%98%AF%E5%80%9F%E7%94%A8%E5%85%B6%E4%BB%96%E7%BC%96%E7%A8%8B%E8%AF%AD%E8%A8%80%E7%9A%84%E6%A6%82%E5%BF%B5%EF%BC%8C%E6%B2%A1%E6%9C%89%E7%89%B9%E5%88%AB%E5%85%B7%E4%BD%93%E7%9A%84%E5%AE%9A%E4%B9%89%EF%BC%8C%E5%8F%AA%E6%98%AF%E4%B8%80%E4%B8%AA%E9%80%9A%E4%BF%97%E7%9A%84%E8%AF%B4%E6%B3%95%EF%BC%8C%E5%B9%B3%E6%97%B6%E8%AF%B4%E7%9A%84%E5%BA%93%E6%97%A2%E5%8F%AF%E4%BB%A5%E6%98%AF%E4%B8%80%E4%B8%AA%E6%A8%A1%E5%9D%97%E4%B9%9F%E5%8F%AF%E4%BB%A5%E6%98%AF%E4%B8%80%E4%B8%AA%E5%8C%85%E3%80%82" target="_blank" rel="noopener">Python中模块、包、库定义</a></p><h2 id="深入理解"><a class="markdownIt-Anchor" href="#深入理解"></a> <a href="https://pyinstaller.org/en/v4.10/runtime-information.html#using-file" target="_blank" rel="noopener">深入理解<code>__file__</code></a></h2><blockquote><p>我们在定义一些配置文件、数据文件位置时，通常是基于脚本的位置，但是如果直接通过相对路径很容易出错，因此一般是通过<code>__file__</code>获得脚本的绝对路径后再进行相对的路径操作，常见代码: <code>os.path.dirname(os.path.abspath(__file__))</code>从而获得当前脚本所在目录的绝对路径。(现在关于路径相关的操作是提倡用pathlib.path代替os.path，对应操作为<code>Path(__file__).resolve().parent</code>)</p></blockquote><blockquote><p>But: pyinstaller打包后使用<code>__file__</code>报错有问题, <a href="https://github.com/pyinstaller/pyinstaller/issues/6719%E3%80%81https://github.com/pyinstaller/pyinstaller/pull/6616" target="_blank" rel="noopener">https://github.com/pyinstaller/pyinstaller/issues/6719、https://github.com/pyinstaller/pyinstaller/pull/6616</a></p></blockquote><p><code>os.path.dirname(__file__)</code>返回的是当前脚本的所在路径，使用pycharm和直接点击运行py文件，这个路径均为脚本的所在路径，而生成exe之后点击运行，这个路径变为exe释放路径<code>C:Users...AppDataLocalTemp_MEI***</code>，所以log文件生成在这个路径下，在结束运行后，这个路径文件夹会被删除。</p><h2 id="冻结二进制pyinstaller"><a class="markdownIt-Anchor" href="#冻结二进制pyinstaller"></a> 冻结二进制——Pyinstaller</h2><blockquote><p>自我学Python以来推荐的就是这个，经过时光变迁，这个仍然是主流。</p><ul><li>PyInstaller是<strong>一个跨平台</strong>的Python应用打包工具，支持 Windows/Linux/MacOS三大主流平台，能够把 Python 脚本及其所在的 Python 解释器打包成可执行文件，从而允许最终用户在无需安装 Python 的情况下执行应用程序。</li><li>PyInstaller 制作出来的<strong>执行文件并不是跨平台的</strong>，如果需要为不同平台打包，就要在相应平台上运行PyInstaller进行打包。</li><li>PyInstaller打包的流程：读取编写好的Python项目–&gt;分析其中条用的模块和库，并收集其文件副本（包括Python的解释器）–&gt;将副本和Python项目文件（放在一个文件夹//封装在一个可执行文件）中。</li></ul></blockquote><p>安装就不细说了，主要讲用法：</p><ul><li><p>通过命令行命令打包<code>pyinstaller -F main.py</code>，常用参数</p><ul><li>-F： 表示生成单个可执行文件；对应的是-D： 生成文件夹形式的可执行程序（默认）</li><li>-w： 表示去掉控制台窗口，这在GUI界面时非常有用。不过如果是命令行程序的话那就把这个选项删除吧！</li><li>-p： 表示你自己自定义需要加载的库路径，一般情况下用不到</li><li>-i： 表示可执行文件的图标</li></ul></li><li><p>通过<code>.spec</code>打包定义文件来打包，对应命令行的参数，<code>.spec</code>文件都会有对应的生成内容。</p><blockquote><p>首先根据main文件生成spec文件: <code>pyi-makespec -D main.py</code>、填写好后再<code>pyinstaller main.spec</code></p></blockquote><ul><li>-D是让spec中多一个coll的实例，从而变成文件夹</li><li><code>-i</code>相当于<code>.spec</code>中EXE中<code>icon=&quot;.\\debugs\\favicon.ico&quot;</code></li><li>…(更多spec文件参数选择见：<a href="https://blog.csdn.net/tangfreeze/article/details/112240342" target="_blank" rel="noopener">https://blog.csdn.net/tangfreeze/article/details/112240342</a>)</li><li>-p: 相当于Analysis实例中的<code>pathex</code>，就是填入自己的模块</li><li>–hiden-import: 相当于Analysis实例中的hiddenimports</li></ul></li></ul><p>实际上，根据命令行的参数会生成对应的<code>.spec</code>文件。</p><p>注：可以看到无论是<code>pyinstaller</code>、<code>pyi-makespec</code>后面都是<code>main.py</code>，因为其是main函数入口文件</p><p>附：根据feiyu，在此列两个可行的：</p><ul><li><p><code>F:\aDevelopment\Python\ShowYourCode\env\Scripts\pyinstaller.exe main.py --add-data=&quot;feiyu\push_config.ini;.\feiyu&quot; --add-data=&quot;feiyu\user_config.toml;.\feiyu&quot; -i debugs\favicon.ico</code></p></li><li><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- mode: python ; coding: utf-8 -*-</span></span><br><span class="line">block_cipher = <span class="keyword">None</span> <span class="comment"># 此处在使用--key= 会有变化</span></span><br><span class="line"></span><br><span class="line">a = Analysis([<span class="string">'ai\\main.py'</span>],</span><br><span class="line">             pathex=[<span class="string">'C:\\Users\\Admin\\Downloads\\marsai-master'</span>],</span><br><span class="line">             binaries=[],</span><br><span class="line">             datas=[],<span class="comment"># 此处可以添加静态资源，例如你有个图片文件夹imgs，可以这样写[('imgs','imgs'),('test.txt','.')]，打包以后会有一个一样的文件夹，点表示当前文件夹。</span></span><br><span class="line">             hiddenimports=[],</span><br><span class="line">             hookspath=[],</span><br><span class="line">             runtime_hooks=[],</span><br><span class="line">             excludes=[],</span><br><span class="line">             win_no_prefer_redirects=<span class="keyword">False</span>,</span><br><span class="line">             win_private_assemblies=<span class="keyword">False</span>,</span><br><span class="line">             cipher=block_cipher,</span><br><span class="line">             noarchive=<span class="keyword">False</span>)</span><br><span class="line">pyz = PYZ(a.pure, a.zipped_data,</span><br><span class="line">             cipher=block_cipher)</span><br><span class="line">exe = EXE(pyz,</span><br><span class="line">          a.scripts,</span><br><span class="line">          [],</span><br><span class="line">          exclude_binaries=<span class="keyword">True</span>,</span><br><span class="line">          name=<span class="string">'main'</span>, <span class="comment"># 生成的exe的名字</span></span><br><span class="line">          debug=<span class="keyword">False</span>,</span><br><span class="line">          bootloader_ignore_signals=<span class="keyword">False</span>,</span><br><span class="line">          strip=<span class="keyword">False</span>,</span><br><span class="line">          upx=<span class="keyword">True</span>, <span class="comment"># 打包的时候进行压缩，False表示不压缩</span></span><br><span class="line">          console=<span class="keyword">True</span> <span class="comment"># 是否显示黑窗口，刚开始打包的时候一般都会有问题，建议设为True，解决所有问题后可以设置为False)</span></span><br><span class="line">coll = COLLECT(exe,</span><br><span class="line">               a.binaries,</span><br><span class="line">               a.zipfiles,</span><br><span class="line">               a.datas,</span><br><span class="line">               strip=<span class="keyword">False</span>,</span><br><span class="line">               upx=<span class="keyword">True</span>,</span><br><span class="line">               upx_exclude=[],</span><br><span class="line">               name=<span class="string">'main'</span> <span class="comment"># 文件夹的名字)</span></span><br></pre></td></tr></table></figure></li></ul><p>pyinstaller全都是控制台的命令，因此没那么直观，有人做了GUI相对直白、简单些，见<a href="https://www.jianshu.com/p/e33b4f0373bc" target="_blank" rel="noopener">Python打包利器：auto-py-to-exe</a>，其关于计算器程序与<code>Additional Files</code>的使用会让人对项目该如果布置结构和使用<code>-p</code>参数有个更确信的认识（导入自己些的模块）。</p><h2 id="functoolspartial用法"><a class="markdownIt-Anchor" href="#functoolspartial用法"></a> <a href="https://www.cnblogs.com/wxys/p/13756552.html" target="_blank" rel="noopener">functools.partial用法！</a></h2><blockquote><p><code>functools.partial</code> (偏函数)是用来包装一个函数的。并且针对的是函数的部分的参数。</p></blockquote><p>其主要作用有两个：</p><ul><li><p>在不修改别人代码的基础上, 给部分参数绑定值固定住，变成&quot;新&quot;的函数</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">show</span><span class="params">(name: str, age: int)</span>:</span></span><br><span class="line">print(<span class="string">f"your name is <span class="subst">&#123;name&#125;</span> and age is <span class="subst">&#123;age&#125;</span>"</span>)</span><br><span class="line"></span><br><span class="line">showWithA = functools.partial(show, <span class="string">"mrli"</span>)</span><br><span class="line">showWithA(<span class="number">22</span>)</span><br></pre></td></tr></table></figure><p>从而<code>showWithA</code>变成了只需要填写<code>age</code>参数，但拥有show相同执行逻辑的&quot;新&quot;函数了</p></li><li><p>给不方便传递参数的函数，加上默认参数 —— 比如<code>requests-html</code>中的<code>session.run</code></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">AsyncHTMLSession</span><span class="params">(BaseSession)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self, *coros)</span>:</span></span><br><span class="line">        <span class="string">""" Pass in all the coroutines you want to run, it will wrap each one</span></span><br><span class="line"><span class="string">            in a task, run it and wait for the result. Return a list with all</span></span><br><span class="line"><span class="string">            results, this is returned in the same order coros are passed in. """</span></span><br><span class="line">        tasks = [</span><br><span class="line">            asyncio.ensure_future(coro()) <span class="keyword">for</span> coro <span class="keyword">in</span> coros</span><br><span class="line">        ]</span><br><span class="line">        done, _ = self.loop.run_until_complete(asyncio.wait(tasks))</span><br><span class="line">        <span class="keyword">return</span> [t.result() <span class="keyword">for</span> t <span class="keyword">in</span> done]</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">getDataByChromeDriver</span><span class="params">(index: Union[int, str])</span>:</span></span><br><span class="line">    <span class="keyword">pass</span></span><br><span class="line"><span class="comment"># run方法中传的都是 asnyc def 函数(协程对象)</span></span><br><span class="line"><span class="comment"># session.run(asyncGetDataByChromeDriver, asyncGetUrls) 无法指定参数</span></span><br><span class="line">session.run(functools.partial(asyncGetDataByChromeDriver, <span class="number">1</span>))</span><br></pre></td></tr></table></figure><p>可见，默认使用的话，只能传递函数地址，并不能指定要执行异步函数的参数。此时便可以通过<code>functools.partial</code>来给这些参数绑定默认值，从而实现了指定参数</p><p>与此类似的还有处理回调函数: <a href="https://www.cnblogs.com/wxys/p/13756552.html%E3%80%82" target="_blank" rel="noopener">https://www.cnblogs.com/wxys/p/13756552.html。</a></p><p>⭐️本质上都是内部函数<code>func()</code>调用时，参数没办法在框架代码中指定，只能在传入函数地址的时候，将参数指定好，这样<code>func()</code>调用时，就是想要运行的参数了</p></li></ul><h2 id="拓展容器"><a class="markdownIt-Anchor" href="#拓展容器"></a> 拓展容器</h2><p><strong>defaultdict</strong></p><blockquote><p>defaultdict接受一个工厂函数作为参数，如下来构造：</p></blockquote><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">dict =defaultdict( factory_function)</span><br><span class="line"></span><br><span class="line">dict1 = defaultdict(int)</span><br><span class="line">dict2 = defaultdict(set)</span><br><span class="line">dict3 = defaultdict(str)</span><br><span class="line">dict4 = defaultdict(list)</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">1</span>, <span class="number">5</span>):</span><br><span class="line">    d = getattr(__import__(<span class="string">"__main__"</span>), <span class="string">f"dict<span class="subst">&#123;i&#125;</span>"</span>)</span><br><span class="line">    print(d[<span class="string">"hello"</span>])</span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">0</span></span><br><span class="line"><span class="string">set()</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">[]</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="comment"># 弹出hello的v, 如果没有, 则返回[]</span></span><br><span class="line">d.pop(<span class="string">"hello"</span>, [])</span><br></pre></td></tr></table></figure><p>其作用是在获得不存在的key时提供默认值，比如<code>dict1[&quot;hello&quot;]</code>返回的是<code>0</code></p><p><strong>heap</strong></p><h2 id="io多路复用"><a class="markdownIt-Anchor" href="#io多路复用"></a> IO多路复用</h2><blockquote><p>selectors库：python3.4版本中引进的，它封装了IO多路复用中的select和epoll，能够更快，更方便的实现多并发效果。</p><p>它的功能与linux的epoll，还是select模块,poll等类似；实现高效的I/O multiplexing, 常用于非阻塞的socket的编程中； 简单介绍一下这个模块，更多内容查看 python文档：<a href="https://docs.python.org/3/library/selectors.html" target="_blank" rel="noopener">https://docs.python.org/3/library/selectors.html</a></p></blockquote><p>模块定义了一个 BaseSelector的抽象基类， 以及它的子类，包括：SelectSelector， PollSelector, EpollSelector, DevpollSelector, KqueueSelector.  另外还有一个<strong>DefaultSelector</strong>类，它其实是以上其中一个子类的别名而已，它自动选择为当前环境中最有效的Selector，<u>所以平时用 DefaultSelector类就可以了，其它用不着</u>。</p><p>模块定义了两个常量，用于描述 event Mask</p><ul><li>EVENT_READ ：   表示可读的； 它的值其实是1；</li><li>EVENT_WRITE：   表示可写的； 它的值其实是2；</li></ul><h3 id="api"><a class="markdownIt-Anchor" href="#api"></a> API</h3><ul><li><p><code>register(fileobj, events, data=None)</code>  作用：注册一个文件对象。</p><p>参数：</p><ul><li>fileobj——即可以是fd 也可以是一个拥有fileno()方法的对象；</li><li>events——上面的event Mask 常量；</li><li>data: 绑定的data属性，可以是处理函数。在获得SelectorKey时，通过SelectorKey.data来获得</li></ul><p>返回值： 一个<strong>SelectorKey</strong>类的实例——一般用这个类的实例 来描述一个已经注册的文件对象的状态， 这个类的几个属性常用到：</p><ul><li>fileobj：  表示已经注册的文件对象；</li><li>fd:     表示文件对象的描述符，是一个整数，它是文件对象的 fileno()方法的返回值；</li><li>events:  表示注册一个文件对象时，我们等待的events, 即上面的event Mask, 是可读呢还是可写呢！！</li><li>data:   注册时我们传入的参数，可以是任意值，绑定到一个属性上，方便之后使用。</li></ul></li><li><p><code>unregister(fileobj)</code> 作用： 注销一个已经注册过的文件对象；</p><p>返回值：一个SelectorKey类的实例；</p></li><li><p><code>select(timeout=None)</code>作用： 用于选择满足我们监听的event的文件对象；</p><p>返回值： 是一个(key, events)的元组， 其中key是一个SelectorKey类的实例， 而events 就是 event Mask（EVENT_READ或EVENT_WRITE,或者二者的组合)</p></li><li><p><code>close</code>() 作用：关闭 selector。 最后一定要记得调用它， 要确保所有的资源被释放；</p></li><li><p><code>get_key(fileobj)</code>：</p><p>返回与已注册文件对象关联的密钥。这将返回与此文件对象关联的<a href="https://www.docs4dev.com/docs/zh/python/3.7.2rc1/all/library-selectors.html#selectors.SelectorKey" target="_blank" rel="noopener">SelectorKey</a>实例，如果未注册文件对象，则返回<a href="https://www.docs4dev.com/docs/zh/python/3.7.2rc1/all/library-exceptions.html#KeyError" target="_blank" rel="noopener">KeyError</a>。</p></li><li><p><code>get_map()</code></p><p>返回文件对象到 selectors 键的 Map。</p></li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> selectors</span><br><span class="line"><span class="keyword">import</span> socket</span><br><span class="line"><span class="comment"># selectors模块默认会用epoll，如果你的系统中没有epoll(比如windows)则会自动使用select</span></span><br><span class="line">sel = selectors.DefaultSelector()  <span class="comment"># 生成一个select对象</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">accept</span><span class="params">(sock, mask)</span>:</span></span><br><span class="line">    <span class="comment"># ②当有客户端进行sock.connect的时候会进入accept处理函数, 然后通过accept()获得连接信息</span></span><br><span class="line">    conn, addr = sock.accept()  <span class="comment"># Should be ready</span></span><br><span class="line">    print(<span class="string">'accepted'</span>, conn, <span class="string">'from'</span>, addr)</span><br><span class="line">    conn.setblocking(<span class="keyword">False</span>)  <span class="comment"># 设定非阻塞</span></span><br><span class="line">    <span class="comment"># ③将对于conn的读操作注册到时间循环中，绑定read函数为槽函数</span></span><br><span class="line">    sel.register(conn, selectors.EVENT_READ, read)  <span class="comment"># 新连接注册read回调函数</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">read</span><span class="params">(conn, mask)</span>:</span></span><br><span class="line">    <span class="comment"># ④当conn进行读操作时, 会进入read()函数，对其recv进行处理</span></span><br><span class="line">    data = conn.recv(<span class="number">1024</span>)  <span class="comment"># Should be ready</span></span><br><span class="line">    <span class="keyword">if</span> data:</span><br><span class="line">        <span class="comment"># ⑤如果有数据则说明在正常的通信, 则进行交互</span></span><br><span class="line">        print(<span class="string">'echoing'</span>, repr(data), <span class="string">'to'</span>, conn)</span><br><span class="line">        conn.send(data)</span><br><span class="line">    <span class="keyword">else</span>:</span><br><span class="line">        <span class="comment"># ⑤如果没有数据了, 则表示断开连接， 此时需要将conn的IO read事件卸载掉</span></span><br><span class="line">        print(<span class="string">'closing'</span>, conn)</span><br><span class="line">        sel.unregister(conn)</span><br><span class="line">        conn.close()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">sock = socket.socket()</span><br><span class="line">sock.bind((<span class="string">'localhost'</span>, <span class="number">8080</span>))</span><br><span class="line">sock.listen()</span><br><span class="line">sock.setblocking(<span class="keyword">False</span>)</span><br><span class="line">sel.register(sock, selectors.EVENT_READ, accept)  <span class="comment"># ①把刚生成的sock连接对象注册到select连接列表中，并交给accept函数处理</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">while</span> <span class="keyword">True</span>:</span><br><span class="line">    events = sel.select()  <span class="comment"># 默认是阻塞，有活动连接就返回活动的连接列表</span></span><br><span class="line">    <span class="comment"># 这里看起来是select，其实有可能会使用epoll，如果你的系统支持epoll，那么默认就是epoll</span></span><br><span class="line">    <span class="keyword">for</span> key, mask <span class="keyword">in</span> events:</span><br><span class="line">        print(<span class="string">"key, mask"</span>, key, mask)</span><br><span class="line">        callback = key.data  <span class="comment"># 去调accept函数</span></span><br><span class="line">        callback(key.fileobj, mask)  <span class="comment"># key.fileobj就是readable中的一个socket连接对象</span></span><br></pre></td></tr></table></figure><p><strong>Socket编程</strong></p><p>创建<a href="https://so.csdn.net/so/search?q=%E5%A5%97%E6%8E%A5%E5%AD%97&amp;spm=1001.2101.3001.7020" target="_blank" rel="noopener">套接字</a>的函数是socket()，函数原型为：<code>socket.socket(family=-1, type=-1, proto=-1, fileno=None)</code></p><ul><li>AF_UNIX（本机通信）</li><li>AF_INET（TCP/IP – IPv4）</li><li>AF_INET6（TCP/IP – IPv6）</li></ul><p>其中 “type”参数指的是套接字类型，常用的类型有：</p><ul><li>SOCK_STREAM（TCP流）</li><li>SOCK_DGRAM（UDP数据报）</li><li>SOCK_RAW（原始套接字）</li></ul><p>最后一个 <strong>“protocol”一般设置为“0”</strong>，也就是当确定套接字使用的协议簇和类型时，这个参数的值就为0，但是有时候创建原始套接字时，并不知道要使用的协议簇和类型，也就是domain参数未知情况下，这时protocol这个参数就起作用了，它可以确定协议的种类。</p><p>默认为<code>AF_INET、SOCK_STREAM、protocol=0</code></p><p><img src="/2022/03/27/Python进阶/E:%5Chexo%5Csource_posts%5CPython%E8%BF%9B%E9%98%B6%5Csocket.jpg" alt="socket"></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h2 id=&quot;xxxatrr家族&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#xxxatrr家族&quot;&gt;&lt;/a&gt; xxxatrr家族&lt;/h2&gt;
&lt;h3 id=&quot;hasattr&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#h
      
    
    </summary>
    
    
      <category term="Python" scheme="https://nymrli.top/tags/Python/"/>
    
  </entry>
  
  <entry>
    <title>JS逆向-webpack打包网站实战</title>
    <link href="https://nymrli.top/2022/03/23/JS%E9%80%86%E5%90%91-webpack%E6%89%93%E5%8C%85%E7%BD%91%E7%AB%99%E5%AE%9E%E6%88%98/"/>
    <id>https://nymrli.top/2022/03/23/JS逆向-webpack打包网站实战/</id>
    <published>2022-03-23T08:21:50.000Z</published>
    <updated>2022-05-29T09:34:08.556Z</updated>
    
    <content type="html"><![CDATA[<p>[TOC]</p><h1 id="webpack逆向"><a class="markdownIt-Anchor" href="#webpack逆向"></a> Webpack逆向</h1><p><a href="https://so.csdn.net/so/search?q=webpack&amp;spm=1001.2101.3001.7020" target="_blank" rel="noopener">webpack</a>打包是前端js模块化压缩打包常用的手段，特征明显，比如下方的形式的代码就是webpack<strong>分发器</strong></p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 分发器</span></span><br><span class="line">!<span class="function"><span class="keyword">function</span>(<span class="params">x</span>)</span>&#123;</span><br><span class="line">    <span class="function"><span class="keyword">function</span> <span class="title">xx</span>(<span class="params">n</span>)</span>&#123;</span><br><span class="line">        <span class="keyword">return</span> ..call(**.exports, ***, ***.exports, xx)</span><br><span class="line">    &#125;</span><br><span class="line">&#125;()</span><br></pre></td></tr></table></figure><p>又或者更直观的表现n[“xxx”]这种，你可以大概知道了这是调用了webpack打包的js模块代码。</p><p><strong>webpack打包后JS依赖模块代码的固定结构</strong>：</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">(<span class="keyword">this</span>[<span class="string">"webpackJsonpzsgk-pc"</span>] = <span class="keyword">this</span>[<span class="string">"webpackJsonpzsgk-pc"</span>] || []).push([[<span class="number">15</span>], [<span class="function"><span class="keyword">function</span>(<span class="params">e, t, n</span>) </span>&#123;</span><br><span class="line"><span class="meta">    "use strict"</span>;</span><br><span class="line">    e.exports = n(<span class="number">693</span>)</span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">// 参数固定为e, t, n</span></span><br><span class="line">, <span class="function"><span class="keyword">function</span>(<span class="params">e, t, n</span>) </span>&#123;</span><br><span class="line">    e.exports = n(<span class="number">697</span>)()</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>说个逆向webpack的通用方法：</p><ol><li><p>先去找加密网站的<strong>加密入口</strong>。这应该是加密网站都必须要做的==&gt; 直接根据参数名搜索参数</p></li><li><p>找到<strong>分发器的位置</strong>，或者说是加载器，n[“xxx”]这种的n就是分发器，就比如下方中的exports的位置，最后执行了d函数==&gt;一般是runtimexxx.js中(提供环境)；一般以<code>! function(e) {</code>的形式出现</p><p><img src="/2022/03/23/JS逆向-webpack打包网站实战/%E5%88%86%E5%8F%91%E5%99%A8.jpg" alt="分发器"></p></li><li><p>寻找分发编号、加密使用模块（用到了哪些模块就导入哪些模块）==&gt;一般在chunk-lib.js，以<code>(window.webpackJsonp = window.webpackJsonp || []).push([</code>的形式出现</p></li><li><p>将函数入口的地方返回全局变量，最终返回: <code>var sign; var window = global;!function(){... sign = d}</code>， 赋值为分发器返回的d</p></li><li><p>使用自定义的sign代替webpack代码中的n进行加密</p></li></ol><p>from: <a href="https://blog.csdn.net/weixin_41586984/article/details/116268341" target="_blank" rel="noopener">https://blog.csdn.net/weixin_41586984/article/details/116268341</a></p><h2 id="调试技巧"><a class="markdownIt-Anchor" href="#调试技巧"></a> 调试技巧</h2><h3 id="定位请求参数"><a class="markdownIt-Anchor" href="#定位请求参数"></a> 定位请求参数</h3><ol><li><p>打开开发者工具后，F5刷新后<code>Ctrl + Shift + F</code>搜索参数名，如signdata，会显示多个JS文件，选择后仔细查看(点击左下角<code>{}</code>美观格式化按钮)。</p><p>more： 如果文件太多，则直接通过请求的链接去找，比如<code>user/login</code></p></li><li><p>Network找到新发出的xhr条目后，查看Initiator里的调用栈信息，如Login；</p></li></ol><p>注： 如果加密参数名称比较简单如s，比较难定位的话，可以借助请求的其他参数来查找，比如verificationCode</p><h3 id="调试工具"><a class="markdownIt-Anchor" href="#调试工具"></a> 调试工具</h3><ul><li>断点调试breakpoints</li><li>XHR断点： XHR/fetch breakpoints</li></ul><h2 id="附录-js记录"><a class="markdownIt-Anchor" href="#附录-js记录"></a> 附录-Js记录</h2><ul><li><p>时间戳: <code>(new Date).getTime()</code></p></li><li><p><code>var a = (f1(), f2(), f3())</code>后，f1、f2、f3函数都会执行，而a最后的结果为f3的返回值</p></li><li><p>javascript:void(0):  <strong>void</strong> 是 JavaScript 中非常重要的关键字，该操作符指定要计算一个表达式但是不返回值。</p></li><li><p><code>TypeError: window.btoa is not a function</code></p><p>btoa-atob 模块没有输出一个编程接口，它只提供命令行工具。</p><p>如果你需要转换为Base64，你可以用Buffer来完成。</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">console</span>.log(Buffer.from(<span class="string">'Hello World!'</span>).toString(<span class="string">'base64'</span>));</span><br></pre></td></tr></table></figure><p>相反的，假设你要解码的内容是一个base64编码过的字符串。</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">console</span>.log(Buffer.from(b64Encoded, <span class="string">'base64'</span>).toString());</span><br></pre></td></tr></table></figure></li></ul><h1 id="做题记录"><a class="markdownIt-Anchor" href="#做题记录"></a> 做题记录</h1><h2 id="nstr题型"><a class="markdownIt-Anchor" href="#nstr题型"></a> n[“str”]题型：</h2><h3 id="天安财险"><a class="markdownIt-Anchor" href="#天安财险"></a> <a href="https://www.95505.com.cn/" target="_blank" rel="noopener">天安财险</a></h3><blockquote><p><code>var m = this.newEncrypt(JSON.stringify(h));</code></p></blockquote><ul><li><p>需要对this.privaKey的值细化下，传入拿到固定的str</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//           , p = t("NFKh")          , s = t("cg2h")</span></span><br><span class="line">l.prototype.newEncrypt = <span class="function"><span class="keyword">function</span>(<span class="params">l</span>) </span>&#123;</span><br><span class="line">                <span class="keyword">var</span> n = p.enc.Utf8.parse(<span class="keyword">this</span>.privaKey)</span><br><span class="line">                  , t = p.enc.Utf8.parse(<span class="keyword">this</span>.privaKey)</span><br><span class="line">                  , e = p.enc.Utf8.parse(l)</span><br><span class="line">                  , a = p.AES.encrypt(e, n, &#123;</span><br><span class="line">                    iv: t,</span><br><span class="line">                    mode: p.mode.CBC,</span><br><span class="line">                    padding: p.pad.Pkcs7</span><br><span class="line">                &#125;);</span><br><span class="line">                <span class="keyword">return</span> p.enc.Base64.stringify(a.ciphertext)</span><br><span class="line">            &#125;</span><br></pre></td></tr></table></figure></li></ul><h3 id="财新网"><a class="markdownIt-Anchor" href="#财新网"></a> <a href="https://www.caixin.com/" target="_blank" rel="noopener">财新网</a></h3><blockquote><p><code>password: this.encode(this.encrypt(this.form.password)),</code></p></blockquote><ul><li><code>c = a(&quot;3452&quot;)</code>、<code>n = a.n(c)</code>，看到需要依赖3452后立马Ctrl+shift+F全局搜3452，然后把整个webpack模块扒下来</li></ul><h3 id="中远海运"><a class="markdownIt-Anchor" href="#中远海运"></a> 中远海运</h3><ul><li>n(“MuMZ”)中又有<code>r = n(&quot;XBrZ&quot;);</code>，在另一个文件中，module需要放两个</li></ul><h3 id="天翼云"><a class="markdownIt-Anchor" href="#天翼云"></a> <a href="https://www.ctyun.cn/h5/auth/" target="_blank" rel="noopener">天翼云</a></h3><blockquote><p><code>var t = encodeURIComponent(c[&quot;c&quot;].Des.encrypt(this.form.email, this.form.pwd)),</code></p></blockquote><p>webpack实现</p><p><code>c = (mycode(&quot;ac6a&quot;), mycode(&quot;b3ae&quot;))</code></p><ul><li>分发器和ac6a模块在同一个文件中、而ac6a模块依赖模块在另一个文件内；</li><li>分发器()({})，无感叹号</li><li>分发器()({})大括号中自带较多依赖模块</li></ul><p>自己实现：直接扒下来encrypt加密的JS内容</p><h3 id="看准网"><a class="markdownIt-Anchor" href="#看准网"></a> <a href="https://www.kanzhun.com/" target="_blank" rel="noopener">看准网</a></h3><ul><li>分发器n(“xxx”)定位后跟一般的固定格式返回a、n、r不同，写的是个函数==&gt;还是可以根据obj.Func来赋值<code>mycode = obj;</code></li><li><a href="https://www.bilibili.com/video/BV1bb4y127bc?spm_id_from=333.999.0.0" target="_blank" rel="noopener">JS逆向实战分析–看准网webpack加解密分析</a>——Python中使用execjs示范</li></ul><h3 id="企名片"><a class="markdownIt-Anchor" href="#企名片"></a> <a href="https://www.qimingpian.cn/finosda/project/pinvestment" target="_blank" rel="noopener">企名片</a></h3><blockquote><ul><li><code>u = i(&quot;x4Ab&quot;)</code></li><li><code>return e.encrypt_data &amp;&amp; (e.data = Object(u.a)(e.encrypt_data)),</code></li></ul></blockquote><ul><li><p>x4Ab模块依赖aqBw，aqBw又依赖YuTi、yLpj，因此依赖项中放&quot;x4Ab&quot;、“aqBw”、“YuTi”、&quot;yLpj&quot;函数定义</p></li><li><p>模拟解析函数</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">encrypt</span>(<span class="params">data</span>)</span>&#123;</span><br><span class="line">    <span class="keyword">return</span> data &amp;&amp; (<span class="built_in">Object</span>(u.a)(data))</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li></ul><h2 id="nnum题型"><a class="markdownIt-Anchor" href="#nnum题型"></a> n[num]题型：</h2><h3 id="大麦网"><a class="markdownIt-Anchor" href="#大麦网"></a> <a href="https://www.damai.cn/" target="_blank" rel="noopener">大麦网</a></h3><ul><li>删除分发器多余代码</li><li><code>var navigator = {}</code></li></ul><h3 id="掌上高考"><a class="markdownIt-Anchor" href="#掌上高考"></a> <a href="https://www.gaokao.cn/school/263/provinceline" target="_blank" rel="noopener">掌上高考</a>：</h3><ol><li>分发器在html文件内</li><li><code>o = (u=a(42), a.n(u))</code>，使用到了<code>a.n(u)</code>即点n函数</li><li>依赖函数的给出是以数组的形式，而不是字典的形式</li><li>模块中依赖更多模块==&gt;引入整个模块文件，但是跟&quot;xxx&quot;模式不同的是，由于没有用字典<code>{&quot;xxx&quot;: function()}</code>的形式，因此直接require也没用<code>TypeError: Cannot read property '42' of undefined</code>，而是将依赖模块数组作为参数写入到分发器依赖函数中<code>!function(e){}([...])</code>即方括号中，从而才能找到42函数</li></ol><h3 id="酷我"><a class="markdownIt-Anchor" href="#酷我"></a> <a href="https://www.bilibili.com/video/BV1gq4y1D781?from=search&amp;seid=7720105602891609746&amp;spm_id_from=333.337.0.0" target="_blank" rel="noopener">酷我</a></h3><blockquote><p><code>t.data.reqId = n,</code></p></blockquote><ul><li><p>直接通过n(109)定位可能不那么准确(双击后定位的函数)，可以试着直接在分发器位置进行断点，然后console输出e[“109”]</p></li><li><p>只要分发器定义部分(其他的删了，因为只用了<code>l=n(109)、c=n.n(l)</code>)+依赖模块中定义109函数(整个function而不是t.exports)，以及观察其中还依赖什么如n(202)、n(203)就补充拿什么</p><ul><li>n.n(l)是传入什么就返回什么:<a href="https://www.bilibili.com/video/BV1gq4y1D781?from=search&amp;seid=7720105602891609746&amp;spm_id_from=333.337.0.0" target="_blank" rel="noopener">https://www.bilibili.com/video/BV1gq4y1D781?from=search&amp;seid=7720105602891609746&amp;spm_id_from=333.337.0.0</a></li></ul><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 如果不删分发器中其他部分</span></span><br><span class="line">l = mycode(<span class="number">109</span>)</span><br><span class="line">c = mycode.n(l)<span class="comment">// ==&gt;得到l</span></span><br><span class="line"><span class="keyword">var</span> r = c()();</span><br><span class="line"><span class="built_in">console</span>.log(r)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 由于只用到了n.n(l)，所以可以删分发器大代码中其他部分, 在使用时直接让c=l</span></span><br><span class="line">l = mycode(<span class="number">109</span>)</span><br><span class="line">c = l();</span><br><span class="line"><span class="built_in">console</span>.log(c)<span class="comment">// 等价于 r = l(); console(r)</span></span><br></pre></td></tr></table></figure></li></ul><p>文章: <a href="https://blog.csdn.net/weixin_43189702/article/details/119860838" target="_blank" rel="noopener">https://blog.csdn.net/weixin_43189702/article/details/119860838</a></p><h2 id="注意"><a class="markdownIt-Anchor" href="#注意"></a> 注意：</h2><ul><li><p>require模块内容可以放在逆向JS文件里一起，而不是一定得创建新的JS文件导入</p></li><li><p>先登录然后找到加密处加断点，这个断点会在发起登录请求时才触发；往上找分发器，加上断点，分发器位置的断点是在页面刷新时触发，因此要触发这个断点需要刷新页面</p></li><li><p>找加密函数<code>c[&quot;c&quot;].Des.encrypt(this.form.email, this.form.pwd)</code>的时候，找完整的函数如c[“c”].Des.encrypt，而不是直接找c</p></li><li><p>n(42), 或者n(“xxx”)，可以直接搜xxx，也可以在console里面输出后找到对应的FunctionLocation来快速定位</p></li><li><p>如果依赖模块是字典的形式，则分发器依赖中写字典，如<code>!function(e){..}({ 32:function(){...}})</code>（一般情况<code>n(32)、n(&quot;ABCD&quot;)</code>）， 如果不是则需要传函数数组，如<code>n(42)</code>，此处42表示的是第42个函数，见掌上高考。</p></li><li><p><code>var mycode;</code>后赋值的位置直接在分发器的下方即可，不用在最后面</p></li><li><p>提示缺少<code>window</code>时，定义全局变量<code>var window = global;</code>，（JS逆向文件、依赖文件）</p><ul><li><p>window表示<strong>浏览器</strong>打开的窗口，在客户端JavaScript中window对象是全局的对象，所有 JavaScript 全局对象、函数以及变量均自动成为 window 对象的成员。但在nodejs中直接调用window是不存在的，而代替的是global，所以要用nodejs运行时，得用<code>var window = global;</code></p></li><li><p><code>var navigator = this</code>等价于<code>var navigator = {}</code>，因为在NodeJS文件中运行输出this后可以发现<code>this={}</code>，而在浏览器中this默认为window（函数或类作用域内为函数或者类实例）</p></li><li><p>from：<a href="https://blog.csdn.net/mystric594/article/details/72519788" target="_blank" rel="noopener">JS中document和window的区别</a></p></li></ul></li></ul><p><strong>心得</strong></p><p>①所有webpack打包的的js都要先看懂打包后<strong>代码运行</strong>的顺序，找到加密处；②找到webpack对象，一般是 n(数字) 调用③确定分发器。④找依赖模块，有时候各包的依赖关系太多，可以直接把文件爬下来引入，如果各个包的依赖关系不多，就可以只把调用到的函数找出来放到依赖中。⑤最后剩下的就是找到你要的代码，慢慢复现调用加密/解密函数就好了。</p><h2 id="做题案例学习视频"><a class="markdownIt-Anchor" href="#做题案例学习视频"></a> 做题案例学习视频</h2><ul><li>webpack师承：<a href="https://space.bilibili.com/453818250/video?tid=0&amp;page=1&amp;keyword=&amp;order=pubdate" target="_blank" rel="noopener">爬取webpack流程-视频</a>——大多都是n(“Xvmd”)</li><li>而如果n中调用的不是字典的形式，而是列表的形式，则看<a href="https://www.cnblogs.com/Eeyhan/p/15562632.html" target="_blank" rel="noopener">js逆向安全指南（3）-- webpack解包指南</a>、<a href="https://www.bilibili.com/video/BV1gq4y1D781?from=search&amp;seid=7720105602891609746&amp;spm_id_from=333.337.0.0" target="_blank" rel="noopener">使用 webpack 的 js 加密参数的分析——酷我</a></li></ul><h2 id="进阶资料"><a class="markdownIt-Anchor" href="#进阶资料"></a> 进阶资料</h2><ul><li><a href="https://www.cnblogs.com/borntodie/p/15042669.html" target="_blank" rel="noopener">9-爬虫高级实战【js逆向】 </a></li><li><a href="https://blog.csdn.net/Qiled/article/details/109610726" target="_blank" rel="noopener">JS逆向学习笔记 - 持续更新中</a></li><li>webpack补充依赖项做法：<ul><li><a href="https://www.jianshu.com/p/5c42730a4e84" target="_blank" rel="noopener">https://www.jianshu.com/p/5c42730a4e84</a></li><li><a href="https://zhuanlan.zhihu.com/p/79706247" target="_blank" rel="noopener">webpack4之《模块运行机制原理》</a></li></ul></li><li>JS逆向视频<ul><li><a href="https://www.bilibili.com/video/BV1wp4y1z7L6?p=42" target="_blank" rel="noopener">爬虫工程师进阶必会之JS逆向(反爬)</a></li><li><a href="https://www.bilibili.com/video/BV1XR4y147fQ?p=85" target="_blank" rel="noopener">遇到网站加密爬不了？？这几个视频教你如何学会爬虫高阶内容js逆向，入狱必学！</a></li></ul></li></ul><h2 id="掌上高考解密过程"><a class="markdownIt-Anchor" href="#掌上高考解密过程"></a> 掌上高考解密过程</h2><h3 id="解析响应datatext"><a class="markdownIt-Anchor" href="#解析响应datatext"></a> 解析响应data.text</h3><blockquote><p>相应的data.text是加密的，页面通过JS解密后渲染</p></blockquote><ol><li><p>确定加密位置</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">return</span> <span class="literal">null</span> != l &amp;&amp; <span class="literal">null</span> !== (a = l.data) &amp;&amp; <span class="keyword">void</span> <span class="number">0</span> !== a &amp;&amp; a.text &amp;&amp; (l.data = (n = (e = &#123;</span><br><span class="line">            iv: u.uri,</span><br><span class="line">            text: l.data.text,</span><br><span class="line">            SIGN: h</span><br><span class="line">        &#125;).iv,</span><br></pre></td></tr></table></figure></li><li><p>确定分发器位置，在html内</p><p>通过打断<code>o = (u=a(42), a.n(u))// 等价于 o = a(42)</code></p></li><li><p>确定依赖模块：给<code>return e[a].call(c.exports, c, c.exports, r),</code>打断点后console输出<code>e[&quot;42&quot;]</code>查看a(42)位置：</p></li></ol><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span>(<span class="params">e, t, n</span>) </span>&#123;</span><br><span class="line">    e.exports = (e = n(<span class="number">21</span>),</span><br><span class="line">    n(<span class="number">201</span>),</span><br><span class="line">    n(<span class="number">825</span>),</span><br><span class="line">    ...</span><br><span class="line">    n(<span class="number">847</span>),</span><br><span class="line">    e)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>可以看到需要依赖多个，因此直接把整个文件引入</p><ol start="4"><li>扣解密函数：注意<code>return</code>表达式后是逗号的情况：会从左到右执行执行，并返回最后一个。注意：JS函数并<strong>不能返回多个返回值</strong></li></ol><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line">then((<span class="function"><span class="keyword">function</span>(<span class="params">l</span>) </span>&#123;</span><br><span class="line"><span class="keyword">var</span> e, a, t, b, n;</span><br><span class="line"><span class="keyword">return</span> <span class="literal">null</span> != l &amp;&amp; <span class="literal">null</span> !== (a = l.data) &amp;&amp; <span class="keyword">void</span> <span class="number">0</span> !== a &amp;&amp; a.text &amp;&amp; (l.data = (n = (e = &#123;</span><br><span class="line">            iv: u.uri,</span><br><span class="line">            text: l.data.text,</span><br><span class="line">            SIGN: h</span><br><span class="line">        &#125;).iv,</span><br><span class="line">        a = e.text,</span><br><span class="line">        e = e.SIGN,</span><br><span class="line">        e = o.a.PBKDF2(e, <span class="string">"secret"</span>, &#123;</span><br><span class="line">            keySize: <span class="number">8</span>,</span><br><span class="line">            iterations: <span class="number">1e3</span>,</span><br><span class="line">            hasher: o.a.algo.SHA256</span><br><span class="line">        &#125;).toString(),</span><br><span class="line">        n = o.a.PBKDF2(n, <span class="string">"secret"</span>, &#123;</span><br><span class="line">            keySize: <span class="number">4</span>,</span><br><span class="line">            iterations: <span class="number">1e3</span>,</span><br><span class="line">            hasher: o.a.algo.SHA256</span><br><span class="line">        &#125;).toString(),</span><br><span class="line">        a = o.a.lib.CipherParams.create(&#123;</span><br><span class="line">            ciphertext: o.a.enc.Hex.parse(a)</span><br><span class="line">        &#125;),</span><br><span class="line">        n = o.a.AES.decrypt(a, o.a.enc.Hex.parse(e), &#123;</span><br><span class="line">            iv: o.a.enc.Hex.parse(n)</span><br><span class="line">        &#125;),</span><br><span class="line"><span class="comment">// data.text解析结果</span></span><br><span class="line">        <span class="built_in">JSON</span>.parse(n.toString(o.a.enc.Utf8)))),</span><br><span class="line">    v &amp;&amp; (t = r,</span><br><span class="line">        b = l,</span><br><span class="line">        <span class="literal">null</span> !== (n = <span class="built_in">window</span>.apiConfig) &amp;&amp; <span class="keyword">void</span> <span class="number">0</span> !== n &amp;&amp; <span class="literal">null</span> !== (n = n.filterCacheList) &amp;&amp; <span class="keyword">void</span> <span class="number">0</span> !== n &amp;&amp; n.length ? <span class="built_in">window</span>.apiConfig.filterCacheList.forEach((<span class="function"><span class="keyword">function</span>(<span class="params">l</span>) </span>&#123;</span><br><span class="line">            <span class="keyword">new</span> <span class="built_in">RegExp</span>(l).test(t) || d.set(t, b)</span><br><span class="line">        &#125;)) : d.set(t, b)),</span><br><span class="line">    l</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>难点：</p><ul><li><p>跟&quot;xxx&quot;模式不同的是，由于没有用字典<code>{&quot;xxx&quot;: function()}</code>的形式，因此直接require也没用<code>TypeError: Cannot read property '42' of undefined</code>，而是将依赖模块函数数组作为参数写入到分发器依赖函数中<code>!function(e){}([...])</code>即方括号中，从而才能找到42函数</p><ul><li><p>挑选push后第二个<code>[]</code>中的函数数组</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">(<span class="keyword">this</span>[<span class="string">"webpackJsonpzsgk-pc"</span>] = <span class="keyword">this</span>[<span class="string">"webpackJsonpzsgk-pc"</span>] || []).push([[<span class="number">15</span>], [<span class="function"><span class="keyword">function</span>(<span class="params">e, t, n</span>) </span>&#123;</span><br><span class="line"><span class="meta">    "use strict"</span>;</span><br><span class="line">    e.exports = n(<span class="number">693</span>)</span><br><span class="line">&#125;,</span><br><span class="line">  ...</span><br><span class="line">  &#125;</span><br><span class="line">]]);<span class="comment">// 第一个]</span></span><br></pre></td></tr></table></figure></li></ul></li><li><p>理解了a.n的含义后，可以直接把<code>o = (u=a(42), a.n(u))</code>转化为<code>o=a(42)</code></p></li></ul><h3 id="获得加密参数signsafe"><a class="markdownIt-Anchor" href="#获得加密参数signsafe"></a> 获得加密参数signsafe</h3><blockquote><p>大致流程跟data.text差不多，但是<code>p = c()(g)</code>执行时，会报错</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">&gt; Md5.prototype.update = <span class="function"><span class="keyword">function</span>(<span class="params">e</span>) </span>&#123;</span><br><span class="line">&gt;     <span class="keyword">if</span> (!<span class="keyword">this</span>.finalized) &#123;</span><br><span class="line">&gt;         <span class="keyword">var</span> t, n = <span class="keyword">typeof</span> e;</span><br><span class="line">&gt;         <span class="keyword">if</span> (<span class="string">"string"</span> != n) &#123;</span><br><span class="line">&gt;             <span class="keyword">if</span> (<span class="string">"object"</span> != n)</span><br><span class="line">&gt;                 <span class="keyword">throw</span> ERROR;</span><br><span class="line">&gt;             <span class="keyword">if</span> (<span class="literal">null</span> === e)</span><br><span class="line">&gt;</span><br></pre></td></tr></table></figure></blockquote><blockquote><p>根据一步步调试之后发现，还是<code>c = (u=a(291),a.n(u))</code>直接替换出的问题</p></blockquote><ol><li><p>Ctrl + shift + F定位参数</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">g = <span class="keyword">void</span> <span class="number">0</span>,</span><br><span class="line">    g = (t = &#123;</span><br><span class="line">    SIGN: h,</span><br><span class="line">    str: f.replace(<span class="regexp">/^\/|https?:\/\/\/?/</span>, <span class="string">""</span>)</span><br><span class="line">&#125;).SIGN,</span><br><span class="line">    t = t.str,</span><br><span class="line">    g = o.a.HmacSHA1(o.a.enc.Utf8.parse(t), g),</span><br><span class="line">    g = o.a.enc.Base64.stringify(g).toString(),</span><br><span class="line">    p = c()(g),</span><br><span class="line">    u.signsafe = p,</span><br></pre></td></tr></table></figure></li><li><p>往上找c和o.a: <code>o = (u = a(42),a.n(u))</code>, <code>c = (u = a(291),a.n(u))</code></p></li><li><p>找到分发器扣出==&gt;这次不能删除分发器中多余的函数，比如r.a、r.d、r.n因为后面得用</p></li><li><p>将依赖模块跟data.text一样，放入分发器依赖模块中</p></li><li><p>扣加密函数</p><blockquote><p>网页上是return后多段内容，以及g变量不断被修改，因此通过一步步调试确定入参，以及分解return抽离出真正的加密参数signsafe</p></blockquote><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line">h = <span class="string">"D23ABC@#56"</span></span><br><span class="line"><span class="keyword">var</span> o = &#123;&#125;</span><br><span class="line"></span><br><span class="line">o.a = mycode(<span class="number">42</span>)</span><br><span class="line"><span class="comment">// ▲</span></span><br><span class="line">c = (u = mycode(<span class="number">291</span>), mycode.n(u))</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 对url进行加密</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">encrypt</span>(<span class="params">f</span>) </span>&#123;</span><br><span class="line">    g = (t = &#123;</span><br><span class="line">            SIGN: h,</span><br><span class="line">            str: f.replace(<span class="regexp">/^\/|https?:\/\/\/?/</span>, <span class="string">""</span>)</span><br><span class="line">        &#125;).SIGN,</span><br><span class="line">        t = t.str;</span><br><span class="line">    <span class="comment">// console.log(t, g);</span></span><br><span class="line">    g = o.a.HmacSHA1(o.a.enc.Utf8.parse(t), g);</span><br><span class="line">    <span class="comment">// console.log(g)</span></span><br><span class="line">    g = o.a.enc.Base64.stringify(g).toString();</span><br><span class="line">    <span class="comment">// console.log(g)</span></span><br><span class="line">    p = c()(g);</span><br><span class="line">    <span class="keyword">return</span> p;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">res = encrypt(<span class="string">"https://api.eol.cn/web/api/counter?cid=1&amp;did=263"</span>)</span><br><span class="line"><span class="built_in">console</span>.log(res)</span><br></pre></td></tr></table></figure></li></ol><p>可以看到o.a和c的赋值是不一样的，虽然说大多数情况x = a.n(u)等价于x=u，但难免有时会有不一样，因此谨慎期间，还是还原到底最好。</p><h4 id="python调用"><a class="markdownIt-Anchor" href="#python调用"></a> Python调用</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> execjs</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_signsafe_by_javascript</span><span class="params">(url)</span>:</span></span><br><span class="line">    <span class="comment"># 两个 JavaScript 脚本，两种方法均可</span></span><br><span class="line">    <span class="keyword">with</span> open(<span class="string">'gk_signsafe.js'</span>, <span class="string">'r'</span>, encoding=<span class="string">'utf-8'</span>) <span class="keyword">as</span> f:</span><br><span class="line">        exec_js = f.read()</span><br><span class="line">    signsafe = execjs.compile(exec_js).call(<span class="string">'encrypt'</span>, url)</span><br><span class="line">    <span class="keyword">return</span> signsafe</span><br><span class="line">signsafe = get_encrypted_password_by_javascript(<span class="string">"https://api.eol.cn/web/api/counter?cid=1&amp;did=263"</span>)</span><br><span class="line">print(signsafe)</span><br></pre></td></tr></table></figure><h2 id="rsa的加密步骤"><a class="markdownIt-Anchor" href="#rsa的加密步骤"></a> RSA的加密步骤</h2><ol><li>获取公钥</li><li>实例化  ===&gt; 扣出网站RSA实例化对象的代码</li><li>设置公钥</li><li>对文本进行加密 ==&gt; 扣出复现RSA加密的逻辑代码</li></ol><p>注： <code>var window=global</code>、<code>var navigator={}</code></p><ul><li><p><a href="https://www.bilibili.com/video/BV1jB4y1K7uw?spm_id_from=333.999.0.0" target="_blank" rel="noopener">B站RSA</a>：</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// window.JSEncrypt is not a constructor 在抠出来的JS Encrypt代码中加上</span></span><br><span class="line"><span class="built_in">window</span>.JSEncrpt = ze</span><br><span class="line"><span class="comment">// 网页中位var n = new JSEncrypt ==&gt; JSEncrypt is not defined</span></span><br><span class="line"><span class="keyword">var</span> n = <span class="keyword">new</span> <span class="built_in">window</span>.JSEncrypt;</span><br><span class="line">n.setPublicKey...</span><br><span class="line"><span class="keyword">var</span> a = n.encrypt(t.data.hash + password);</span><br><span class="line"><span class="built_in">console</span>.log(a)</span><br></pre></td></tr></table></figure></li><li><p><a href="https://www.bilibili.com/video/BV1Qh411B77h?spm_id_from=333.999.0.0" target="_blank" rel="noopener">网易云爬评论</a>：python通过execjs来调用JS代码，代码中用到了<code>CryptoJS</code>库， 需要<code>os.environ[&quot;NODE_PATH&quot;]=&quot;F:/..../node_modules&quot;</code>把库导入</p></li><li><p><a href="https://www.bilibili.com/video/BV1eQ4y1Q7RE?spm_id_from=333.999.0.0" target="_blank" rel="noopener">JS逆向实战分析–某铁网分析</a>：document返回类型，initiator是一条条文本(Other)，因为其没有用ajax(XHR)，而是通过原生的网页表单提交</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">loginForm.password.value = encryptByDES(loginForm.password.value), loginForm.publickey.value);</span><br><span class="line">loginForm.submit();</span><br></pre></td></tr></table></figure><ul><li><p>直接require导入CryptoJS模块</p></li><li><p>or直接扣encryptByDES的加密函数==&gt;出现<code>cannot read property 'createEncryptor' of undefined</code></p></li><li><p>MD5加密：</p><ul><li><p>JS: <code>const crypto = require(&quot;CryptoJS&quot;); crypto.MD5('待加密字符串').toString()</code></p></li><li><p>Python: <a href="https://blog.csdn.net/weixin_44799217/article/details/112486097" target="_blank" rel="noopener">https://blog.csdn.net/weixin_44799217/article/details/112486097</a></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 法一：创建md5对象</span></span><br><span class="line">hl = hashlib.md5()</span><br><span class="line"><span class="comment"># Tips</span></span><br><span class="line"><span class="comment"># 此处必须声明encode，若写法为hl.update(str) 报错为： Unicode-objects must be encoded before hashing</span></span><br><span class="line">hl.update(str.encode(encoding=<span class="string">'utf-8'</span>))</span><br><span class="line"></span><br><span class="line"><span class="comment"># 法二：</span></span><br><span class="line">str_md5 = hashlib.md5(str.encode(encoding=<span class="string">'utf-8'</span>)).hexdigest()</span><br></pre></td></tr></table></figure></li></ul></li><li><p>base64编码</p><ul><li><p>JS: <code>CryptoJS.enc.Base64.parse(&quot;待解密字符串&quot;).toString(CryptoJS.enc.Utf8)</code></p></li><li><p>Python</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 字符串</span></span><br><span class="line">encode_str = base64.encodebytes(test_str.encode(<span class="string">'utf8'</span>))  <span class="comment"># b'aGVsbG8gd29ybGQh\n'</span></span><br><span class="line">print(encode_str.decode())  <span class="comment"># 默认以utf8解码，结果 aGVsbG8gd29ybGQh</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 图片</span></span><br><span class="line"><span class="keyword">with</span> open(<span class="string">"D:\\redis.png"</span>, <span class="string">'rb'</span>) <span class="keyword">as</span> f:</span><br><span class="line">    encode_img = base64.b64encode(f.read())</span><br><span class="line">    file_ext = os.path.splitext(<span class="string">"D:\\redis.png"</span>)[<span class="number">1</span>]</span><br><span class="line">    print(<span class="string">'data:image/&#123;&#125;;base64,&#123;&#125;'</span>.format(file_ext[<span class="number">1</span>:], encode_img.decode()))</span><br><span class="line">    f.close()</span><br></pre></td></tr></table></figure></li></ul></li></ul></li></ul><h1 id="加密-摘要算法结果特征"><a class="markdownIt-Anchor" href="#加密-摘要算法结果特征"></a> 加密、摘要算法结果特征</h1><h2 id="urlencode"><a class="markdownIt-Anchor" href="#urlencode"></a> urlencode</h2><blockquote><p>urlencode是一个函数，可将字符串以<a href="https://baike.baidu.com/item/URL%E7%BC%96%E7%A0%81/3703727" target="_blank" rel="noopener">URL编码</a>，用于编码处理。</p><p>URL编码(URL encoding)，也称作<strong>百分号编码</strong>(Percent-encoding)， 是特定上下文的<a href="https://baike.baidu.com/item/%E7%BB%9F%E4%B8%80%E8%B5%84%E6%BA%90%E5%AE%9A%E4%BD%8D%E7%AC%A6/4438100" target="_blank" rel="noopener">统一资源定位符</a> (URL)的编码机制。</p></blockquote><h2 id="base64特征"><a class="markdownIt-Anchor" href="#base64特征"></a> Base64特征</h2><blockquote><p>最常见的用于传输8Bit<a href="https://baike.baidu.com/item/%E5%AD%97%E8%8A%82%E7%A0%81/9953683" target="_blank" rel="noopener">字节码</a>的<strong>编码方式</strong>之一</p></blockquote><ul><li>相同内容，结果是相同的</li><li>a-zA-Z,0-9,+/共64个字符进行编码；每3个字节编码成4个字节，不足的在<strong>结尾</strong>有无意义的**=**来填补<ul><li>一般情况下结尾都会有1个或者2个等号，明文长度是3的倍数时没有=；</li></ul></li><li>内容越长，结果越长</li></ul><p>注：跟下面的算法区分一下，base64是编码方式，并不能算加密算法。应用场景还有传输图片:<code>data:image/jpg;base64,/9j/4QMZRXhpZgAASUk...</code></p><h2 id="md5特征"><a class="markdownIt-Anchor" href="#md5特征"></a> md5特征</h2><blockquote><p><strong>消息摘要算法</strong></p></blockquote><ul><li>确定唯一性：相同内容，结果是相同的；但一般会有时间戳等参数，所以导致了每次不同</li><li>不可逆性：有损的加密过程，理论上无法解密（逆向推出），除非暴力破解。安全，这也是其成为校验是否被修改的最关键的性质</li><li>碰撞性：原始数据与其MD5值并不是一一对应的，有可能多个原始数据计算出来的MD5值是一样的，这就是碰撞。</li><li>一般MD5值是32位，由数字“0-9”和字母“a-f”所组成的字符串；字母可以是全大写或者全小写<ul><li>密文一般为 16 位或者 32 位，其中 16 位是取的 32 位第 9~25 位的值；</li></ul></li><li>长度：32个<strong>十六进制</strong>字符组成的字符串 (128位)</li></ul><h2 id="rsa特征"><a class="markdownIt-Anchor" href="#rsa特征"></a> RSA特征</h2><ul><li>相同内容，结果也是不同的</li><li>明文长度需要小于密钥长度，而密文长度则等于密钥长度。一般为1024、2048、3072、4096或512（低于1024的安全不建议）</li><li>通过公钥加密结果，必须私钥解密。 同样私钥加密结果，公钥可以解密</li></ul><p>注：RSA加解密中必须考虑到的密钥长度、明文长度和密文长度问题；</p><p>▲.一般会使用 JSEncrypt 库，会有 new 一个实例对象的操作；</p><h2 id="sha-系列"><a class="markdownIt-Anchor" href="#sha-系列"></a> SHA 系列</h2><blockquote><p>SHA 是比 MD5 更安全一点的摘要算法，SHA 通常指 SHA 家族算法，</p></blockquote><p><strong>sha1</strong></p><blockquote><p>字母（a-f）和数字（0-9）混合</p><p>密文特征跟MD5差不多，只不过数字是40位，bit位数（160）==&gt;4位<strong>十六进制</strong>表示一个数</p></blockquote><p><strong>Sha256</strong></p><blockquote><p>字母（a-f）和数字（0-9）混合</p><p>对于任意长度的消息，SHA256都会产生一个256位的哈希值，即64位十六进制数，称作消息摘要。</p></blockquote><p><strong>HMAC</strong></p><blockquote><p>在md5和sha1加密的基础上引入了秘钥，而秘钥又只有传输双方才知道，所以基本上是破解不了的，常用于接口签名验证</p></blockquote><h2 id="aes-des-3des-rc4-rabbit-等"><a class="markdownIt-Anchor" href="#aes-des-3des-rc4-rabbit-等"></a> AES、DES、3DES、RC4、Rabbit 等</h2><p>AES、DES、3DES、RC4、Rabbit 等加密算法的密文通常没有固定的长度，他们通常使用<code>crypto-js</code>库来实现</p><p>参考：<a href="https://juejin.cn/post/7052978567390429215" target="_blank" rel="noopener">https://juejin.cn/post/7052978567390429215</a></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;[TOC]&lt;/p&gt;
&lt;h1 id=&quot;webpack逆向&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#webpack逆向&quot;&gt;&lt;/a&gt; Webpack逆向&lt;/h1&gt;
&lt;p&gt;&lt;a href=&quot;https://so.csdn.net/so/search
      
    
    </summary>
    
    
      <category term="爬虫" scheme="https://nymrli.top/tags/%E7%88%AC%E8%99%AB/"/>
    
  </entry>
  
  <entry>
    <title>深入学习使用Spring</title>
    <link href="https://nymrli.top/2022/02/27/%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0%E4%BD%BF%E7%94%A8Spring/"/>
    <id>https://nymrli.top/2022/02/27/深入学习使用Spring/</id>
    <published>2022-02-27T05:52:08.000Z</published>
    <updated>2022-02-27T05:54:23.566Z</updated>
    
    <content type="html"><![CDATA[<blockquote><p>From : <a href="https://www.bilibili.com/video/BV1fr4y1Y7UH?p=6" target="_blank" rel="noopener">https://www.bilibili.com/video/BV1fr4y1Y7UH?p=6</a></p></blockquote><h2 id="创建bean的方式"><a class="markdownIt-Anchor" href="#创建bean的方式"></a> 创建bean的方式：</h2><h3 id="声明式"><a class="markdownIt-Anchor" href="#声明式"></a> 声明式</h3><h4 id="beanbean"><a class="markdownIt-Anchor" href="#beanbean"></a> <code>&lt;bean&gt;&lt;/bean&gt;</code></h4><ul><li><p><code>&lt;bean&gt;&lt;/bean&gt;</code>： 通过xml来配置bean</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="php"><span class="meta">&lt;?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"UTF-8"</span><span class="meta">?&gt;</span></span></span><br><span class="line"><span class="tag">&lt;<span class="name">beans</span> <span class="attr">xmlns</span>=<span class="string">"http://www.springframework.org/schema/beans"</span></span></span><br><span class="line"><span class="tag">       <span class="attr">xmlns:xsi</span>=<span class="string">"http://www.w3.org/2001/XMLSchema-instance"</span></span></span><br><span class="line"><span class="tag">       <span class="attr">xmlns:aop</span>=<span class="string">"http://www.springframework.org/schema/aop"</span></span></span><br><span class="line"><span class="tag">       <span class="attr">xmlns:tx</span>=<span class="string">"http://www.springframework.org/schema/tx"</span></span></span><br><span class="line"><span class="tag">       <span class="attr">xmlns:context</span>=<span class="string">"http://www.springframework.org/schema/context"</span></span></span><br><span class="line"><span class="tag">       <span class="attr">xsi:schemaLocation</span>=<span class="string">"</span></span></span><br><span class="line"><span class="tag"><span class="string">       http://www.springframework.org/schema/beans</span></span></span><br><span class="line"><span class="tag"><span class="string">       http://www.springframework.org/schema/beans/spring-beans-3.0.xsd</span></span></span><br><span class="line"><span class="tag"><span class="string">       http://www.springframework.org/schema/tx</span></span></span><br><span class="line"><span class="tag"><span class="string">       http://www.springframework.org/schema/tx/spring-tx-3.0.xsd</span></span></span><br><span class="line"><span class="tag"><span class="string">       http://www.springframework.org/schema/context</span></span></span><br><span class="line"><span class="tag"><span class="string">       http://www.springframework.org/schema/context/spring-context-3.0.xsd</span></span></span><br><span class="line"><span class="tag"><span class="string">       http://www.springframework.org/schema/aop</span></span></span><br><span class="line"><span class="tag"><span class="string">       http://www.springframework.org/schema/aop/spring-aop-3.0.xsd"</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">bean</span> <span class="attr">id</span>=<span class="string">"p1"</span> <span class="attr">class</span>=<span class="string">"top.nymrli.tryboot.po.User"</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">property</span> <span class="attr">name</span>=<span class="string">"name"</span> <span class="attr">value</span>=<span class="string">"张三"</span>/&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">property</span> <span class="attr">name</span>=<span class="string">"age"</span> <span class="attr">value</span>=<span class="string">"18"</span>/&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">bean</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">beans</span>&gt;</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    @Test</span><br><span class="line">    public void testPerson()</span><br><span class="line">    &#123;</span><br><span class="line"> ApplicationContext ctx = new ClassPathXmlApplicationContext("applicationContext.xml");</span><br><span class="line"></span><br><span class="line">        User user = annotationConfigApplicationContext.getBean("p1", User.class);</span><br><span class="line">        System.out.println(user);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li></ul><h4 id="bean"><a class="markdownIt-Anchor" href="#bean"></a> <code>@Bean</code></h4><ul><li><p><code>@Bean</code>: 通过注解来标注SpringBean对象</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">Config</span> </span>&#123;</span><br><span class="line">    <span class="meta">@Bean</span>(<span class="string">"user"</span>)</span><br><span class="line">    <span class="function"><span class="keyword">public</span> User <span class="title">getUser</span><span class="params">()</span></span>&#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">new</span> User(<span class="string">"name"</span>, <span class="number">20</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@Test</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">testPerson</span><span class="params">()</span></span></span><br><span class="line"><span class="function">    </span>&#123;</span><br><span class="line">        AnnotationConfigApplicationContext ctx = <span class="keyword">new</span> AnnotationConfigApplicationContext();</span><br><span class="line">        ctx.register(Config.class);</span><br><span class="line">        <span class="comment">// 或者 AnnotationConfigApplicationContext annotationConfigApplicationContext = new AnnotationConfigApplicationContext(Config.class);</span></span><br><span class="line"></span><br><span class="line">        ctx.refresh();</span><br><span class="line"></span><br><span class="line">        User user = ctx.getBean(<span class="string">"user"</span>, User.class);</span><br><span class="line">        System.out.println(user);</span><br><span class="line">    &#125;</span><br></pre></td></tr></table></figure><p><code>AnnotationConfigApplicationContext annotationConfigApplicationContext = new AnnotationConfigApplicationContext(Config.class);</code>与<code>ApplicationContext ctx = new ClassPathXmlApplicationContext(&quot;applicationContext.xml&quot;);</code>一样，都是指定bean的配置路径后解析，获得bean的配置。</p></li></ul><h4 id="component"><a class="markdownIt-Anchor" href="#component"></a> <code>@Component</code></h4><ul><li><p><code>@Component</code>: 通过标注来创建一个组件SpringBean对象，如下的User。 ==&gt; 等价于<code>ctx.registerBean(User.class)</code>，registerBean方法的实现最终也是调用了registerBeanDefinition</p><p>而Config不是一个bean，他的作用跟applicationContext.xml一样，他是存放bean定义路径的一个配置文件，在这里是通过ComponentScan让Spring能够找到top.nymrli.tryboot.po下的所有Bean并解析成SpringBean</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@ComponentScan</span>(<span class="string">"top.nymrli.tryboot.po"</span>)</span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">Config</span> </span>&#123;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@Data</span></span><br><span class="line"><span class="meta">@AllArgsConstructor</span></span><br><span class="line"><span class="meta">@NoArgsConstructor</span></span><br><span class="line"><span class="meta">@Component</span><span class="comment">// ==&gt; beanName为user, 类名小写</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">User</span> </span>&#123;</span><br><span class="line">    <span class="keyword">private</span> String name;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">int</span> age;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">testPerson</span><span class="params">()</span></span></span><br><span class="line"><span class="function"></span>&#123;</span><br><span class="line">    AnnotationConfigApplicationContext ctx = <span class="keyword">new</span> AnnotationConfigApplicationContext(Config.class);</span><br><span class="line"></span><br><span class="line">    User user = ctx.getBean(User.class);</span><br><span class="line">    System.out.println(user);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>会自动调用User的无参构造方式来创建JavaBean， 因此得到的User是<code>User(name=null, age=0)</code></p></li></ul><p>▲.上述三种创建方式本质上都是创建了BeanDefinition，只不过Spring进行了封装，因此使用起来比较简单直观</p><h4 id="import"><a class="markdownIt-Anchor" href="#import"></a> @Import</h4><blockquote><p>spring4.2之后<code>@Import</code>注解可以实例化普通类的bean实例</p></blockquote><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Data</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">Role</span> </span>&#123;</span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="keyword">private</span> String name;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="meta">@Import</span>(Role.class)<span class="comment">// @Import(&#123;Role.class, User.class&#125;)</span></span><br><span class="line"><span class="meta">@Configuration</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">MyConfig</span> </span>&#123;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="meta">@RequestMapping</span>(<span class="string">"/"</span>)</span><br><span class="line"><span class="meta">@RestController</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">TestController</span> </span>&#123;</span><br><span class="line">    <span class="meta">@Autowired</span></span><br><span class="line">    <span class="keyword">private</span> Role role;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@GetMapping</span>(<span class="string">"/test"</span>)</span><br><span class="line">    <span class="function"><span class="keyword">public</span> String <span class="title">test</span><span class="params">()</span> </span>&#123;</span><br><span class="line">        System.out.println(role);</span><br><span class="line">        <span class="keyword">return</span> <span class="string">"test"</span>;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>Spring在创建MyConfig的时候，会根据@Import将Role也实例化成一个Bean</p><h3 id="编程式"><a class="markdownIt-Anchor" href="#编程式"></a> 编程式</h3><h4 id="beandefinition"><a class="markdownIt-Anchor" href="#beandefinition"></a> BeanDefinition</h4><blockquote><p>通过创建并配置 BeanDefinition 来创建一个Bean</p><p>Q: 什么是BeanDefinition?</p><p>A: BeanDefinition表示Bean的定义信息， Spring会根据BeanDefinition来创建Bean对象。BeanDefinition有很多的属性用来描述Bean， BeanDefinition是Spring中非常核心的概念。</p><ul><li>beanClass: 表示Bean对象的类类型， Spring在创建Bean的过程中根据此属性来判断实例化对象具体的类类型</li><li>scope：作用域，[singleton、prototype]，单例或者原型bean</li><li>isLazy：是否需要懒加载，原型Bean的懒加载不奏效。懒加载的单例bean，会在第一次getBean的时候生成该bean，非懒加载的单例bean，则会在Spring.启动过程中直接生成好。</li><li>dependsOn：表示一个bean在创建之前所依颗的其他bean，在一个bean创建之前，它所依赖的这些bean得先全部创建好。</li><li>primary：表示一个bean是主bean，在Spring中一个类型可以有多个bean对象，在进行依赖注入时，如果根据类型找到了多个b©an，此时会判断这些bean中是否存在一个主bean，如果存在，则直接将这个bean注入给属性。</li><li>initMethodName： 表示一个bean的初始化方法，一个bean的生命周期过程中有一个步骤叫初始化，Spring会在这个步骤中去调用bean的初始化方法。初始化逻辑由程序员自己控制，使得程序员可以自定义逻辑对bean进行加工。</li></ul><p>通过<code>&lt;bean&gt;&lt;/bean&gt;</code>、<code>@Bean</code>、<code>@Component</code>标注的类都会解析为BeanDefinition，供Spring创建（实例化）具体的Bean。</p></blockquote><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">testPerson</span><span class="params">()</span></span></span><br><span class="line"><span class="function"></span>&#123;</span><br><span class="line">    AnnotationConfigApplicationContext ctx = <span class="keyword">new</span> AnnotationConfigApplicationContext(Config.class);</span><br><span class="line">    AbstractBeanDefinition bd = BeanDefinitionBuilder.genericBeanDefinition().getBeanDefinition();</span><br><span class="line">    bd.setBeanClass(User.class);</span><br><span class="line">    <span class="comment">// 将BeanDefinition注册到上下文中</span></span><br><span class="line">    ctx.registerBeanDefinition(<span class="string">"user"</span>, bd);</span><br><span class="line">    <span class="comment">// 获得Bean对象</span></span><br><span class="line">    User user = (User)ctx.getBean(<span class="string">"user"</span>);</span><br><span class="line">    System.out.println(user);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h4 id="factorybean"><a class="markdownIt-Anchor" href="#factorybean"></a> FactoryBean</h4><blockquote><p>FactoryBean是Spring.所提供的一种较灵活的创建Bean的方式，可以通过实现FactoryBean：接口中的getObject()方法来返回一个对象，这个对象就是最终的Bean对象。</p></blockquote><p>FactoryBean接口中的方法</p><ul><li>Object getObject()：返回的是Bean对象</li><li>boolean isSingleton()：返回的是否是单例Bean对象</li><li>Class getObjectType()：返回的是Bean对象的类型</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Component</span>(<span class="string">"zhouyu"</span>)</span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">ZhouyuFactoryBean</span> <span class="keyword">implements</span> <span class="title">FactoryBean</span> </span>&#123;</span><br><span class="line">    <span class="meta">@Override</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">isSingleton</span><span class="params">()</span> </span>&#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">true</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@Override</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> Object <span class="title">getObject</span><span class="params">()</span> </span>&#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">new</span> User(<span class="string">"cl"</span>, <span class="number">18</span>);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@Override</span></span><br><span class="line">    <span class="keyword">public</span> Class&lt;?&gt; getObjectType() &#123;</span><br><span class="line">        <span class="keyword">return</span> User.class;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>上述代码，实际上对应了两个Bean对象：<br>1、beanName为&quot;zhouyu’，bean对象为getObject方法所返回的User对象。=&gt;<code>applicationcontext.getBean(name：&quot;zhouyu&quot;，ZhouyuFactory.class);</code><br>2、beanName为&quot;&amp;zhouyu’，bean对象为ZhouyuFactoryBean类的实例对象。=&gt;<code>applicationcontext.getBean(name：&quot;&amp;zhouyu&quot;，ZhouyuFactoryBean.class);</code></p><p>与BeanFactory的区别：</p><ul><li>FactoryBean对象本身也是一个Bean，同时它相当于一个小型工厂，可以生产出<strong>另外</strong>的Bean。==》指定类型的Bean对象<ul><li>FactoryBean机制被广泛的应用在Spring内部和Spring-与第三方框架或组件的整合过程中。</li></ul></li><li>BeanFactory是一个Spring容器，是一个大型工厂，它可以生产出<strong>各种各样</strong>的Beano</li></ul><h4 id="supplier"><a class="markdownIt-Anchor" href="#supplier"></a> Supplier</h4><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">testPerson</span><span class="params">()</span></span></span><br><span class="line"><span class="function"></span>&#123;</span><br><span class="line">        AnnotationConfigApplicationContext ctx = <span class="keyword">new</span> AnnotationConfigApplicationContext(Config.class);</span><br><span class="line">        ctx.registerBean(User.class, <span class="keyword">new</span> Supplier&lt;User&gt;() &#123;</span><br><span class="line">            <span class="meta">@Override</span></span><br><span class="line">            <span class="function"><span class="keyword">public</span> User <span class="title">get</span><span class="params">()</span> </span>&#123;</span><br><span class="line">                User userx = <span class="keyword">new</span> User();</span><br><span class="line">                userx.setAge(<span class="number">222</span>);</span><br><span class="line">                userx.setName(<span class="string">"supplier"</span>);</span><br><span class="line">                <span class="keyword">return</span> userx;</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;);</span><br><span class="line"></span><br><span class="line">        User user = (User)ctx.getBean(<span class="string">"user"</span>);</span><br><span class="line">        System.out.println(user);</span><br><span class="line">    &#125;</span><br></pre></td></tr></table></figure><h2 id="beanfactory"><a class="markdownIt-Anchor" href="#beanfactory"></a> BeanFactory</h2><p>BeanFactory是一种“Spring容器”， BeanFactory翻译过来就是Bean工厂，顾名思义，它可以用来创建Bean、获取Bean。</p><p>概念区分：</p><ul><li>BeanFactory将利用BeanDefinition：来生成Bean对象</li><li>BeanDefinition相当于BeanFactory的原材料</li><li>Bean对象就相当于BeanFactory）所生产出来的产品</li></ul><p>BeanFactory的核心子接口和实现类：</p><ul><li>ListableBeanFactory</li><li>ConfigurableBeanFactory</li><li>AutowireCapableBeanFactory</li><li>AbstractBeanFactory</li><li>DefaultListableBeanFactory</li></ul><p>···</p><h2 id="bean生命周期"><a class="markdownIt-Anchor" href="#bean生命周期"></a> Bean生命周期</h2><blockquote><p>Bean生命周期描述的是Spring中一个Bean创建过程和销毁过程中所经历的步骤，其中Bean创建过程是重点。程序员可以利用Bean生命周期机制对Bean进行自定义加工。</p></blockquote><ul><li>Bean定义：通过xml或者编写类的形式来创建BeanDefinition</li><li>构造方法推断：从bean的多个构造方法中选出一个构造方法</li><li>实例化：通过构造方法反射获得对象，在Spring中可以通过BeanPostProcessor机制对实例化进行干预</li><li>属性填充(自动注入、依赖注入)：给属性进行自动填充，比如@Value，@AutoWired</li><li>初始化：在一个对象的属性填充之后，Spring提供了初始化机制，让程序员可以对其他属性进行赋值、校验等自定义加工(利用InitializingBean接口)</li><li>初始化后：Aop、生成代理对象。常说的Aop机制就是在这个步骤中通过BeanPostProcessor机制实现的，这步之后获得的对象才是真正的Bean对象</li></ul><p>创建过程：</p><p><img src="/2022/02/27/深入学习使用Spring/E:%5Chexo%5Csource_posts%5C%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0%E4%BD%BF%E7%94%A8Spring%5Cbean%E5%88%9B%E5%BB%BA%E8%BF%87%E7%A8%8B.jpg" alt="bean创建过程"></p><p>生命周期图：</p><p><img src="/2022/02/27/深入学习使用Spring/E:%5Chexo%5Csource_posts%5C%E6%B7%B1%E5%85%A5%E5%AD%A6%E4%B9%A0%E4%BD%BF%E7%94%A8Spring%5Cbean%E7%94%9F%E5%91%BD%E5%91%A8%E6%9C%9F-1.jpg" alt="bean生命周期-1"></p><h2 id="属性填充注解autowired-resource-value"><a class="markdownIt-Anchor" href="#属性填充注解autowired-resource-value"></a> 属性填充注解：@Autowired、@Resource、@Value</h2><h3 id="autowired"><a class="markdownIt-Anchor" href="#autowired"></a> @Autowired.</h3><p>表示某个属性是否需要进行依赖注入，可以写在属性和方法上。注解中的required属性默认为ture，表示如果没有对象可以注入给属性则抛异常。</p><ul><li><p>@Autowired：加在某个<strong>属性</strong>上，Spring在进行Bean的生命周期过程中，在<strong>属性填充</strong>这一步会基于实例化出来的对象，对该对象中加了@Autowired的属性自动给属性赋值。具体实现为： Spring会先根据属性的类型去Spring容器中找出该类型所有的Bean对象，如果找出来多个，则再根据属性的名字从多个中再确定一个。如果required属性为true，并且根据属性信息找不到对象，则直接抛异常。</p></li><li><p>当@Autowired注解写在<strong>某个方法</strong>上时，Spring在B ean生命周期的属性填充阶段，会根据方法的参数类型、参数名字从Spring容器找到对象当做方法入参，自动反射调用该方法。</p></li><li><p>@Autowired加在<strong>构造方法</strong>上时，Spring会在<strong>推断构造方法阶段</strong>，<u>选择该构造方法来进行实例化</u>，在反射调用构造方法之前，会先根据构造方法参数类型、参数名从Spring容器中找到Bean对象，当做构造方法入参。</p></li></ul><h3 id="resource"><a class="markdownIt-Anchor" href="#resource"></a> @Resource</h3><p>@Resource注解与@Autowired类似，也是用来进行依赖注入的，@Resource是<strong>Java层面</strong>所提供的注解，@Autowired.是<strong>Spring</strong>所提供的注解，它们依赖注入的底层实现逻辑也不同。</p><p>@Resource：注解中有一个name属性，针对na me属性是否有值，@Resource的依赖注入底层流程是不同的。</p><ul><li>@Reousrce：如果name属性有值，那么Spring会直接根据所指定的name值去Spring容器找Bean对象，如果找到了则成功，如果没有找到，则报错。</li><li>如果@Resource中的name属性没有值，则：<ol><li>先判断该属性名字在Spring容器中是否存在Bean对象。</li><li>如果存在，则成功找到Bean对象进行注入。</li><li>如果不存在，则根据属性类型去Spring容器找Bean对象，找到一个则进行注入。</li></ol></li></ul><h3 id="value"><a class="markdownIt-Anchor" href="#value"></a> @Value</h3><p>@Value注解和@Resource、@Autowired类似，也是用来对属性进行依赖注入的，只不过@Value是用来从<strong>Properties.文件中</strong>来获取<strong>值</strong>的，并且@Value可以解析SpEL（Spring表达式）。</p><ul><li><p><code>@Value(&quot;yuzhou&quot;)</code>: 直接将字符串“zhouyu”赋值给属性，如果属性类型不是String，或无法进行类型转化，则报错。</p></li><li><p><code>@Value(${&quot;yuzhou&quot;})</code>： 将会把<code>${}</code>中的字符串当做key，从Properties文件中找出对应的value赋值给属性，如果没找到，则会把<code>${zhouyu}</code>当做普通字符串注入给属性。</p></li><li><p><code>@Value(#{&quot;yuzhou&quot;})</code>:会将<code>#{}</code>中的字符串当做Spring表达式进行解析，Spring会把&quot;zhouyu&quot;当做beanName，并从Spring容器中找对应bean，如果找到则进行属性注入，没找到则报错。</p></li><li></li></ul><h2 id="applicationcontext"><a class="markdownIt-Anchor" href="#applicationcontext"></a> ApplicationContext</h2><p>ApplicationContext是比BeanFactory更加强大的Spring容器，它既可以创建bean、获取bean，还支持国际化、事件广播、获取资源等BeanFactor y不具备的功能。</p><p>Application Context，所继承的接口</p><ul><li>EnvironmentCapable：拥有获取环境变量的能力：操作系统环境变量和Jvm环境变量</li><li>ListableBeanFactory：拥有了获取所有beanNames、判断某个bean Name是否存在peanDefinition对象、统计Be anDefinition个数、获取某个类型对应的所有beanNames等功能。</li><li>HierarchicalBeanFactory：拥有了获取父BeanFactory、。判断某个name是否存在bean对象的功能。</li><li>MessageSource：拥有了国际化功能，比如可以直接利用Messa geSource对象获取某个国际化资源（比如不同国家语言所对应的字符）</li><li>ApplicationEventPublisher： 拥有了事件发布功能，可以发布事件，这是Application Context相对于BeanFactory比较突出、常用的功能。</li><li>ResourcePatternResolver: 拥有了加载并获取资源的功能，这里的资源可以是文件，图片等某个URL资源都可以。</li></ul><h3 id="获得bean的方式"><a class="markdownIt-Anchor" href="#获得bean的方式"></a> 获得bean的方式：</h3><blockquote><p><em>getBean() API</em></p></blockquote><ul><li><p>按名称获取Bean</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Object lion = context.getBean(<span class="string">"lion"</span>);</span><br><span class="line"> </span><br><span class="line">assertEquals(Lion.class, lion.getClass());</span><br></pre></td></tr></table></figure><p>根据bean名称获取bean，如果在spring ico容器中存在和bean，则返回<em>Object</em> 类的实例。否则，抛出如下异常NoSuchBeanDefinitionException。</p><p>▲: 主要的缺点是，在获取bean之后，我们<strong>必须将它指定转换为所需的类型</strong>。如果返回的bean的类型与我们期望的不同，则可能会产生异常。</p></li><li><p>通过名称和类型获取Bean</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">Lion lion = context.getBean(<span class="string">"lion"</span>, Lion.class);</span><br><span class="line"><span class="comment">// 与按名称获取Bean相比，此方法更安全，因为我们可以编译阶段就发现错误而不是在运行阶段。</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">assertThrows(BeanNotOfRequiredTypeException.class, () -&gt; </span><br><span class="line">    context.getBean(<span class="string">"lion"</span>, Tiger.class));</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li><li><p>按类型获取Bean</p></li></ul>  <figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// right</span></span><br><span class="line">Lion lion = context.getBean(Lion.class);</span><br><span class="line"></span><br><span class="line"><span class="comment">// 注意: 在下面的情况下，需要特别注意可能存在的歧义：</span></span><br><span class="line">assertThrows(NoUniqueBeanDefinitionException.class, () -&gt; </span><br><span class="line">    context.getBean(Animal.class));</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>由于<em>Lion</em>和<em>Tiger都</em>实现了<em>Animal</em>接口，因此仅指定类型不足以明确确定结果。因此，我们会得到一个*<code>NoUniqueBeanDefinitionException</code>*。即在<strong>同一个IOC 容器中，如果<u>有相同类型的多个bean</u>，则不能通过类型获取bean</strong>。</p><ul><li>按名称和构造函数参数对Bean进行筛选</li></ul>  <figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Tiger tiger = (Tiger) context.getBean(<span class="string">"tiger"</span>, <span class="string">"Siberian"</span>);</span><br><span class="line">Tiger secondTiger = (Tiger) context.getBean(<span class="string">"tiger"</span>, <span class="string">"Striped"</span>);</span><br><span class="line"> </span><br><span class="line">assertEquals(<span class="string">"Siberian"</span>, tiger.getName());</span><br><span class="line">assertEquals(<span class="string">"Striped"</span>, secondTiger.getName());</span><br></pre></td></tr></table></figure><ul><li><p>按类型和构造函数参数对Bean进行筛选</p><blockquote><p>▲: 这个方法有点不同，因为它<strong>只适用于具有原型作用域</strong>的bean。</p></blockquote><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Tiger tiger = context.getBean(Tiger.class, <span class="string">"Shere Khan"</span>);</span><br><span class="line">assertEquals(<span class="string">"Shere Khan"</span>, tiger.getName());</span><br></pre></td></tr></table></figure></li></ul><h2 id="beanpostprocessor机制"><a class="markdownIt-Anchor" href="#beanpostprocessor机制"></a> BeanPostProcessor机制</h2><p>BeanPostProcessor是Spring所提供的一种扩展机制，可以利用该机制对Bean进行定制化加工，在Spring底层源码实现中，也广泛的用到了该机制，BeanPostProcessor通常也叫做<strong>Bean后置处理器</strong>。</p><p>BeanPostProcessor在Spring中是一个接口，我们定义一个后置处理器，就是提供一个类实现该接口，在Spring中还存在一些接口继承了BeanPostProcessor，这些子接口是在BeanPostProcessor的基础上增加了一些其他的功能。</p><p>BeanPostProcessori中的方法</p><ul><li>postProcessBeforelnitialization()：初始化前方法，表示可以利用这个方法来对Bean在初始化前进行自定义加工。</li><li>postProcessAfterInitialization()：初始化后方法，表示可以利用这个方法来对Bean在初始化后进行自定义加工。</li></ul><h3 id="instantiationawarebeanpostprocessor"><a class="markdownIt-Anchor" href="#instantiationawarebeanpostprocessor"></a> InstantiationAwareBeanPostProcessor</h3><blockquote><p>BeanPostProcessor的一个子接口</p></blockquote><ul><li>postProcessBeforelnstantiation()：实例化前</li><li>postProcessAfterInstantiation()：实例化后</li><li>postProcessProperties()：属性注入后</li></ul><h2 id="aop是什么"><a class="markdownIt-Anchor" href="#aop是什么"></a> AOP是什么？</h2><p>AOP就是面向切面编程，是一种非常适合在<strong>无需修改业务代码的前提</strong>下，对<u>某个或某些业务增加统一的功能</u>（横向增加功能），比如日志记录、权限控制、事务管理等，能很好的使得代码解耦，提高开发效率。</p><ul><li></li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;blockquote&gt;
&lt;p&gt;From : &lt;a href=&quot;https://www.bilibili.com/video/BV1fr4y1Y7UH?p=6&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;https://www.bilibili.com/vid
      
    
    </summary>
    
    
      <category term="Web" scheme="https://nymrli.top/tags/Web/"/>
    
      <category term="Spring" scheme="https://nymrli.top/tags/Spring/"/>
    
  </entry>
  
  <entry>
    <title>研究生work——地图匹配</title>
    <link href="https://nymrli.top/2022/02/26/%E7%A0%94%E7%A9%B6%E7%94%9Fwork%E2%80%94%E2%80%94%E5%9C%B0%E5%9B%BE%E5%8C%B9%E9%85%8D/"/>
    <id>https://nymrli.top/2022/02/26/研究生work——地图匹配/</id>
    <published>2022-02-26T07:24:46.000Z</published>
    <updated>2022-03-09T13:33:00.468Z</updated>
    
    <content type="html"><![CDATA[<h1 id="地图匹配"><a class="markdownIt-Anchor" href="#地图匹配"></a> 地图匹配</h1><p>概念：</p><blockquote><ul><li>地图匹配（Map-Matching）是将运营车辆的有序的GPS位置关联到电子地图的路网上，将GPS坐标下采样序列转换为数字路网坐标序列的过程；本质上是平面线段序列的模式匹配问题（ Alt等，2003）</li><li>地图匹配是一种将原始GPS位置映射到路网上的路段的过程，以创建对车辆所走路线的估算。</li></ul></blockquote><p>两个主要的地图匹配用例</p><ul><li>乘车结束时，计算驾驶员行进的距离，以计算票价。——路线图匹配(EORMM)</li><li><strong>实时</strong> ，为ETA团队提供准确的位置并做出<strong>调度</strong>决策，并在rider应用程序上<strong>显示</strong>驾驶员的汽车。——实时地图匹配(RTMM)</li></ul><p>不同点在于，<em>实时地图匹配</em>的要求比较高，必须低延时，因此相比之下，<em>路线图匹配</em>对等待时间的要求不那么严格，并且可以使用乘车的全部历史记录</p><h3 id="参考"><a class="markdownIt-Anchor" href="#参考"></a> 参考：</h3><ul><li><p><a href="https://zhuanlan.zhihu.com/p/83039334" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/83039334</a></p></li><li><p><a href="https://blog.csdn.net/weixin_26713521/article/details/108134220" target="_blank" rel="noopener">Lyft的地图匹配算法论文翻译</a></p><ul><li><p>地图匹配算法的性能取决于<strong>道路网络数据</strong>的<strong>质量</strong></p></li><li><p>解决问题的一种好的方法是使用<strong>状态空间模型</strong> 。 状态空间模型是时间序列模型，其中系统具有“隐藏”状态，这些状态无法直接观察到，但会引起可见的观察。 在这里，我们的隐藏状态是我们要估算的汽车在道路网络上的实际位置。 我们仅观察到隐藏状态的修改版本：观察值(原始位置数据)。 我们假设系统的状态以仅取决于当前状态的方式演化(马尔可夫假设)，并进一步定义了从隐藏状态到隐藏状态的转移密度和从隐藏状态到观察的密度。</p><p>常用的地图匹配状态空间模型是离散状态<strong>隐马尔可夫模型</strong> (Newson＆Krumm [2]，DiDi的IJCAI-19教程[3]，Uber的Map Matching [4])。 在该系统中，我们通过查看路段上的最近点来生成候选对象，并使用<a href="https://en.wikipedia.org/wiki/Viterbi_algorithm" target="_blank" rel="noopener">维特比算法</a>查找最可能的隐藏状态序列。</p><ul><li>对于不同的建模选择和输入数据而言，它相对不灵活</li><li>它缩放严重(O(N²)，其中N是每个状态下可能的候选数)</li><li>它不能很好地应对高频观测(请参阅Newson＆Krumm [2])。</li></ul></li></ul></li><li><p><a href="http://qikan.cqvip.com/Qikan/Article/Detail?id=1000359922&amp;from=Qikan_Search_Index" target="_blank" rel="noopener">GPS定位与地图匹配方法研究</a></p><ul><li>地图匹配算法从原理上可以解释为两个独立的过程：（1）找到车辆当前行使的道路——确定候选路段（2）将当前GPS定位点投影到车辆行使的道路上——候选路段匹配规则<ul><li>圆心拓展半径找到最近的路段（唯一）——构建第一条边的算法</li></ul></li></ul></li><li><p>[1]苏洁, 周东方, 岳春生. GPS车辆导航中的实时地图匹配算法[J]. 测绘学报, 2001, 30(3):5.</p><ul><li>另外，为了提高算法的鲁棒性，对于误差引起 的速度异常，我们利用推测航位法和线性插值来 进行 GPS数据补偿，以消除部分GPS接收外部 粗差</li></ul></li><li><p>[1]李清泉, 黄练. 基于GPS轨迹数据的地图匹配算法[J]. 测绘学报, 2010(2):6.</p><ul><li>针对传统导航系统的地图匹配方法的研究较多。其中点到线的匹配由于缺乏对整体轨迹趋势的考虑，在复杂的道路网环境下的匹配易导致误匹配。也有方法使用DR、差分GPS等辅助设备，采用滤波、模糊逻辑、证据理论等方法提高地图匹配的准确率。GPS浮动车轨迹数据提供了整体曲线的变化趋势，可以采用全局整体匹配的思想，保证轨迹的完整性和准确性14？。基于曲线相似度的算法一般 较为复杂，但匹配精度高，利于轨迹的直接恢复，适合进行数据的后处理</li><li>与整个轨迹相对 应的路径必然是连通 的路段集 ，可以基于道路拓扑与连通性设计地图匹配算法</li><li>key：除了道路的几何连通性，实际行车还会受到交通规则的限制，本文基于道路网的行车限制信息提出了一种GPS浮动车轨迹数据的全局地图匹配算法，综合考虑备选路段的几何连通性与交通网络条件约束构建整体备选路径，然后使用<strong>改进的扫描线法</strong>判断全局轨迹曲线与备选路径的相似度，完成地图匹配 。<ul><li>一句话概括算法：通过行车限制将候选线段找出来后，选择曲线相似度最高的</li></ul></li></ul></li><li><p>[1]陈滨, 王平, 施文灶,等. GPS轨迹数据的综合地图匹配算法研究[J]. 电子科技, 2014, 27(12):4.</p><ul><li>从实际的匹配效果来看，此匹配算法在交叉路口等路段较复杂的地方可有较高地匹配准确率，但该方法依赖于前后GPS定位点匹配准确度，若前一定位点匹配错误就会出现连锁反应，从而导致后面一系列点匹配错误；且历史轨迹推算匹配法计算量较大，匹配速度较慢，不利于高速实时定位。因此，需和其他地图匹配方法相结合使用才能取得较好的匹配效果。</li></ul></li><li><p>智能交通系统中GPS地图匹配算法设计与实现_罗杰涛</p><ul><li>模糊逻辑： 效率高，实时性好，对绝大多数的路段状况都适用。不过在车俩拐弯处以及车速较慢的情况下匹配效果不尽如人意，且不同路段建模的系数凭靠经验值，没有相应的缺乏理论依据。——01、06、08论文</li></ul></li><li><p>高级地图匹配算法：研究现状和趋势[2021]</p><ul><li><p>从实现技术或模型角度对近十年提出的算法进行分类，箭头标记算法间的继承关系。从图2可见，HMM模型是主流，其次是基于最大权重的模型.HMM-News-on、ST-Matching2 3IVMM3和HRIS8被引用对比最多，是具有开创性的工作。另外，从2019年开始，有研究采用深度学习技术来解决地图匹配问题。</p><p><img src="/2022/02/26/研究生work——地图匹配/E:%5Chexo%5Csource_posts%5C%E7%A0%94%E7%A9%B6%E7%94%9Fwork%E2%80%94%E2%80%94%E5%9C%B0%E5%9B%BE%E5%8C%B9%E9%85%8D%5C09-20%E5%B9%B4%E5%9C%B0%E5%9B%BE%E5%8C%B9%E9%85%8D%E7%AE%97%E6%B3%95%E5%8F%91%E5%B1%95%E5%8E%86%E7%A8%8B.jpg" alt="09-20年地图匹配算法发展历程"></p></li></ul></li></ul><h3 id="points"><a class="markdownIt-Anchor" href="#points"></a> Points:</h3><ul><li><p>另外，一个重要的实际问题是，即使车辆定位精度可以保证在10米以内，当电子地图缩放到较大的比例尺时，也会<br>出现车辆偏离行使的道路而造成的视觉混乱现象.</p></li><li><p>由于城区内地物特征复杂，受密集的高大建筑物、隧道、立交桥、树木等地物的反射和遮蔽等影响，车栽GPS接收机接收到的卫星信号存在严重的多径效应，在某些区域内甚至会形成GPS定位育区，解决GPS盲区问题，一种方法是采用航位推算法（DR-Dead Reckoning），这种方法需要将DR设备装在车辆前端，一般在GPS接收机卫星信号受阻时一样可以得到正确的用户位置，但这种方法需要附加设备。另一种方法是在记忆正确GPS位置信息的基础上在一段时间内预测车辆位置，这种方法的优点是易于实现，缺点是精度不够（没有，作者使用卡尔曼滤波进行位置预测，预测结果如图所示。在规则行进时，预测效果较好，见图3、图4.但是当车辆转弯时，效果很差，见图5、图6</p></li><li><p>到折线的距离定义为点到折线上所有直线段的最短距离：点到线段的距离定义为如果点到直线段所在直线的投影在直线段上，则为垂线长度，否则为其到两个端点的最短距离。</p></li><li><p>定位误差： GPS误差、电子地图库误差、坐标投影变换投影</p></li><li><p>难点：</p><ul><li><a href="http://qikan.cqvip.com/Qikan/Article/ReadIndex?id=33650279&amp;info=X19Wa%2bQedw8TOqUh1eM7U0SrEn1nIM08tSny4w0YhPo%3d" target="_blank" rel="noopener">Y-junction问题</a>——点到线的匹配方式，没有考虑全局匹配</li><li>平行双线路</li></ul></li><li><p>MapMatching实现的思路</p><ul><li><strong>离散点集匹配</strong>：相对简单，随机离散点没有形状和拓扑关系，用欧氏距离作吸附即可，典型应用如离散热力图。</li><li><strong>曲线拟合</strong>： 实际中更有应用价值的是曲线拟合匹配关系，比如轨迹和路网，GPS序列和导航路的相似性。</li></ul></li><li><p>MapMatching算法的分类：</p><p><strong>以使用到的信息来划分</strong></p><p>现有的算法可被分成四类：几何、拓扑、概率、高级。</p><p>a）基于几何的算法考虑GPS点与道路的几何信息，如距离、角度等；</p><p>b）基于拓扑的算法使用道路拓扑信息来控制；</p><p>c）概率方法通过考虑GPS点的概率；</p><p>d）高级的算法往往综合考虑使用全面信息，有卡尔曼滤波、模糊逻辑模型、隐式马尔可夫模型等等。</p><p><strong>2.2 以考虑采样点的范围来划分</strong></p><p>根据考虑采样点的范围，可分成局部/增量算法、全局算法。</p><p>a）局部/增量算法是贪婪算法，每次确定一个匹配点，下个点从已经确定的匹配点开始。这些方法根据距离和方向相似性来找到局部最优点或边。（在线匹配）</p><p>b）全局算法是要从路网中找到一条与采样轨迹最接近的匹配轨迹。为了测量采样轨迹和匹配轨迹的相似性，大多数算法使用“Frechet距离”或者是“弱Frechet距离”。还有时空匹配算法、投票算法等。（离线匹配）</p><p><strong>以采样点的频率来划分</strong></p><p>根据轨迹数据的采样频率，现有的地图匹配算法可分成：</p><p>a）高频采样算法（所有局部算法、部分全局算法如Frechet距离判别法等）</p><p>b）低频采样算法（ST-matching算法、IVVM算法）</p><p>一般认为30s及其以上为低频采样，1s～10s为高频采样。</p><p><a href="https://www.cnblogs.com/LBSer/p/4612031.html#!comments" target="_blank" rel="noopener">https://www.cnblogs.com/LBSer/p/4612031.html#!comments</a></p></li><li><p>智能交通系统中GPS地图匹配算法设计与实现</p><p><img src="/2022/02/26/研究生work——地图匹配/E:%5Chexo%5Csource_posts%5C%E7%A0%94%E7%A9%B6%E7%94%9Fwork%E2%80%94%E2%80%94%E5%9C%B0%E5%9B%BE%E5%8C%B9%E9%85%8D%5C%E5%9C%B0%E5%9B%BE%E5%8C%B9%E9%85%8D%E7%BB%84%E4%BB%B6.jpg" alt="地图匹配组件"></p></li></ul><h3 id="算法评估的标准"><a class="markdownIt-Anchor" href="#算法评估的标准"></a> 算法评估的标准</h3><ul><li>实时性</li><li>可靠性(鲁棒性)</li><li>匹配的精度</li></ul><h2 id="朴素算法"><a class="markdownIt-Anchor" href="#朴素算法"></a> 朴素算法：</h2><p>快速匹配算法的执行步骤如下：</p><ol><li>步1接收GPS定位数据；</li><li>步2判断定位数据是否无效，若无效，则根据历史定位数据进行推测匹配，然后转（8）；</li><li>步3判断车辆当前是否处于停止或低速滑行状态，若是，对其作相应处理，然后转（8）；</li><li>步4由车辆当前位置点计算其对应的候选网格，进而获取其中的路段；</li><li>步5对步4得到的路段进行<strong>连接性拓扑检查</strong>，将通过拓扑检查的路段作为匹配候选路段；</li><li>步6判断匹配候选路段数目，若唯一，则直接将其作为匹配路段，并由定位点向其作投影，然后转（8）；否则，计算所有候选路段的匹配度度量值f，（i=1,2，…，N），从中选出最大值fm和次大值fm；</li><li>步7判断最大值fm和次大值fm之差是否大于阈值fh，且最大值fm是否大于阈值fh，如果大于，则将路段m作为匹配路段，并由定位点向其作投影；否则，暂不对本次定位结果进行匹配，待后面对其进行延时匹配处理；</li><li>步8结束本次匹配</li></ol><p>from : <a href="https://wenku.baidu.com/view/14551bfcfab069dc50220144.html?fr=xueshu_top" target="_blank" rel="noopener">一种适于车辆导航系统的快速地图匹配算法</a>——2003</p><h2 id="key"><a class="markdownIt-Anchor" href="#key"></a> Key:</h2><ul><li><p>出行数据：采用GPS定位，那么是否是WGS-84数据，但是GIS部门的路网数据坐标是什么坐标系下的</p></li><li><p>双线路：</p></li><li><p>当下，现在很多用于MapMatching的方法，大多来自于推理、预测的数学方法，如隐马尔可夫链、贝叶斯模型、神经网络模型等，但在数据结构上的创新比较少，GIS算法，更多要在计算机的基础上，结合比如时空观念、数据特征、拓扑关系等对于GIS相关的基础理论</p></li></ul><h2 id="软件"><a class="markdownIt-Anchor" href="#软件"></a> 软件</h2><ul><li>mapinfo: 当今世界上流行的桌面地理信息系统</li><li>graphhopper: 路径规划库</li><li>arcgis</li><li>openstreetmap——开源地图，简称为OSM</li></ul><p>线要素的匹配，主要通过几何、拓扑或语义相似度来进行识别，其中通过空间距离来进行要素匹配的常用方式有：</p><ul><li>闵可夫斯基距离(Minkowski Distance)</li><li>欧氏距离(Euclidean Distance)</li><li>曼哈顿距离(Manhattan Distance)</li><li>切比雪夫距离(Chebyshev Distance)</li><li>汉明距离(Hamming distance)</li><li>杰卡德相似系数(Jaccard similarity coefficient)</li><li>豪斯多夫距离(Hausdorff Distance)</li><li>弗雷歇距离(Fréchet距离)</li></ul><p>from：<a href="http://www.biaozhu.net/9224.html" target="_blank" rel="noopener">高德地图：地图数据处理之道路匹配篇</a></p><h1 id="隐马尔科夫hmm"><a class="markdownIt-Anchor" href="#隐马尔科夫hmm"></a> 隐马尔科夫HMM</h1><p><strong>HMM有三个典型(canonical)问题</strong>:</p><ul><li><strong>概率计算问题</strong>：已知模型参数，计算某一特定输出序列的概率.通常使用**<a href="http://en.wikipedia.org/wiki/Forward_algorithm" target="_blank" rel="noopener">Forward</a>算法**解决；</li><li><strong>预测问题或者解码（decoding）问题</strong>：已知模型参数，寻找最可能的能产生某一特定输出序列的<strong>隐含状态</strong>的序列。通常使用**<a href="http://en.wikipedia.org/wiki/Viterbi_algorithm" target="_blank" rel="noopener">Viterbi</a>算法**解决；</li><li><strong>学习问题</strong>：已知输出序列，寻找最可能的状态转移以及输出概率。通常使用**<a href="http://en.wikipedia.org/wiki/Baum-Welch_algorithm" target="_blank" rel="noopener">Baum-Welch</a>算法**。</li></ul><h2 id="hmm的几个重要元素"><a class="markdownIt-Anchor" href="#hmm的几个重要元素"></a> HMM的几个重要元素：</h2><ul><li><p>π(startprob_): 隐藏状态初始向量</p><p class="katex-block"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><mi>π</mi><mo>=</mo><mrow><mo fence="true">(</mo><msub><mi>π</mi><mi>i</mi></msub><mo fence="true">)</mo></mrow><mo>:</mo><msub><mi>π</mi><mi>i</mi></msub><mo>=</mo><mi>P</mi><mrow><mo fence="true">(</mo><msub><mi>q</mi><mn>1</mn></msub><mo>=</mo><mi>i</mi><mo fence="true">)</mo></mrow><mo separator="true">,</mo><mn>1</mn><mo>≤</mo><mi>i</mi><mo>≤</mo><mi>N</mi></mrow><annotation encoding="application/x-tex">\pi=\left(\pi_{i}\right): \pi_{i}=P\left(q_{1}=i\right), 1 \leq i \leq N</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.43056em;vertical-align:0em;"></span><span class="mord mathdefault" style="margin-right:0.03588em;">π</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="minner"><span class="mopen delimcenter" style="top:0em;">(</span><span class="mord"><span class="mord mathdefault" style="margin-right:0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:-0.03588em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">i</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mclose delimcenter" style="top:0em;">)</span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.58056em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right:0.03588em;">π</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.31166399999999994em;"><span style="top:-2.5500000000000003em;margin-left:-0.03588em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">i</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathdefault" style="margin-right:0.13889em;">P</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="minner"><span class="mopen delimcenter" style="top:0em;">(</span><span class="mord"><span class="mord mathdefault" style="margin-right:0.03588em;">q</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.30110799999999993em;"><span style="top:-2.5500000000000003em;margin-left:-0.03588em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mord mathdefault">i</span><span class="mclose delimcenter" style="top:0em;">)</span></span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.79549em;vertical-align:-0.13597em;"></span><span class="mord mathdefault">i</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.68333em;vertical-align:0em;"></span><span class="mord mathdefault" style="margin-right:0.10903em;">N</span></span></span></span></span></p></li><li><p>A(transmat_):  状态转移概率矩阵:</p><p class="katex-block"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><mi>A</mi><mo>=</mo><msub><mrow><mo fence="true">[</mo><msub><mi>a</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo fence="true">]</mo></mrow><mrow><mi>N</mi><mo>×</mo><mi>N</mi></mrow></msub><mtext>, 其中 </mtext><msub><mi>a</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo><mi>P</mi><mrow><mo fence="true">(</mo><msub><mi>q</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo>=</mo><mi>j</mi><mo>∣</mo><msub><mi>q</mi><mi>t</mi></msub><mo>=</mo><mi>i</mi><mo fence="true">)</mo></mrow><mo separator="true">,</mo><mn>1</mn><mo>≤</mo><mi>i</mi><mo separator="true">,</mo><mi>j</mi><mo>≤</mo><mi>N</mi></mrow><annotation encoding="application/x-tex">A=\left[a_{i j}\right]_{N \times N} \text {, 其中 } a_{i j}=P\left(q_{t+1}=j \mid q_{t}=i\right), 1 \leq i, j \leq N</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.68333em;vertical-align:0em;"></span><span class="mord mathdefault">A</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1.144139em;vertical-align:-0.394139em;"></span><span class="minner"><span class="minner"><span class="mopen delimcenter" style="top:0em;">[</span><span class="mord"><span class="mord mathdefault">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.311664em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">i</span><span class="mord mathdefault mtight" style="margin-right:0.05724em;">j</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.286108em;"><span></span></span></span></span></span></span><span class="mclose delimcenter" style="top:0em;">]</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.14252299999999996em;"><span style="top:-2.364192em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right:0.10903em;">N</span><span class="mbin mtight">×</span><span class="mord mathdefault mtight" style="margin-right:0.10903em;">N</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.394139em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord text"><span class="mord">, </span><span class="mord cjk_fallback">其中</span><span class="mord"> </span></span><span class="mord"><span class="mord mathdefault">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.311664em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">i</span><span class="mord mathdefault mtight" style="margin-right:0.05724em;">j</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.286108em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathdefault" style="margin-right:0.13889em;">P</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="minner"><span class="mopen delimcenter" style="top:0em;">(</span><span class="mord"><span class="mord mathdefault" style="margin-right:0.03588em;">q</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.301108em;"><span style="top:-2.5500000000000003em;margin-left:-0.03588em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.208331em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mord mathdefault" style="margin-right:0.05724em;">j</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">∣</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right:0.03588em;">q</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2805559999999999em;"><span style="top:-2.5500000000000003em;margin-left:-0.03588em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mord mathdefault">i</span><span class="mclose delimcenter" style="top:0em;">)</span></span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.85396em;vertical-align:-0.19444em;"></span><span class="mord mathdefault">i</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord mathdefault" style="margin-right:0.05724em;">j</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.68333em;vertical-align:0em;"></span><span class="mord mathdefault" style="margin-right:0.10903em;">N</span></span></span></span></span></p></li><li><p>B(emissionprob_): 观测状态概率矩阵</p><p class="katex-block"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><mi>B</mi><mo>=</mo><msub><mrow><mo fence="true">[</mo><msub><mi>b</mi><mi>j</mi></msub><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow><mrow><mi>N</mi><mo>×</mo><mi>M</mi></mrow></msub><mi mathvariant="normal">，</mi><mtext> 其中 </mtext><msub><mi>b</mi><mi>j</mi></msub><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo><mo>=</mo><mi>P</mi><mrow><mo fence="true">(</mo><msub><mi>o</mi><mi>t</mi></msub><mo>=</mo><msub><mi>o</mi><mi>k</mi></msub><mo>∣</mo><msub><mi>q</mi><mi>t</mi></msub><mo>=</mo><mi>j</mi><mo fence="true">)</mo></mrow><mo separator="true">,</mo><mn>1</mn><mo>≤</mo><mi>j</mi><mo>≤</mo><mi>N</mi><mo separator="true">,</mo><mn>1</mn><mo>≤</mo><mi>k</mi><mo>≤</mo><mi>M</mi></mrow><annotation encoding="application/x-tex">B=\left[b_{j}(k)\right]_{N \times M} ， \text { 其中 } b_{j}(k)=P\left(o_{t}=o_{k} \mid q_{t}=j\right), 1 \leq j \leq N, 1 \leq k \leq M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.68333em;vertical-align:0em;"></span><span class="mord mathdefault" style="margin-right:0.05017em;">B</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1.144139em;vertical-align:-0.394139em;"></span><span class="minner"><span class="minner"><span class="mopen delimcenter" style="top:0em;">[</span><span class="mord"><span class="mord mathdefault">b</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.311664em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right:0.05724em;">j</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.286108em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathdefault" style="margin-right:0.03148em;">k</span><span class="mclose">)</span><span class="mclose delimcenter" style="top:0em;">]</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.14252299999999996em;"><span style="top:-2.364192em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right:0.10903em;">N</span><span class="mbin mtight">×</span><span class="mord mathdefault mtight" style="margin-right:0.10903em;">M</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.394139em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord cjk_fallback">，</span><span class="mord text"><span class="mord"> </span><span class="mord cjk_fallback">其中</span><span class="mord"> </span></span><span class="mord"><span class="mord mathdefault">b</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.311664em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right:0.05724em;">j</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.286108em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathdefault" style="margin-right:0.03148em;">k</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathdefault" style="margin-right:0.13889em;">P</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="minner"><span class="mopen delimcenter" style="top:0em;">(</span><span class="mord"><span class="mord mathdefault">o</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2805559999999999em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mord"><span class="mord mathdefault">o</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.33610799999999996em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight" style="margin-right:0.03148em;">k</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">∣</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right:0.03588em;">q</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2805559999999999em;"><span style="top:-2.5500000000000003em;margin-left:-0.03588em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathdefault mtight">t</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mord mathdefault" style="margin-right:0.05724em;">j</span><span class="mclose delimcenter" style="top:0em;">)</span></span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.85396em;vertical-align:-0.19444em;"></span><span class="mord mathdefault" style="margin-right:0.05724em;">j</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.8777699999999999em;vertical-align:-0.19444em;"></span><span class="mord mathdefault" style="margin-right:0.10903em;">N</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.16666666666666666em;"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.83041em;vertical-align:-0.13597em;"></span><span class="mord mathdefault" style="margin-right:0.03148em;">k</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.68333em;vertical-align:0em;"></span><span class="mord mathdefault" style="margin-right:0.10903em;">M</span></span></span></span></span></p></li><li><p>HMM的状态变量数目：N</p></li><li><p>HMM的观察变量数目：M</p></li></ul><p>如果观测序列是一维的，则观测状态的概率密度函数是一维的普通高斯分布。如果观测序列是N维的，则隐藏状态对应的观测状态的概率密度函数是N维高斯分布。高斯分布的概率密度函数参数可以用μμ表示高斯分布的期望向量，Σ表示高斯分布的协方差矩阵。在GaussianHMM类中，“means”用来表示各个隐藏状态对应的高斯分布期望向量μ形成的矩阵，而“covars”用来表示各个隐藏状态对应的高斯分布协方差矩阵Σ形成的三维张量。</p><p>from :<a href="https://www.cnblogs.com/pinard/p/7001397.html" target="_blank" rel="noopener">用hmmlearn学习隐马尔科夫模型HMM</a></p><h2 id="mapmatching与hmm"><a class="markdownIt-Anchor" href="#mapmatching与hmm"></a> MapMatching与Hmm</h2><ul><li>观察变量：从GPS设备中得到的位置信息（经度，纬度）</li><li>隐藏状态：拥有GPS设备的物体（车，人等）实际所在的位置路段。</li><li>观测概率：例如，现测的GPS点离旁边路段上的位置越近，那么这个真实点在这个路段上的概率越大状态。</li><li>状态转移概率：例如，前后两个真实的位置点的距离越近，那么状态转移的概率越大</li></ul><p>在下面相关论文工作中会说明在这几篇论文中其实只用到了预测问题的<a href="http://en.wikipedia.org/wiki/Viterbi_algorithm" target="_blank" rel="noopener">Viterbi</a>算法,下面也会另开一小节具体描述下<a href="http://en.wikipedia.org/wiki/Viterbi_algorithm" target="_blank" rel="noopener">Viterbi</a>算法。</p><p><strong>Map-Matching的两个变量</strong>：</p><ul><li>从GPS设备中得到的位置信息（经度，纬度）：HMM中观察变量；</li><li>拥有GPS设备的物体（车，人等）实际所在的位置：HMM中的隐藏状态变量，实际地图是不知道GPS设备的准确位置的。</li></ul><p>这样就把Map-Matching问题与HMM结合起来了。三个问题在Map-Matching中有用的是两个问题：（1）预测问题；（2）学习问题。</p><p>在论文中，定义的规则要满足人的直观上的感觉，即人的先验知识，主要有以下两种：</p><ul><li>观测概率：观测的GPS点离旁边路段上的位置越近，那么这个真实点在这个路段上的概率越大。</li><li>状态转移概率：这里有两种解决思路：（1）前后两个真实的位置点的距离越近，那么状态转移的概率越大；或者（2）真实路段上的前后两个点的距离与GPS观测的前后两个点的距离越接近，状态转移概率越大。</li></ul><p>from: <a href="https://www.cnblogs.com/mindpuzzle/p/3653043.html" target="_blank" rel="noopener">基于隐马尔科夫模型(HMM)的地图匹配(Map-Matching)算法</a></p><h3 id="barefoot"><a class="markdownIt-Anchor" href="#barefoot"></a> <a href="https://github.com/bmwcarit/barefoot/wiki#stand-alone-servers" target="_blank" rel="noopener">barefoot</a></h3><ul><li>观测概率: 测量位置与其真实位置之间的距离，用于对测量误差进行建模，测量误差用具有一些标准偏差<em>σ</em>的高斯分布来描述（默认为<em>σ = 5 米</em>）。</li><li>转移概率: 用各个位置测量之间的路由距离和视线距离的差异来量化的。转移概率呈负指数分布，速率参数<em>λ</em>（默认为<em>λ = 0.1</em>）是均值的倒数</li></ul><p>==&gt;</p><ul><li><p><em><strong>序列概率</strong></em>： 定义<em>p(s 0 … s t |z 0 … z t )<em>为最可能的序列到达匹配候选</em>s t</em>的概率，称为 s t 的序列概率， 可以通过递归确定</p></li><li><p><em><strong>过滤概率</strong></em>：我们的 HMM 滤波器确定对象当前位置的估计值<em>s̅ t</em>，它是最可能匹配的候选对象<span class="katex"><span class="katex-mathml"><math><semantics><mrow><msub><mi>s</mi><mi>t</mi></msub><mo>∈</mo><msub><mi>S</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">s_t ∈ S_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6891em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathdefault">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2805559999999999em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2777777777777778em;"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2777777777777778em;"></span></span><span class="base"><span class="strut" style="height:0.83333em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathdefault" style="margin-right:0.05764em;">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2805559999999999em;"><span style="top:-2.5500000000000003em;margin-left:-0.05764em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> 给定测量值<code>z0 ... zt</code>，其定义为：<code>s̅t = argmax(st ∈ St) p(st|z0 ... zt)</code>, <code>p(st|z0 ... zt)</code>称为<span class="katex"><span class="katex-mathml"><math><semantics><mrow><msub><mi>s</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">s_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.58056em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathdefault">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2805559999999999em;"><span style="top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathdefault mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span>的过滤概率***，可以递归确定：</p><p><code>p(st|z0...zt) = p(st|zt) · Σ(st-1 ∈ St-1) p(st|st-1) · p(st-1|z0 ... zt-1)</code>。</p></li></ul><p>from: <a href="https://github.com/bmwcarit/barefoot/wiki#stand-alone-servers" target="_blank" rel="noopener">https://github.com/bmwcarit/barefoot/wiki#stand-alone-servers</a></p><h2 id="coding"><a class="markdownIt-Anchor" href="#coding"></a> Coding</h2><blockquote><p>在之前的<a href="http://www.cnblogs.com/pinard/p/6945257.html" target="_blank" rel="noopener">HMM系列</a>中，我们对隐马尔科夫模型HMM的原理以及三个问题的求解方法做了总结。本文我们就从实践的角度用Python的<a href="https://hmmlearn.readthedocs.io/en/latest/api.html#hmmlearn-hmm" target="_blank" rel="noopener">hmmlearn</a>库来学习HMM的使用。sklearn库中将HMM弃用了，新开了一个<code>hmmlearn</code>的新库，安装命令为：<code>pip install hmmlearn</code>，关于hmmlearn的更多资料在<a href="http://hmmlearn.readthedocs.io/" target="_blank" rel="noopener">官方文档</a>有介绍。</p></blockquote><p>hmmlearn实现了三种HMM模型类，按照观测状态是连续状态还是离散状态，可以分为两类。</p><ol><li>GaussianHMM 观测状态连续型且符合高斯分布</li><li>GMMHMM 观测状态连续型且符合混合高斯分布</li><li>MultinomialHMM 观测状态离散型</li></ol><p><strong>HMM主要解决的三个问题</strong><br>假设隐藏状态序列和观测状态序列分别使用Z和X表示，则解决的3个问题可表示为:<br>1.解码问题：已知模型参数和X，估计最可能的Z；维特比算法<br>2.概率问题：已知模型参数和X，估计X出现的概率；向前-向后算法<br>3.学习问题：仅给出X和隐藏层个数，估计模型参数。 B-W算法，通常是经过一定数量的训练以后，得到模型，然后解决问题1和2。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#coding=utf-8</span></span><br><span class="line"><span class="string">'''</span></span><br><span class="line"><span class="string">Created on 2017-12-4</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">解码问题：本例为天气和行为的关系</span></span><br><span class="line"><span class="string">'''</span></span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"><span class="comment"># hmmlearn可以在安装numpy以后，再使用pip install hmmlearn安装</span></span><br><span class="line"><span class="keyword">from</span> hmmlearn <span class="keyword">import</span> hmm</span><br><span class="line"></span><br><span class="line">states = [<span class="string">"Rainy"</span>, <span class="string">"Sunny"</span>]<span class="comment">##隐藏状态</span></span><br><span class="line">n_states = len(states)<span class="comment">##隐藏状态长度</span></span><br><span class="line"></span><br><span class="line">observations = [<span class="string">"walk"</span>, <span class="string">"shop"</span>, <span class="string">"clean"</span>]<span class="comment">##可观察的状态</span></span><br><span class="line">n_observations = len(observations)<span class="comment">##可观察序列的长度</span></span><br><span class="line"></span><br><span class="line">start_probability = np.array([<span class="number">0.6</span>, <span class="number">0.4</span>])<span class="comment">##开始转移概率，即开始是Rainy和Sunny的概率</span></span><br><span class="line"><span class="comment">##隐藏间天气转移混淆矩阵，即Rainy和Sunny之间的转换关系，例如[0,0]表示今天Rainy，明天Rainy的概率</span></span><br><span class="line">transition_probability = np.array([</span><br><span class="line">  [<span class="number">0.7</span>, <span class="number">0.3</span>],</span><br><span class="line">  [<span class="number">0.4</span>, <span class="number">0.6</span>]</span><br><span class="line">])</span><br><span class="line"><span class="comment">##隐藏状态天气和可视行为混淆矩阵，例如[0,0]表示今天Rainy，walk行为的概率为0.1</span></span><br><span class="line">emission_probability = np.array([</span><br><span class="line">  [<span class="number">0.1</span>, <span class="number">0.4</span>, <span class="number">0.5</span>],</span><br><span class="line">  [<span class="number">0.6</span>, <span class="number">0.3</span>, <span class="number">0.1</span>]</span><br><span class="line">])</span><br><span class="line"></span><br><span class="line"><span class="comment">#构建了一个MultinomialHMM模型，这模型包括开始的转移概率，隐藏间天气转换混淆矩阵（transmat），隐藏状态天气和可视行为混淆矩阵emissionprob，对模型参数初始化</span></span><br><span class="line">model = hmm.MultinomialHMM(n_components=n_states)</span><br><span class="line">model.startprob_= start_probability</span><br><span class="line">model.transmat_ = transition_probability</span><br><span class="line">model.emissionprob_ = emission_probability</span><br><span class="line"></span><br><span class="line"><span class="comment">#给出一个可见序列</span></span><br><span class="line">bob_Actions = np.array([[<span class="number">2</span>, <span class="number">0</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">0</span>]]).T</span><br><span class="line"></span><br><span class="line"><span class="comment"># 解决问题1,解码问题,已知模型参数和X，估计最可能的Z； 维特比算法 </span></span><br><span class="line">logprob, weathers = model.decode(bob_Actions, algorithm=<span class="string">"viterbi"</span>)</span><br><span class="line"><span class="keyword">print</span> <span class="string">"Bob Actions:"</span>, <span class="string">", "</span>.join(map(<span class="keyword">lambda</span> x: observations[x], bob_Actions))</span><br><span class="line"><span class="keyword">print</span> <span class="string">"weathers:"</span>, <span class="string">", "</span>.join(map(<span class="keyword">lambda</span> x: states[x], weathers))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">解码问题： 盒子</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">from</span> hmmlearn <span class="keyword">import</span> hmm</span><br><span class="line"></span><br><span class="line">states = [<span class="string">"box 1"</span>, <span class="string">"box 2"</span>, <span class="string">"box3"</span>]</span><br><span class="line">n_states = len(states)</span><br><span class="line"></span><br><span class="line">observations = [<span class="string">"red"</span>, <span class="string">"white"</span>]</span><br><span class="line">n_observations = len(observations)</span><br><span class="line"></span><br><span class="line">start_probability = np.array([<span class="number">0.2</span>, <span class="number">0.4</span>, <span class="number">0.4</span>])</span><br><span class="line"></span><br><span class="line">transition_probability = np.array([</span><br><span class="line">  [<span class="number">0.5</span>, <span class="number">0.2</span>, <span class="number">0.3</span>],</span><br><span class="line">  [<span class="number">0.3</span>, <span class="number">0.5</span>, <span class="number">0.2</span>],</span><br><span class="line">  [<span class="number">0.2</span>, <span class="number">0.3</span>, <span class="number">0.5</span>]</span><br><span class="line">])</span><br><span class="line"></span><br><span class="line">emission_probability = np.array([</span><br><span class="line">  [<span class="number">0.5</span>, <span class="number">0.5</span>],</span><br><span class="line">  [<span class="number">0.4</span>, <span class="number">0.6</span>],</span><br><span class="line">  [<span class="number">0.7</span>, <span class="number">0.3</span>]</span><br><span class="line">])</span><br><span class="line"></span><br><span class="line">model = hmm.MultinomialHMM(n_components=n_states)</span><br><span class="line">model.startprob_=start_probability</span><br><span class="line">model.transmat_=transition_probability</span><br><span class="line">model.emissionprob_=emission_probability</span><br><span class="line"></span><br><span class="line">seen = np.array([[<span class="number">0</span>,<span class="number">1</span>,<span class="number">0</span>]]).T</span><br><span class="line">logprob, box = model.decode(seen, algorithm=<span class="string">"viterbi"</span>)</span><br><span class="line">print(<span class="string">"The ball picked:"</span>, <span class="string">", "</span>.join(map(<span class="keyword">lambda</span> x: observations[x], seen)))</span><br><span class="line">print(<span class="string">"The hidden box"</span>, <span class="string">", "</span>.join(map(<span class="keyword">lambda</span> x: states[x], box)))</span><br><span class="line">(<span class="string">'The ball picked:'</span>, <span class="string">'red, white, red'</span>)</span><br><span class="line">(<span class="string">'The hidden box'</span>, <span class="string">'box3, box3, box3'</span>)</span><br></pre></td></tr></table></figure><h2 id="限制"><a class="markdownIt-Anchor" href="#限制"></a> 限制</h2><ul><li>对于不同的建模选择和输入数据而言，它相对不灵活</li><li>它缩放严重(O(N²)，其中N是每个状态下可能的候选数)</li><li>它不能很好地应对高频观测(请参阅Newson＆Krumm [2])。</li></ul><p>基于(无味)卡尔曼滤波器的新模型——<a href="https://blog.csdn.net/weixin_26713521/article/details/108134220" target="_blank" rel="noopener">https://blog.csdn.net/weixin_26713521/article/details/108134220</a></p><h1 id="help-code"><a class="markdownIt-Anchor" href="#help-code"></a> help Code:</h1><ul><li><a href="https://github.com/bmwcarit/barefoot" target="_blank" rel="noopener">https://github.com/bmwcarit/barefoot</a></li><li><a href="https://github.com/oldrev/mapmatchingkit" target="_blank" rel="noopener">https://github.com/oldrev/mapmatchingkit</a></li></ul><h1 id="网络文章"><a class="markdownIt-Anchor" href="#网络文章"></a> 网络文章</h1><ul><li><a href="https://blog.csdn.net/weixin_42599077/article/details/114801735" target="_blank" rel="noopener">地图匹配实例-几何匹配</a>——效果图</li><li><a href="https://blog.csdn.net/weixin_26713521/article/details/108134220" target="_blank" rel="noopener">定位匹配 模板匹配 地图_什么是地图匹配？</a></li><li><a href="https://blog.csdn.net/Feng512275/article/details/96199675?spm=1001.2014.3001.5502" target="_blank" rel="noopener">出租车轨迹点地图匹配研究</a></li><li><a href="https://blog.csdn.net/yixianfeng41/article/details/73010792" target="_blank" rel="noopener">地图匹配小结</a></li><li><a href="https://www.cnblogs.com/mindpuzzle/p/3653043.html" target="_blank" rel="noopener">基于隐马尔科夫模型(HMM)的地图匹配(Map-Matching)算法</a></li></ul><h1 id="工作安排"><a class="markdownIt-Anchor" href="#工作安排"></a> 工作安排</h1><p>你的工作主要完成什么（概括说明就行），分为那几步，每一步完成什么（概括说明就行），每一步的工作量（预计完成这步工作需要多少个小时），每一步工作预计在什么时候完成（比如 3.5）</p><ul><li>工作包含：地图匹配，将车辆的有序GPS位置数据关联到电子地图的路网上，将GPS坐标下采样序列转换为数字路网路径序列的过程。工作内容：解析过滤点并进行坐标转换-&gt;点过滤(点稀疏)-&gt;地图匹配算法-&gt;点映射</li></ul><table><thead><tr><th>内容</th><th>预估时间</th></tr></thead><tbody><tr><td>查阅地图匹配算法相关的论文-&gt;找到解决方案(16小时)</td><td>2022年2月28日</td></tr><tr><td>学习隐马尔科夫模型相关理论知识 (6小时)</td><td>2022年3月2日</td></tr><tr><td>找寻隐马尔科夫模型资料、代码(8小时)——barefoot、graphhopper</td><td>2022年3月4日</td></tr><tr><td>熟悉项目中有关地图对象的代码（4小时）</td><td>2022年3月5日</td></tr><tr><td>移植隐马尔科夫模型到项目中，目前参考：开源<a href="https://github.com/bmwcarit/barefoot/wiki#stand-alone-servers" target="_blank" rel="noopener">barefoot</a>实现：熟悉代码(16小时)、坐标转换（3小时）、输入输出数据格式改造(10小时)、适配类(56小时)、效果检验（10小时）</td><td>2022年3月20日</td></tr><tr><td>点过滤（24小时）: 栅格化（16小时） + 双队列（8小时）</td><td>2022年3月24日</td></tr><tr><td>点映射（6小时）</td><td>2022年3月25日</td></tr></tbody></table><h1 id="库"><a class="markdownIt-Anchor" href="#库"></a> 库：</h1><ul><li><a href="https://developers.arcgis.com/javascript/latest/api-reference/esri-geometry-geometryEngine.html" target="_blank" rel="noopener">geometryEngine</a></li><li>net.sf.geographiclib</li><li>com.esri.core.geometry——QuadTree</li><li>graphhopper——org.locationtech.jts.geom  ==&gt; Envelope --&gt; findCandidateSnapsInBBox</li></ul><hr><h1 id="简单版实现思路"><a class="markdownIt-Anchor" href="#简单版实现思路"></a> 简单版实现思路</h1><ol><li>找到起始点O和终点D，以两点为半径画圆，把所有范围内的NormalEdge全部加入List（只要起始点or终止点有一个在范围内）</li><li>确定第一条===&gt; 找离第一个GPS点最近的边</li><li>likelyRoute.forEach从上一条的lowerNormalEdge中(保证拓扑可达连接)，从中选择得分最高（距离和方向）的边作为后续边 。  如果没有lowerNormalEdge（如果所有剩余的点都能映射到最后的这条边上，则认为是终点，此时不在意死路）， 平常如果是死路，则放弃这条序列路径。TODO：没有后续路径时是否要考虑增加可能边集？</li><li>到达终点边结束</li><li>从likelyRoute中选择得分最高的路径</li></ol><p>用到的结构：</p><ul><li><code>likelyRoute: list[list]</code>, 存放多条可能</li></ul>]]></content>
    
    <summary type="html">
    
      
      
        &lt;h1 id=&quot;地图匹配&quot;&gt;&lt;a class=&quot;markdownIt-Anchor&quot; href=&quot;#地图匹配&quot;&gt;&lt;/a&gt; 地图匹配&lt;/h1&gt;
&lt;p&gt;概念：&lt;/p&gt;
&lt;blockquote&gt;
&lt;ul&gt;
&lt;li&gt;地图匹配（Map-Matching）是将运营车辆的有序的GPS位置关联到
      
    
    </summary>
    
    
  </entry>
  
  <entry>
    <title>私有Gitlab配置SSH连接</title>
    <link href="https://nymrli.top/2022/02/25/%E7%A7%81%E6%9C%89Gitlab%E9%85%8D%E7%BD%AESSH%E8%BF%9E%E6%8E%A5/"/>
    <id>https://nymrli.top/2022/02/25/私有Gitlab配置SSH连接/</id>
    <published>2022-02-25T05:38:19.000Z</published>
    <updated>2022-02-25T06:04:05.992Z</updated>
    
    <content type="html"><![CDATA[<blockquote><p>SSH连接并不是像<a href="https://so.csdn.net/so/search?q=http&amp;spm=1001.2101.3001.7020" target="_blank" rel="noopener">http</a>连接一样，输入用户名和密码就可以了。SSH连接需要一些额外的配置</p></blockquote><h1 id="生成ssh秘钥和公钥"><a class="markdownIt-Anchor" href="#生成ssh秘钥和公钥"></a> 生成SSH秘钥和公钥</h1><h2 id="查询有效邮箱"><a class="markdownIt-Anchor" href="#查询有效邮箱"></a> 查询有效邮箱</h2><p>首先要确定使用的邮箱，是不是自己认为的邮箱，比如我以为是<code>chenli@sucsoft.com</code>结果，却发现是<code>399-chenli@sucsoft.com</code>，那么按照自认为的邮箱申请的keys那自然不会work。——一般情况不会有歧义邮箱出现，但我这边确实跟想的不太一样</p><h3 id="查询方法一新建项目"><a class="markdownIt-Anchor" href="#查询方法一新建项目"></a> 查询方法(一)——新建项目</h3><p>创建空的仓库，提示中提及的邮箱就是有效邮箱</p><p><img src="/2022/02/25/私有Gitlab配置SSH连接/gitlab-ssh.jpg" alt="gitlab-ssh"></p><h3 id="查询方法二个人preference"><a class="markdownIt-Anchor" href="#查询方法二个人preference"></a> 查询方法(二)——个人Preference</h3><ol><li>打开<code>User Settings-&gt;Profile</code>，其中有<code>Email</code>、<code>public email</code>和<code>commit email</code>三个邮箱，其中跟我们开发命令比较有关的就是这个Commit Email，之前我的默认是<code>399-chenli@sucsoft.com</code>，这边要设置自己想要的，比如<code>chenli@sucsoft.com</code></li></ol><p><img src="/2022/02/25/私有Gitlab配置SSH连接/email_settings.jpg" alt="email_settings"></p><p>▲选择完成后，滑到最下面选择Update profile settings</p><ol start="2"><li>打开<code>User Settings-&gt;Emails</code>可以看到，<code>chenli@sucsoft@sucsoft.com</code>的标签中有commit email，即设置成功</li></ol><p><img src="/2022/02/25/私有Gitlab配置SSH连接/preference_email.jpg" alt="preference_email"></p><h2 id="gitlab设置公钥"><a class="markdownIt-Anchor" href="#gitlab设置公钥"></a> Gitlab设置公钥</h2><p>确保邮箱是有效的后就可以生成秘钥了。</p><ol><li>设置git信息</li></ol><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 1. 设置git信息</span></span><br><span class="line">git config --global user.name <span class="string">"chenli"</span></span><br><span class="line">git config --global user.email <span class="string">"chenli@sucsoft.com"</span></span><br><span class="line"><span class="comment"># 2. 生成秘钥和公钥</span></span><br><span class="line">$ ssh-keygen -t rsa -C chenli@sucsoft.com</span><br><span class="line">Generating public/private rsa key pair.</span><br><span class="line">Enter file <span class="keyword">in</span> <span class="built_in">which</span> to save the key (/c/Users/mrli/.ssh/id_rsa):</span><br><span class="line">Enter passphrase (empty <span class="keyword">for</span> no passphrase):</span><br><span class="line">Enter same passphrase again:</span><br><span class="line">Your identification has been saved <span class="keyword">in</span> /c/Users/mrli/.ssh/id_rsa</span><br><span class="line">Your public key has been saved <span class="keyword">in</span> /c/Users/mrli/.ssh/id_rsa.pub</span><br><span class="line">The key fingerprint is:</span><br><span class="line">SHA256:zKPeKkm0bNzDqZcHUEDcvvDqPG2znRqDrD30yoBb8MM chenli@sucsoft.com</span><br><span class="line">The key<span class="string">'s randomart image is:</span></span><br><span class="line"><span class="string">+---[RSA 3072]----+</span></span><br><span class="line"><span class="string">|   ooo           |</span></span><br><span class="line"><span class="string">|    . o          |</span></span><br><span class="line"><span class="string">|     o           |</span></span><br><span class="line"><span class="string">|    + .o         |</span></span><br><span class="line"><span class="string">|.  + B oS        |</span></span><br><span class="line"><span class="string">| = .B.O. .       |</span></span><br><span class="line"><span class="string">|. E+o*++         |</span></span><br><span class="line"><span class="string">| o BB.B=..       |</span></span><br><span class="line"><span class="string">|. ..*B=*+        |</span></span><br><span class="line"><span class="string">+----[SHA256]-----+</span></span><br></pre></td></tr></table></figure><p>根据命令输出结果可以看到，生成的结果在<code>C:\Users\mrli\.ssh</code>路径下，有<code>id_rsa</code>、<code>id_rsa.pub</code>，其中<code>id_rsa.pub</code>为公钥。</p><ol start="2"><li><p>将SSH公钥添加到GitHub账户</p><ol><li>复制文件路径内.pub后缀里的内容</li><li>进入SSH Keys的设置页面（登录GitHub，在右上角头像点击设置）</li></ol><p><img src="/2022/02/25/私有Gitlab配置SSH连接/add_ssh.jpg" alt="add_ssh"></p><ol start="3"><li>粘贴后，Title会自动提取生成，点击<code>Add key</code>则完成添加</li></ol></li></ol><h2 id="测试"><a class="markdownIt-Anchor" href="#测试"></a> 测试</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ ssh -T git@gitlab.sucsoft.com</span><br><span class="line">Welcome to GitLab, @chenli!</span><br></pre></td></tr></table></figure><h1 id="more"><a class="markdownIt-Anchor" href="#more"></a> More</h1><h2 id="配置多个git信息"><a class="markdownIt-Anchor" href="#配置多个git信息"></a> 配置多个Git信息</h2><ol><li>将默认的<code>id_rsa</code>和<code>id_rsa.pub</code>改成指定的如<code>id_rsa_sucsoft.pub</code></li><li><code>.ssh/</code>路径下添加<code>config</code>配置文件，对应填写</li></ol><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">Host github.com                 </span><br><span class="line">    HostName github.com</span><br><span class="line">    IdentityFile C:\\Users\\mrli\\.ssh\\id_rsa_github</span><br><span class="line">    PreferredAuthentications publickey</span><br><span class="line">    User Freedomisgood</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">Host gitlab.sucsoft.com</span><br><span class="line">    HostName gitlab.sucsoft.com</span><br><span class="line">    IdentityFile C:\\Users\\mrli\\.ssh\\id_rsa_sucsoft</span><br><span class="line">    PreferredAuthentications publickey</span><br><span class="line">    User chenli</span><br></pre></td></tr></table></figure><p>注意：Host后写的就是<code>ssh -T git@gitlab.sucsoft.com</code>中@后面的内容，如果写的是<code>sucsoft</code>，则写成<code>ssh -T git@sucsoft</code>，但是这样其实跟Gitlab页面中直接提供的不一样: <code>git@gitlab.sucsoft.com:suc-frame/xxxx.git</code>，所以最好还是规范的写<code>gitlab.sucsoft.com</code></p><p>★配置完成后最终文件夹中的内容：</p><p><img src="/2022/02/25/私有Gitlab配置SSH连接/more-1.jpg" alt="more-1"></p><p>注： known_host是自动生成的，如果第一次SSH连接网站，则会将host添加到这个文件中</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;blockquote&gt;
&lt;p&gt;SSH连接并不是像&lt;a href=&quot;https://so.csdn.net/so/search?q=http&amp;amp;spm=1001.2101.3001.7020&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;http&lt;/a&gt;连
      
    
    </summary>
    
    
      <category term="git" scheme="https://nymrli.top/tags/git/"/>
    
  </entry>
  
  <entry>
    <title>Golang protoc的使用</title>
    <link href="https://nymrli.top/2022/02/22/protoc%E7%9A%84%E4%BD%BF%E7%94%A8/"/>
    <id>https://nymrli.top/2022/02/22/protoc的使用/</id>
    <published>2022-02-22T04:04:33.000Z</published>
    <updated>2022-02-22T06:11:07.536Z</updated>
    
    <content type="html"><![CDATA[<blockquote><p>使用Grpc需要有protoc的支持, 但默认安装的<code>/Go/bin</code>目录下只有<code>go.exe</code>、<code>gofmt.exe</code>两个可执行文件，因此要使用<code>protoc</code>的话，需要自己去下载</p></blockquote><ol><li><p>在<a href="https://links.jianshu.com/go?to=https%3A%2F%2Fgithub.com%2Fprotocolbuffers%2Fprotobuf%2Freleases" target="_blank" rel="noopener">https://github.com/protocolbuffers/protobuf/releases</a>中下载Windows64的压缩包，为<code>$protoc -go_out=. *.proto</code>使用到的命令可执行文件</p></li><li><p>编写<code>.proto</code>文件， 如下所示</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">syntax = &quot;proto3&quot;;</span><br><span class="line"></span><br><span class="line"># 注意, 这边使用的是go_package, 类似的还有java_package, 后面的参数是output_path和包名</span><br><span class="line">option go_package = &quot;.;geecachepb&quot;;</span><br><span class="line"></span><br><span class="line">message Request&#123;</span><br><span class="line">  string group = 1;</span><br><span class="line">  string key = 2;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">message Response&#123;</span><br><span class="line">  bytes value = 1;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">service GroupCache&#123;</span><br><span class="line">  rpc Get(Request) returns (Response);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li><li><p>使用<code>protoc -go_out=. *.proto</code>命令，将<code>.proto</code>转换成要使用的go文件</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Code generated by protoc-gen-go. DO NOT EDIT.</span></span><br><span class="line"><span class="comment">// versions:</span></span><br><span class="line"><span class="comment">// protoc-gen-go v1.27.1</span></span><br><span class="line"><span class="comment">// protoc        v3.19.4</span></span><br><span class="line"><span class="comment">// source: geecachepb.proto</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">package</span> geecachepb</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line">protoreflect <span class="string">"google.golang.org/protobuf/reflect/protoreflect"</span></span><br><span class="line">protoimpl <span class="string">"google.golang.org/protobuf/runtime/protoimpl"</span></span><br><span class="line">reflect <span class="string">"reflect"</span></span><br><span class="line">sync <span class="string">"sync"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">const</span> (</span><br><span class="line"><span class="comment">// Verify that this generated code is sufficiently up-to-date.</span></span><br><span class="line">_ = protoimpl.EnforceVersion(<span class="number">20</span> - protoimpl.MinVersion)</span><br><span class="line"><span class="comment">// Verify that runtime/protoimpl is sufficiently up-to-date.</span></span><br><span class="line">_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - <span class="number">20</span>)</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> Request <span class="keyword">struct</span> &#123;</span><br><span class="line">state         protoimpl.MessageState</span><br><span class="line">sizeCache     protoimpl.SizeCache</span><br><span class="line">unknownFields protoimpl.UnknownFields</span><br><span class="line"></span><br><span class="line">Group <span class="keyword">string</span> <span class="string">`protobuf:"bytes,1,opt,name=group,proto3" json:"group,omitempty"`</span></span><br><span class="line">Key   <span class="keyword">string</span> <span class="string">`protobuf:"bytes,2,opt,name=key,proto3" json:"key,omitempty"`</span></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// ... 省略更多</span></span><br></pre></td></tr></table></figure></li><li><p><code>go get -u github.com/golang/protobuf/protoc-gen-go</code>安装使用<code>xxxx.pb.go</code>文件的库</p></li><li><p>在业务代码中使用</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> PeerGetter <span class="keyword">interface</span> &#123;</span><br><span class="line">Get(in *pb.Request, out *pb.Response) error</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(p *HttpPool)</span> <span class="title">ServeHTTP</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> &#123;</span><br><span class="line">    <span class="comment">// ...</span></span><br><span class="line">    body, err := proto.Marshal(&amp;pb.Response&#123;Value: view.ByteSlice()&#125;)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">        http.Error(w, err.Error(), http.StatusInternalServerError)</span><br><span class="line">        <span class="keyword">return</span></span><br><span class="line">    &#125;</span><br><span class="line">    <span class="comment">// ...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(h *httpGetter)</span> <span class="title">Get</span><span class="params">(in *pb.Request, out *pb.Response)</span> <span class="title">error</span></span> &#123;</span><br><span class="line">u := fmt.Sprintf(</span><br><span class="line"><span class="string">"%v%v/%v"</span>,</span><br><span class="line">h.baseURL,</span><br><span class="line">url.QueryEscape(in.GetGroup()),</span><br><span class="line">url.QueryEscape(in.GetKey()),</span><br><span class="line">)</span><br><span class="line">res, err := http.Get(u)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line"><span class="keyword">return</span> err</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">defer</span> res.Body.Close()</span><br><span class="line"><span class="keyword">if</span> res.StatusCode != http.StatusOK &#123;</span><br><span class="line"><span class="keyword">return</span> fmt.Errorf(<span class="string">"server return %v"</span>, res.Status)</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">bytes, err := ioutil.ReadAll(res.Body)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line"><span class="keyword">return</span> err</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> err = proto.Unmarshal(bytes, out); err != <span class="literal">nil</span> &#123;</span><br><span class="line"><span class="keyword">return</span> fmt.Errorf(<span class="string">"decoding response body: %v"</span>, err)</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h2 id="推荐风格"><a class="markdownIt-Anchor" href="#推荐风格"></a> 推荐风格</h2><ul><li>文件(Files)<ul><li>文件名使用小写下划线的命名风格，例如 lower_snake_case.proto</li><li>每行不超过 80 字符</li><li>使用 2 个空格缩进</li></ul></li><li>包(Packages)<ul><li>包名应该和目录结构对应，例如文件在<code>my/package/</code>目录下，包名应为 <code>my.package</code></li></ul></li><li>消息和字段(Messages &amp; Fields)<ul><li>消息名使用首字母大写驼峰风格(CamelCase)，例如<code>message StudentRequest { ... }</code></li><li>字段名使用小写下划线的风格，例如 <code>string status_code = 1</code></li><li>枚举类型，枚举名使用首字母大写驼峰风格，例如 <code>enum FooBar</code>，枚举值使用全大写下划线隔开的风格(CAPITALS_WITH_UNDERSCORES )，例如 FOO_DEFAULT=1</li></ul></li><li>服务(Services)<ul><li>RPC 服务名和方法名，均使用首字母大写驼峰风格，例如<code>service FooService{ rpc GetSomething() }</code></li></ul></li></ul></li></ol>]]></content>
    
    <summary type="html">
    
      
      
        &lt;blockquote&gt;
&lt;p&gt;使用Grpc需要有protoc的支持, 但默认安装的&lt;code&gt;/Go/bin&lt;/code&gt;目录下只有&lt;code&gt;go.exe&lt;/code&gt;、&lt;code&gt;gofmt.exe&lt;/code&gt;两个可执行文件，因此要使用&lt;code&gt;protoc&lt;/code
      
    
    </summary>
    
    
      <category term="Go" scheme="https://nymrli.top/tags/Go/"/>
    
  </entry>
  
</feed>
