20170430 python爬蟲攻防戰-攻防與金融大數據分析班
26 likes5,323 views
本文件主要介绍如何利用 Python 爬虫技术从上市公司获取 EPS 数据和财报信息，包括使用 requests 库抓取网页数据、使用 pandas 处理数据，以及如何利用 Google Trends 数据分析金融市场行为。此外，还探讨了如何通过 Selenium 模拟用户行为进行数据抓取，同时介绍了常用的数据可视化技术。
![dfs = pd.read_html(soup.prettify('utf-8'), encoding= 'utf-8', header=0)
dfall = []
for df in dfs:
dfall.append(df[df['公司代號'].str.match('^[0-9]+')])
只抓取公司代號有數字的
11
刪除掉不必要的列](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-11-320.jpg)
![stockdf[stockdf['營業收入']==stockdf['營業收入'].max()]
猜猜誰是獲利王
13](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-13-320.jpg)
![df1 = stockdf.groupby(['產業別'], sort=False)['營業
收入'].max()
print df1
根據產業別統計
15](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-15-320.jpg)
![def getEPS(year, season):
dfall = []
payload = {
'encodeURIComponent':'1',
'step':'1',
'firstin':'1',
'TYPEK':'sii',
'code':'',
'year':year,
'season':season
}
res = requests.post('http://mops.twse.com.tw/mops/web/ajax_t163sb19', data=payload)
res.encoding= 'utf-8'
soup = bs(res.text)
dfs = pd.read_html(soup.prettify('utf-8'), encoding= 'utf-8', header=0)
for df in dfs:
dfall.append(df[df['公司代號'].str.match('^[0-9]+')])
return dfall
定義函式
19](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-19-320.jpg)
![dftotal = []
seasons = ['01','02','03','04']
for year in range(102,105):
for season in seasons:
dftotal.extend(getEPS(year, season))
批次執行102年到105 年的資料
20](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-20-320.jpg)
![ stockdf[stockdf['公司代號'] == '2330']
篩選出台積電股票
22](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-22-320.jpg)
![df2330 = stockdf[stockdf['公司代號'] == '2330']
df2330.plot(kind='line', rot=0)
繪製折線圖
23](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-23-320.jpg)
![payload = {
'__EVENTTARGET':'',
'__EVENTARGUMENT':'',
'__LASTFOCUS':'',
'RadioButton_Normal':'RadioButton_Normal',
'TextBox_Stkno':'2330',
'CaptchaControl1':'E3QL8'
'btnOK':'查詢'
}
for inp in soup.select('input[type==hidden]'):
payload[inp['id']] = inp['value']
r3 = rs.post('http://bsr.twse.com.tw/bshtm/bsMenu.aspx', data=payload, headers = headers)
需要POST 資訊
39
必須從上一頁的Hidden Value 擷取
如何破解Captcha](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-39-320.jpg)
![data['debt_mavg'] = pd.rolling_mean(data.debt, 3)
data.head()
算出Search Word 三週的移動平均
49
debt 的三週移動平均](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-49-320.jpg)
![data['debt_mavg'] = data.debt_mavg.shift(1)
data.head(10)
將三週平均資料平移一天
50](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-50-320.jpg)
![data['order'] = 0
# 黃金交叉做多
data['order'][data.debt > data.debt_mavg] = -1
# 死亡交叉做空
data['order'][data.debt < data.debt_mavg] = 1
data.head(10)
設定買賣策略
51
採取隔週買賣動作
e.g. 如果dept > dept_mavg
當週第一天買進道瓊，次
週第一天交易日末賣出](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-51-320.jpg)
![data['ret_djia'] = data.djia.pct_change()
data.head()
計算道瓊報酬
52](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-52-320.jpg)
![ 報酬t是根據t-1週計算的，但買賣策略是發生在ｔ
週，但在t+1周賣出，因此必須要將報酬平移一日
data['ret_djia'] = data['ret_djia'].shift(-1)
根據買賣日調整
53](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-53-320.jpg)
![data['ret_google'] = data.order * data.ret_djia
data.head(10)
計算根據Search Term 買賣的報酬
54](https://image.slidesharecdn.com/20170430-python-170429224724/85/20170430-python-54-320.jpg)
20170430 python爬蟲攻防戰-攻防與金融大數據分析班
- 2. 關於我 大數軟體有限公司創辦人 前趨勢科技工程師 ywchiu.com 大數學堂 http://www.largitdata.com/ 粉絲頁 https://www.facebook.com/largitdata R for Data Science Cookbook https://www.packtpub.com/big-data-and- business-intelligence/r-data-science-cookbook Machine Learning With R Cookbook https://www.packtpub.com/big-data-and- business-intelligence/machine-learning-r- cookbook 2
- 3. Machine Learning With R Cookbook (机器学习与R语言实战) & R for Data Science Cookbook Author: David (YU-WEI CHIU) Chiu 3
- 6. 請求 Request 回應 Response 網頁網頁連結器 Web Connector 資料剖析 Data Parser 資料中心 Data Center 爬蟲是怎麼運作的 6
- 8. 進入公開資訊觀測站 8
- 10. import requests payload = { 'encodeURIComponent':'1', 'step':'1', 'firstin':'1', 'TYPEK':'sii', 'code':'', 'year':'104', 'season':'03' } res = requests.post('http://mops.twse.com.tw/mops/web/ajax_t163sb19', data=payload) res.encoding= 'utf-8' soup = bs(res.text) 讀取EPS資料 10
- 11. dfs = pd.read_html(soup.prettify('utf-8'), encoding= 'utf-8', header=0) dfall = [] for df in dfs: dfall.append(df[df['公司代號'].str.match('^[0-9]+')]) 只抓取公司代號有數字的 11 刪除掉不必要的列
- 12. stockdf = pd.concat(dfall, ignore_index=True) 列出所有公司第三季的EPS 12
- 15. df1 = stockdf.groupby(['產業別'], sort=False)['營業 收入'].max() print df1 根據產業別統計 15
- 17. 切換到matplotlib 設定目錄下 C:Anaconda2Libsite-packagesmatplotlibmpl-data 修改matplotlibrc (將註解#拿掉) font.family : sans-serif font.sans-serif : Microsoft YaHei, … 如何讓matplotlib 出現中文? 17 增加微軟雅黑字體
- 18. zhfont = mpl.font_manager.FontProperties( fname='C:UsersUsertwmtmTW.ttf') plt.title('中文', fontproperties=zhfont) 指定中文字型 18 Windows10 要改成ttc
- 19. def getEPS(year, season): dfall = [] payload = { 'encodeURIComponent':'1', 'step':'1', 'firstin':'1', 'TYPEK':'sii', 'code':'', 'year':year, 'season':season } res = requests.post('http://mops.twse.com.tw/mops/web/ajax_t163sb19', data=payload) res.encoding= 'utf-8' soup = bs(res.text) dfs = pd.read_html(soup.prettify('utf-8'), encoding= 'utf-8', header=0) for df in dfs: dfall.append(df[df['公司代號'].str.match('^[0-9]+')]) return dfall 定義函式 19
- 20. dftotal = [] seasons = ['01','02','03','04'] for year in range(102,105): for season in seasons: dftotal.extend(getEPS(year, season)) 批次執行102年到105 年的資料 20
- 21. stockdf = pd.concat(dftotal, ignore_index=True) len(stockdf) 合併所有的Data Frame 21
- 22. stockdf[stockdf['公司代號'] == '2330'] 篩選出台積電股票 22
- 23. df2330 = stockdf[stockdf['公司代號'] == '2330'] df2330.plot(kind='line', rot=0) 繪製折線圖 23
- 24. 圖表資料抓取 24
- 27. import requests headers = { 'X-Requested-With':'XMLHttpRequest', 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36', } rs = requests.session() res = rs.get('https://statementdog.com/analysis/analysis_ajax/2330/2011/1/20 16/4/1', headers = headers) json = res.text 抓取圖表資料 27
- 32. 匯出Selenium 腳本 32
- 35. # -*- coding: utf-8 -*- from selenium import webdriver from bs4 import BeautifulSoup import unittest, time, re driver = webdriver.Firefox() driver.implicitly_wait(30) driver.get("http://justdata.yuanta.com.tw/z/zk/zk00-f.asp") soup = BeautifulSoup(driver.page_source) print soup driver.close() 執行Selenium 腳本 35 需要執行pip install selenium 可以切換成ChromeDriver
- 36. 買賣日報表資料抓取 36
- 38. headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' } rs = requests.session() r2 = rs.get('http://bsr.twse.com.tw/bshtm/bsMenu.aspx', headers = headers) 抓取bsMenu.aspx 38 使用GET 抓取bsMenu.aspx
- 39. payload = { '__EVENTTARGET':'', '__EVENTARGUMENT':'', '__LASTFOCUS':'', 'RadioButton_Normal':'RadioButton_Normal', 'TextBox_Stkno':'2330', 'CaptchaControl1':'E3QL8' 'btnOK':'查詢' } for inp in soup.select('input[type==hidden]'): payload[inp['id']] = inp['value'] r3 = rs.post('http://bsr.twse.com.tw/bshtm/bsMenu.aspx', data=payload, headers = headers) 需要POST 資訊 39 必須從上一頁的Hidden Value 擷取 如何破解Captcha
- 40. OCR 辨認裡面數字 pytesser https://code.google.com/p/pytesser/ ocropus https://code.google.com/p/ocropus/ Google Vision API https://cloud.google.com/vision/ 40 大數學堂 http://largitdata.com/course/37 http://largitdata.com/course/38
- 41. r4 = rs.get('http://bsr.twse.com.tw/bshtm/bsContent.aspx?v=t', headers = headers) print r4.text 使用GET 取得分點進出資訊 41
- 44. Google Trend 包含使用的搜尋行為，是否可以利 用使用者的搜尋行為判斷股票漲跌？ Google Trend 44
- 45. http://www.nature.com/articles/srep01684 使用Pandas 實作論文 45 Preis, T., Moat, H. S., & Stanley, H. E. (2013). Quantifying trading behavior in financial markets using Google Trends. Scientific reports, 3.
- 46. Python for Data Analysis 源自於R Table-Like 格式 提供高效能、簡易使用的資料格式(Data Frame)讓使用 者可以快速操作及分析資料 Pandas 46 可參照 pandas_in_finance.ipynb
- 47. 讀取 GoogleTrendsData import pandas as pd data = pd.read_csv('data/GoogleTrendsData.csv', index_col='Date', parse_dates=True) data.head() 讀取Google Trend 資料 47
- 49. data['debt_mavg'] = pd.rolling_mean(data.debt, 3) data.head() 算出Search Word 三週的移動平均 49 debt 的三週移動平均
- 51. data['order'] = 0 # 黃金交叉做多 data['order'][data.debt > data.debt_mavg] = -1 # 死亡交叉做空 data['order'][data.debt < data.debt_mavg] = 1 data.head(10) 設定買賣策略 51 採取隔週買賣動作 e.g. 如果dept > dept_mavg 當週第一天買進道瓊，次 週第一天交易日末賣出
- 53. 報酬t是根據t-1週計算的，但買賣策略是發生在ｔ 週，但在t+1周賣出，因此必須要將報酬平移一日 data['ret_djia'] = data['ret_djia'].shift(-1) 根據買賣日調整 53
- 54. data['ret_google'] = data.order * data.ret_djia data.head(10) 計算根據Search Term 買賣的報酬 54
- 55. (1 + data.ret_google).cumprod().plot(); (1 + data.ret_djia).cumprod().plot(); 觀察資金變化 55 it=(it−1+it−1⋅rt)=(1+rt)⋅it−1,i0=1 利用Search Term 做決策比 隨機買賣報酬要好很多
- 56. 試著用 Google Trend 對「流感」的搜尋擬定對康 那香(9919)的投資策略？ 範例: 流感 v.s.康那香(9919) 56 可參照 googletrend_flu.ipynb
- 57. 57