Sql or no sql, that is the question
5 likes1,013 views
该文档探讨了 SQL 和 NoSQL 数据库的比较，强调了各自的特点和适用场景。NoSQL 数据库因其高性能、高可扩展性和无模式特性而受到青睐，适用于处理大量数据和动态应用。文中还讨论了数据一致性、分布模型和常见的 NoSQL 类型，以及在选择数据库时应考虑的因素。
![聚合資料範例
// in customers
{
“id":1,
“name”:"Martin",
“billingAddress":[{"city":"Chicago"}]
}
// in orders
{
“id":99,
“customerId":1,
“orderItems":[
{
“productId":27,
"price": 32.45,
"productName": "NoSQL Distilled”
}
],
“shippingAddress":[{"city":"Chicago"}]
“orderPayment":[
{
“ccinfo":"1000-1000-1000-1000",
“txnId":"abelif879rft",
"billingAddress": {"city": “Chicago”}
}
],
}
範例出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫](https://image.slidesharecdn.com/sqlornosqlthatisthequestion-150808025921-lva1-app6892/85/Sql-or-no-sql-that-is-the-question-12-320.jpg)
![[插播] Scaling MongoDB
圖⽚片出處：http://leehom59.blogspot.tw/2011/11/mongo-db-sharding.html](https://image.slidesharecdn.com/sqlornosqlthatisthequestion-150808025921-lva1-app6892/85/Sql-or-no-sql-that-is-the-question-18-320.jpg)
![db.runCommand({
mapreduce: "DenormAggCollection",
query: {
filter1: { '$in': [ 'A', 'B' ] },
filter2: 'C',
filter3: { '$gt': 123 }
},
map: function() { emit(
{ d1: this.Dim1, d2: this.Dim2 },
{ msum: this.measure1, recs: 1, mmin: this.measure1,
mmax: this.measure2 < 100 ? this.measure2 : 0 }
);},
reduce: function(key, vals) {
var ret = { msum: 0, recs: 0, mmin: 0, mmax: 0 };
for(var i = 0; i < vals.length; i++) {
ret.msum += vals[i].msum;
ret.recs += vals[i].recs;
if(vals[i].mmin < ret.mmin) ret.mmin = vals[i].mmin;
if((vals[i].mmax < 100) && (vals[i].mmax > ret.mmax))
ret.mmax = vals[i].mmax;
}
return ret;
},
finalize: function(key, val) {
val.mavg = val.msum / val.recs;
return val;
},
out: 'result1',
verbose: true
});
db.result1.
find({ mmin: { '$gt': 0 } }).
sort({ recs: -1 }).
skip(4).
limit(8);
SELECT
Dim1, Dim2,
SUM(Measure1) AS MSum,
COUNT(*) AS RecordCount,
AVG(Measure2) AS MAvg,
MIN(Measure1) AS MMin
MAX(CASE
WHEN Measure2 < 100
THEN Measure2
END) AS MMax
FROM DenormAggTable
WHERE (Filter1 IN (’A’,’B’))
AND (Filter2 = ‘C’)
AND (Filter3 > 123)
GROUP BY Dim1, Dim2
HAVING (MMin > 0)
ORDER BY RecordCount DESC
LIMIT 4, 8
1
2
3
4
5
1
7
6
1
2
3
4
5
Grouped dimension columns are pulled
out as keys in the map function,
reducing the size of the working set.
Measures must be manually aggregated.
Aggregates depending on record counts
must wait until finalization.
Measures can use procedural logic.
Filters have an ORM/ActiveRecord-
looking style.
6 Aggregate filtering must be applied to
the result set, not in the map/reduce.
7 Ascending: 1; Descending: -1
Revision4,Created2010-03-06
RickOsborne,rickosborne.org
mySQL MongoDB
圖⽚片出處：http://nosql.mypopescu.com/post/392418792/translate-sql-to-mongodb-mapreduce](https://image.slidesharecdn.com/sqlornosqlthatisthequestion-150808025921-lva1-app6892/85/Sql-or-no-sql-that-is-the-question-46-320.jpg)
Sql or no sql, that is the question
- 1. SQL or NoSQL, that is the question. Hugo 2015/8/6
- 4. 關聯式資料庫的特點 • 儲存持久化資料 • 透過事務(Transaction)解決並發(Concurrency) • 應⽤用程式共享資料庫 • 近乎標準的模型與操作⽅方式(SQL)
- 5. Web2.0 讓 SQL 疲於奔命 • High Performance - 動態⾴頁⾯面加⼤大資料庫負擔 • Huge Storage - SNS 網站海量資料 • High Scalability - ⽤用⼾戶量、訪問量激增 • High Availability - 服務不能間斷
- 6. NoSQL 登場 • 2009 NoSQL Meetup • 不使⽤用關係模型 • 在集群(Cluster)中運⾏行良好 - ⽔水平擴充能⼒力 • Open-source • Schemaless （打破 Schema 欄位限制）
- 8. 聚合 vs 關係 (Aggregate) (Relational) 圖⽚片出處：http://www.thoughtworks.com/insights/blog/nosql-databases-overview
- 9. 關聯資料模型 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 10. 關聯資料範例 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 11. 聚合資料模型 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 12. 聚合資料範例 // in customers { “id":1, “name”:"Martin", “billingAddress":[{"city":"Chicago"}] } // in orders { “id":99, “customerId":1, “orderItems":[ { “productId":27, "price": 32.45, "productName": "NoSQL Distilled” } ], “shippingAddress":[{"city":"Chicago"}] “orderPayment":[ { “ccinfo":"1000-1000-1000-1000", “txnId":"abelif879rft", "billingAddress": {"city": “Chicago”} } ], } 範例出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 14. 分⽚片(Sharding) 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 15. 主從複製 (Master-Slave Replication) 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 16. 對等複製 (Peer-to-peer Replication) 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 17. 主從複製+分⽚片 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 19. 資料衝突或遺失
- 20. 更新⼀一致性 • 寫衝突(Write-write Conflict) • 順序⼀一致性(Sequential Consistency) - 要確保所以節 點保證以相同次序操作 • 悲觀⽅方式：使⽤用鎖避免發⽣生衝突 (降低效能) • 樂觀⽅方式：讓衝突發⽣生，再檢測衝突、處理衝突 (使 ⽤用version stamp)
- 21. 讀寫⼀一致性 • 讀寫衝突(Read-write Conflict) • 邏輯⼀一致性(Logical Consistency) - 確保不同數據放在⼀一 起，其含意合乎邏輯 • 複製⼀一致性(Replication Consistency) - 要求從不同副本讀 取同⼀一份資料，得到的值相同 • 最終⼀一致性(Eventually Consistent) - 任意時間結點可能存 在不⼀一致，只要不再繼續執⾏行更新操作，最後⼀一次操作結 果最終會反映到全部節點 • 不⼀一致窗⼝口 - 存在不⼀一致⾵風險的時間⻑⾧長度
- 23. 放寬⼀一致性要求 • 集群必須要容忍分區狀況 • 在⼀一致性與可⽤用性間權衡、取捨 (但不是⼆二選⼀一) • 需要知道⽤用⼾戶對舊數據的容忍程度 • 不⼀一致窗⼝口的 mean time, worst case • 需要領域知識(domain knowledge)才能解決 • 按讚可以遺失 • 購物⾞車裡的商品數量不可遺失
- 24. 仲裁 (Quorum) • 寫⼊入仲裁(Write Quorum) • W > N/2 • 如果發⽣生兩個相衝突的寫⼊入操作，只要其中⼀一個操 作能為超過半數的節點認可 • 讀取仲裁(Read Quorum) • R+W > N • 保證讀取的⼀一致性
- 25. 主流的 NoSQL
- 27. NoSQL 特⾊色 Key-Value • 依照鍵查詢值，設置鍵對應的值，或從資料庫刪除鍵 • 值只是資料庫的⼀一塊數據，由應⽤用程式解釋 Document • 操作鬆散、⾮非結構性的資料，⽂文件具備⾃自我描述特 • ⽂文件格式：XML、JSON、BSON Column Family • 兩層映射(two-level map)資料結構，儲存關鍵字與映 射值，映射值由多個column組成 Garph • ⽤用圖學架構節點間關係資料架構，適合處理關係複雜 的資料結構
- 32. NoSQL ⼀一致性、可⽤用性 Key-Value • 針對單個鍵的操作 • 最終⼀一致性模型 • Peer-to-peer Replication 採⽤用 NRW 仲裁 Document • 針對單個⽂文件的操作 • 最終⼀一致性模型；可指定⼀一致性強度 • Master-slave Replication 增強可⽤用性 Column Family • 針對單個row的操作 • 最終⼀一致性模型 • Peer-to-peer Replication 採⽤用 NRW 仲裁 Garph • 修改節點或新增關係前，須先啟動事務確保⼀一致性 • Master-slave Replication 增強可⽤用性
- 33. NoSQL 查詢 Key-Value • 依照鍵查詢 Document • 擴展讀取效能：加⼊入更多從節點(slave node) • 擴展寫⼊入效能：根據字段把資料分⽚片 Column Family • 擴展讀取效能：加⼊入更多節點 Garph • Scale-up 使⽤用記憶體容納所有⼯工作集 • Scale-out 增加從節點改善讀取效能，主節點負責寫⼊入 • 使⽤用“領域特定知識”對應⽤用程式分⽚片
- 34. NoSQL 適合案例 Key-Value • 存放會話資訊 (key=sessionid) • ⽤用⼾戶配置資訊 (key=userid, username) • 購物⾞車資料 (key=userid) Document • 事件記錄 • 內容管理系統、部落格、⽤用⼾戶評論、web⽂文件 • ⽤用⼾戶註冊、⽤用⼾戶配置 • ⽤用來儲存page view, unique visitor資料，做網站分析 • 電⼦子商務應⽤用 Column Family • 事件記錄 • 內容管理、部落格 • 計數器、限期使⽤用 Garph • 互聯數據 • 安排運輸路線、基於位置的服務 • 推薦引擎
- 35. NoSQL 不適⽤用場合 Key-Value • 查詢資料間關係 • 含多項操作的 transaction • 查詢數據 (無法檢視資料庫中的值)、關鍵字集合 Document • 包含多項操作的複雜事物 • 查詢持續變動的聚合結構 Column Family • 根據資料庫查詢結果來聚合數據 (如SUM) • 改變查詢模式要⽐比改變數據模式代價更⾼高 Garph • 更新全部或某⼦子集的實體，只要⼀一個屬性變了，全部 實體都要更新
- 36. 沒有最強，只有更適合 Typical RDBMS Scale to Complexity ScaletoSize SQL Comfort Zone Column Family DB Key-Value DB Document DB Graph DB
- 37. 視圖 (View)
- 38. NoSQL 的軟肋? • 不容易從聚合資料庫找到聚合間的關係：從訂單數據 推算過去的銷售量 • View：根據 base tables 推算出來的 relational table • 解法：使⽤用預先算好存在磁碟中的物化視圖 (Materialized View) • 作法：以 ReduceMap 計算物化視圖
- 39. 映射(Map) 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 40. 歸納(Reduce) 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 41. 合併(Combine) 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 42. 在分區上進⾏行歸納 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 43. 組合技(1/3) 創建產品⽉月銷量紀錄 使⽤用映射-歸納產⽣生與前年同⽉月份銷量⽐比較表 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 44. 組合技(2/3) 創建產品⽉月銷量紀錄 計算同⽐比紀錄增⻑⾧長 所⽤用的基礎資料 使⽤用映射-歸納產⽣生與前年同⽉月份銷量⽐比較表 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 45. 組合技(3/3) 創建產品⽉月銷量紀錄 計算同⽐比紀錄增⻑⾧長 所⽤用的基礎資料 合併兩條不完整的紀錄 使⽤用映射-歸納產⽣生與前年同⽉月份銷量⽐比較表 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 46. db.runCommand({ mapreduce: "DenormAggCollection", query: { filter1: { '$in': [ 'A', 'B' ] }, filter2: 'C', filter3: { '$gt': 123 } }, map: function() { emit( { d1: this.Dim1, d2: this.Dim2 }, { msum: this.measure1, recs: 1, mmin: this.measure1, mmax: this.measure2 < 100 ? this.measure2 : 0 } );}, reduce: function(key, vals) { var ret = { msum: 0, recs: 0, mmin: 0, mmax: 0 }; for(var i = 0; i < vals.length; i++) { ret.msum += vals[i].msum; ret.recs += vals[i].recs; if(vals[i].mmin < ret.mmin) ret.mmin = vals[i].mmin; if((vals[i].mmax < 100) && (vals[i].mmax > ret.mmax)) ret.mmax = vals[i].mmax; } return ret; }, finalize: function(key, val) { val.mavg = val.msum / val.recs; return val; }, out: 'result1', verbose: true }); db.result1. find({ mmin: { '$gt': 0 } }). sort({ recs: -1 }). skip(4). limit(8); SELECT Dim1, Dim2, SUM(Measure1) AS MSum, COUNT(*) AS RecordCount, AVG(Measure2) AS MAvg, MIN(Measure1) AS MMin MAX(CASE WHEN Measure2 < 100 THEN Measure2 END) AS MMax FROM DenormAggTable WHERE (Filter1 IN (’A’,’B’)) AND (Filter2 = ‘C’) AND (Filter3 > 123) GROUP BY Dim1, Dim2 HAVING (MMin > 0) ORDER BY RecordCount DESC LIMIT 4, 8 1 2 3 4 5 1 7 6 1 2 3 4 5 Grouped dimension columns are pulled out as keys in the map function, reducing the size of the working set. Measures must be manually aggregated. Aggregates depending on record counts must wait until finalization. Measures can use procedural logic. Filters have an ORM/ActiveRecord- looking style. 6 Aggregate filtering must be applied to the result set, not in the map/reduce. 7 Ascending: 1; Descending: -1 Revision4,Created2010-03-06 RickOsborne,rickosborne.org mySQL MongoDB 圖⽚片出處：http://nosql.mypopescu.com/post/392418792/translate-sql-to-mongodb-mapreduce
- 48. ⽤用關係資料庫儲存資料 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 49. 混⽤用各種資料庫 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 50. 把資料庫封裝成服務 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 51. 擴充資料庫增強功能 圖⽚片出處：⟪NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence⟫
- 52. 選擇合適的資料庫
- 53. 使⽤用 NoSQL 的原因 • 通過使⽤用符合應⽤用程式所需的資料庫，改善程序員的 ⼯工作效率 • 處理⼤大量數據、降低延遲、增加資料吞吐量，改善資 料的訪問性能
- 54. 挑選策略 • 選擇前要先測試其是否如預期改善效率與性能問題 • 使⽤用服務封裝資料庫，能在需求變更或技術成熟後替 換所封裝的資料庫技術 • 繼續沿⽤用默認的關係型資料庫