Fast and Furious: Handling Edge Computing Data With Oracle 19c Fast Ingest and Fast Lookup
Download as PPSX, PDF0 likes202 views
The Internet of Things (IoT) has deep use cases - energy grids, communications, policing, security, and manufacturing. I’ll show how to use Oracle 19c’s Fast Ingest and Fast Lookup features to load IoT data from “edge” sources to take immediate advantage of that information in nearly real time.
Ad
More Related Content
What's hot (20)
Similar to Fast and Furious: Handling Edge Computing Data With Oracle 19c Fast Ingest and Fast Lookup (20)
![[db tech showcase Tokyo 2017] A11: SQLite - The most used yet least appreciat...](https://cdn.slidesharecdn.com/ss_thumbnails/dbts-tokyo-keynote-20170905-170911071000-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Jim Czuprynski (17)
Ad
Recently uploaded (20)
Fast and Furious: Handling Edge Computing Data With Oracle 19c Fast Ingest and Fast Lookup
- 1. Jim Czuprynski @JimTheWhyGuy Zero Defect Computing, Inc. Fast and Furious: Handling Edge Computing Data With Oracle 19c Fast Ingest and Fast Lookup
- 2. My Credentials • 40 years of database-centric IT experience • Oracle DBA since 2001 • Oracle 9i, 10g, 11g, 12c OCP and ADWC • Oracle ACE Director since 2014 • ODTUG Database Committee Lead and Board Member • Editor of ODTUG TechCeleration • Oracle-centric blog (Generally, It Depends) • Regular speaker at Oracle OpenWorld, CodeONE, KSCOPE, and international and regional OUGs E-mail me at [email protected] Follow me on Twitter (@JimTheWhyGuy) Connect with me on LinkedIn (Jim Czuprynski)
- 3. Edge Computing Opportunities, Everywhere You Look! Industrial Internet of Things (IIoT), especially manufacturing Artificial Intelligence IoT (AIOT) Elon Musk’s $100M carbon capture prize Smart Cities: Scooters, Bicycles, and Traffic Flow Autonomous delivery vehicles Improving farming: More efficient use of resources & eliminating over-watering Reducing unnecessary food waste Improved worker safety for occupations most affected by climate change
- 4. The Likely Best Solution Is Obvious. Electrify Everything! Renewable energy costs (wind & solar) are now cheaper than natural gas and coal Smart Meters for electrical consumption are ubiquitous “Green” hydrogen: A raging (but good!) debate New safer, compartmentalized nuclear technology Shifts in electrical demand cycles as working from home becomes the “new normal” For a brief introduction on this concept, check out this short video
- 5. Application Server Oracle 19c Database Fast Ingest Capabilities of Oracle 19c Internet of Things (IoT) SGA Large Pool Rows Batch 1 Rows Batch 2 Rows Batch 3 Rows Batch n Drainers Smart Meters Social Media Feeds Images (CCTV) High- Frequency INSERTs Permanent Storage INSERTs These are deferred from immediate COMMIT! Verification of Synchronicity Background Processes
- 6. SwingBench Load Generator Oracle 19c Database Simulating IoT Load Testing For Fast Ingest Evaluation IoT SGA_TARGET (8GB) LARGE_POOL_SIZE Simulated Smart Meter Payloads Complex simulated transactions built via PL/SQL procedures … DATA Tablespace: Table T_METER_READINGS MEMOPTIMIZE_ POOL_SIZE (2G) Single- Row INSERTs Verification via DBMS_MEMOPTIMIZE Rows Batch 1 Rows Batch 2 Rows Batch 3 Rows Batch n … and sent by multiple user sessions to Oracle 19c database via SwingBench workload generator W00n W002 W001 W000
- 7. SIMIOT: A Simple IoT Schema Entities and Relationships: SMART_METERS Contains information about individual smart meters for which data is being collected METER_READINGS Individual readings for each smart meter over (often extremely!) short time intervals BUSINESS_DESCRIPTIONS Describes unique business classifications based on licensing issued Test Environment #1: Oracle 19c (19.3) Developer Virtual Appliance Image • VirtualBox 6.1.0 • Linux 7.7 • 8GB memory • 1 vCPU Test Environment #2: Oracle 21c (21.1) OCI DBCS Instance • EE - Extreme Performance • Linux 7.7 • 30GB memory • 2 OCPUs
- 8. Preparing For a Fast Ingest Workload: Some Unexpected Surprises (1) SQL> ALTER SYSTEM SET memoptimize_pool_size = 2G SCOPE=SPFILE; System altered. SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. SQL> startup; ORACLE instance started. Total System Global Area 1.4496E+10 bytes Fixed Size 9702624 bytes Variable Size 3892314112 bytes Database Buffers 1.0570E+10 bytes Redo Buffers 23851008 bytes Database mounted. Database opened. . . . Activating the Memory Optimized Row Store does require an instance bounce . . . SQL> show parameter pool_size NAME TYPE VALUE ------------------------------------ ----------- ------------ java_pool_size big integer 0 large_pool_size big integer 0 memoptimize_pool_size big integer 2G olap_page_pool_size big integer 0 shared_pool_size big integer 0 streams_pool_size big integer 0 SQL> ALTER SESSION SET CONTAINER = pdb19; Session altered. . . . … Awesome!! Almost there … SQL> ALTER TABLE simiot.t_meter_readings MEMOPTIMIZE FOR WRITE; ERROR at line 1: ORA-62172: MEMOPTIMIZE FOR WRITE feature cannot be enabled on table in encrypted tablespace. Wait … what? Seriously?!? See MOS Note #2396777.1, 18c New Memoptimized Rowstore, for list of supported infrastructure
- 9. Preparing For a Fast Ingest Workload: Some Unexpected Surprises (2) SQL> ALTER SYSTEM SET "_exadata_feature_on” = true SCOPE=SPFILE; SQL> SHUTDOWN IMMEDIATE; SQL> STARTUP; Here’s a neat trick to fake out your Oracle 19c test environment into thinking it’s actually an Exadata platform! (Provenance: @JulianDontcheff) SQL> ALTER TABLE simiot.t_meter_readings MEMOPTIMIZE FOR WRITE; ERROR at line 1: ORA-62144: MEMOPTIMIZE FOR WRITE feature not allowed on table with foreign key constraint. Huzzah!! Almost there (again) … Why you little … now what?
- 10. Fast Ingest: Intrinsic Limitations 10 Can I Use Fast Ingest On a Table That … Allowed? Uses RANGE or HASH partitioning? Yes Uses REFERENCE, INTERVAL, SYSTEM, or AUTOLIST partitioning? No Employs a Foreign Key constraint against another table? No Employs triggers? No Is stored within an encrypted tablespace? No Uses any storage compression? No Leverages IN MEMORY compression? No Is EXTERNAL, Index-Organized, TEMPORARY, or NESTED? No Uses a function-based, DOMAIN, BITMAP, or BITMAP JOIN index? No Has a column with default values, or that’s marked INVISIBLE, VIRTUAL, OR UNUSED? No Upon further review, most of these restrictions make sense After all, our intent is to drink from a firehose! See MOS Note #2605883.1, Limitations for Using Memoptimized Fast Ingest, for full list of limitations
- 11. Fast Ingest: How Are Transactions Handled? Feature Advantages Drawbacks Ingested data is captured in batches within the Large Pool, but not immediately written to the database Ingesting data is quite fast, and huge volumes of data from numerous sessions can be captured with extreme efficiency because the database isn’t processing individual rows Should the database instance crash before all ingested data is written to the database, it is possible to lose data Since Fast Ingest is not a transaction in the traditional Oracle sense, COMMITs don’t occur during within its context “Normal” Oracle transaction mechanisms are bypassed to enable rapid data capture • No COMMITs mean no ROLLBACKs, either! • Parent-child transactions must be coordinated to ensure against data loss • Data cannot be queried until it’s actually been flushed to disk from Fast Ingest buffers Index operations and constraint checking only happens when data is finally written from the Large Pool Fast Ingest area to disk Not necessarily a bad thing! Should a primary key violation occur while the “drainer” background processes are writing data from the FI buffers to disk, the violating rows won’t be INSERTed … but no exception will be raised Unless the insert fails for valid reasons, the application itself must ensure that all valid data have actually been inserted DBMS_MEMOPTIMIZE procedures can be called to verify all data has been written successfully the database The application itself must now ensure all data has indeed been written to the database
- 12. Tracking Status Via V$MEMOPTIMIZE_WRITE_AREA SQL> SELECT (total_size / (1024*1024)) total_size_mb ,(used_space / (1024*1024)) used_space_mb ,(free_space / (1024*1024)) free_space_mb ,num_writes ,num_writers ,con_id FROM v$memoptimize_write_area; V$MEMOPTIMIZE_WRITE_AREA contains statistics about the activity of the background “drainer” processes as well as how much memory is in use … TOTAL_SIZE_MB USED_SPACE_MB FREE_SPACE_MB NUM_WRITES NUM_WRITERS CON_ID ------------- ------------- ------------- ---------- ----------- ------ 2055 .156784058 2054.84322 0 106 3 . . . TOTAL_SIZE_MB USED_SPACE_MB FREE_SPACE_MB NUM_WRITES NUM_WRITERS CON_ID ------------- ------------- ------------- ---------- ----------- ------ 2055 1.15670776 2053.84329 8830 107 3 . . . TOTAL_SIZE_MB USED_SPACE_MB FREE_SPACE_MB NUM_WRITES NUM_WRITERS CON_ID ------------- ------------- ------------- ---------- ----------- ------ 2055 3.15655518 2051.84344 24681 109 3 … which makes it easy to see the number of writes to the database continue as the generated workload progresses
- 13. Verifying the State of Rows Written Via Fast Ingest DECLARE applied_hwm NUMBER(15,0); written_hwm NUMBER(15,0); BEGIN applied_hwm := DBMS_MEMOPTIMIZE.GET_APPLY_HWM_SEQID; written_hwm := DBMS_MEMOPTIMIZE.GET_WRITE_HWM_SEQID; DBMS_OUTPUT.PUT_LINE( 'Low-Water-Mark for All Rows Applied (Written Globally): ' || applied_hwm); DBMS_OUTPUT.PUT_LINE( 'High Water Mark for Rows Written For This Session: ' || written_hwm); EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE( 'Fatal unexpected error: ' || SQLCODE || ' - ' || SQLERRM); END; Low-Water-Mark for All Rows Applied (Written Globally): 498413730056 High Water Mark for Rows Written For This Session: 0 Here’s the state of the LWM and HWM before a workload begins … Low-Water-Mark for All Rows Applied (Written Globally): 498536313235 High Water Mark for Rows Written For This Session: 498536328525 … and then after a workload has been generated within this same session … Low-Water-Mark for All Rows Applied (Written Globally): 499537736037 High Water Mark for Rows Written For This Session: 498536328525 … and finally, after more time has passed, note the global LWM has surpassed the session’s HWM
- 14. Controlling Fast Ingest Features Via PL/SQL ----- -- Flush all unwritten data to the database -- for the +current+ session ----- BEGIN DBMS_MEMOPTIMIZE.WRITE_END; EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE( 'Fatal unexpected error: ‘ || SQLCODE || ' - ' || SQLERRM); END; / We can force any as-yet unwritten data to be written to the database at either a session level … ----- -- Flush all unwritten data to the database -- for +all+ sessions ----- BEGIN DBMS_MEMOPTIMIZE_ADMIN.WRITES_FLUSH; EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE( 'Fatal unexpected error: ' || SQLCODE || ' - ' || SQLERRM); END; / … or for the entire database with calls to the appropriate procedure
- 15. Fast Ingest vs. Single-Row Insert: Performance Comparison Note the difference in statements executed – a 36% improvement with just one hint added! Note there are fewer physical writes overall, even with many more rows inserted in the same amount of time
- 16. SwingBench Load Generator Oracle 19c Database Fast Lookup Concepts IoT SGA_TARGET (8GB) LARGE_POOL_SIZE Simulated Smart Meter Payloads T_METER_READINGS MEMOPTIMIZE_ POOL_SIZE (2G) Single- Row INSERTs Rows Batch 1 Rows Batch 2 Rows Batch 3 Rows Batch n T_SMARTMETERS T_METER_READINGS Analytics SQL> ALTER TABLE T_METER_READINGS MEMOPTIMIZE FOR WRITE; ALTER TABLE T_METER_READINGS MEMOPTIMIZE FOR READ; ALTER TABLE T_SMARTMETERS MEMOPTIMIZE FOR READ;
- 17. Fast Lookup: Reporting on Frequently Accessed Data Ideally, Fast Lookup is best deployed for data that will be accessed frequently based on exact primary key values • When a table is in MEMOPTIMIZE READ mode, any Fast Lookup request against its data builds a hash table of PK values • Corresponding rows are added to buffers in MEMOPTIMIZEd pool and then pinned • The hash table is thus shortcut to reading a row directly from its corresponding buffer much more quickly than from traditional buffer cache NOTE: This doesn’t mean Fast Lookup is only useful for tables leveraging Fast Ingest! Scenario Example Often-accessed rows Rows containing values most likely to be accessed most frequently – for example, movies most recently added to a streaming service Most recently loaded rows Users tend to access data about most recent store sales than yesterday’s / last week’s sales Repetitive reads on the same larger “chunks” of data Rows containing a nearly-fully-populated VARCHAR2(2000) column
- 18. Preparing For a Fast Lookup Workload SQL> ALTER TABLE simiot.t_smartmeters MEMOPTIMIZE FOR READ; ERROR at line 1: ORA-62142: MEMOPTIMIZE FOR READ feature requires NOT DEFERRABLE PRIMARY KEY constraint on the table Doh! That’s right, a table must have a primary key for Fast Lookup to work. SQL> ALTER TABLE simiot.t_meter_readings MEMOPTIMIZE FOR READ; ERROR at line 1: ORA-62156: MEMOPTIMIZE FOR READ feature not allowed on segment with deferred storage Jeez, now what? As of now, there is no MOS Note that contains a full list of limitations! Look through Oracle Error Messages in the range of ORA-62100 thru ORA-62200 for a complete list.
- 19. Controlling Fast Lookup Features Via PL/SQL ----- -- Populate an object into the MORS pool -- for Fast Lookup ----- BEGIN DBMS_MEMOPTIMIZE.POPULATE( schema_name => 'SIMIOT' ,table_name => 'T_METER_READINGS' ,partition_name => NULL ); EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE( 'Fatal unexpected error: ‘ || SQLCODE || ' - ' || SQLERRM); END; / ----- -- Remove an object enabled for Fast Lookup -- from the MORS pool ----- BEGIN DBMS_MEMOPTIMIZE.DROP_OBJECT( schema_name => 'SIMIOT' ,table_name => 'T_METER_READINGS' ,partition_name => NULL ); EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE( 'Fatal unexpected error: ‘ || SQLCODE || ' - ' || SQLERRM); END; /
- 20. Fast Lookup: Indications of Success (1) SELECT name, value FROM v$sysstat WHERE name LIKE '%memopt r%' ORDER BY name; NAME VALUE ------------------------------------ ---------- memopt r NO IM tasks accepted 4 memopt r cleanup 1 memopt r entries deleted 36 memopt r lookups 1 memopt r misses 1 memopt r populate tasks accepted 4 System-level statistics for Fast Lookup all contain the string ”memopt r” … SELECT P.program "Executor" ,N.name "Statistic" ,S.value "Value" FROM v$sesstat S ,v$statname N ,v$session P WHERE S.statistic# = N.statistic# AND S.sid = P.sid AND N.name LIKE '%memopt r%' AND S.value > 0 ORDER BY P.program, N.name; Executor Statistic Value ------------------------------------- ---------------------------------------- ---------- SQL Developer memopt r NO IM tasks accepted 4 SQL Developer memopt r populate tasks accepted 2 [email protected] (W000) memopt r blocks populated 457 [email protected] (W000) memopt r populate 1 [email protected] (W000) memopt r puts 74314 [email protected] (W000) memopt r rows populated 74314 [email protected] (W000) memopt r successful puts 74314 [email protected] (W001) memopt r cleanup 1 [email protected] (W001) memopt r entries deleted 32817 [email protected] (W002) memopt r blocks populated 34734 [email protected] (W002) memopt r puts 5660881 [email protected] (W002) memopt r puts:buckets full 681453 [email protected] (W002) memopt r rows populated 5660881 [email protected] (W002) memopt r successful puts 5660881 [email protected] (W002) memopt r successful puts:with evictions 681453 [email protected] (W002) memopt r tag collisions 104 [email protected] (W007) memopt r cleanup 1 [email protected] (W007) memopt r entries deleted 2800857 … and here’s a way to look at Fast Lookup statistics at a session level
- 21. Fast Lookup: Indications of Success (2) … but an equality predicate definitely does! A range scan can’t take advantage of Fast Lookup …
- 22. Analyzing Near-Real-Time Data with Oracle Machine Learning (1) CREATE OR REPLACE VIEW simiot.solar_superusers AS SELECT sm_business_type ,sm_zipcode ,sm_id ,ROUND(AVG(smr_kwh_used),2) avg_kwh_used ,ROUND(AVG(smr_solar_kwh),2) avg_solar_kwh ,ROUND(AVG(smr_solar_kwh) / AVG(smr_kwh_used) ,2) pct_solar ,CASE WHEN ROUND(AVG(smr_solar_kwh) / AVG(smr_kwh_used) ,2) >= 0.15 THEN 1 ELSE 0 END as solar_superuser FROM t_smartmeters ,t_meter_readings WHERE smr_id = sm_id GROUP BY sm_business_type, sm_zipcode, sm_id ORDER BY sm_business_type, sm_zipcode, sm_id ; We’ll use this metric to identify which SmartMeter customers are solar energy super-users (defined as 15% or more of their energy generated via solar power on-site)
- 23. Analyzing Near-Real-Time Data with OML4SQL (2) DECLARE -- Processing variables: SQLERRNUM INTEGER := 0; SQLERRMSG VARCHAR2(255); vcReturnValue VARCHAR2(256) := 'Failure!'; v_setlist DBMS_DATA_MINING.SETTING_LIST; BEGIN v_setlist('ALGO_NAME') := 'ALGO_KMEANS'; v_setlist('PREP_AUTO') := 'ON'; v_setlist('KMNS_DISTANCE') := 'KMNS_EUCLIDEAN'; v_setlist('KMNS_DETAILS') := 'KMNS_DETAILS_ALL'; v_setlist('KMNS_ITERATIONS') := '3'; v_setlist('KMNS_NUM_BINS') := '10’; . . . We’ll use a k-Means algorithm to delve into which Smart Meters are clustered together, based on various attributes of the companies using them …
- 24. DECLARE -- Processing variables: SQLERRNUM INTEGER := 0; SQLERRMSG VARCHAR2(255); vcReturnValue VARCHAR2(256) := 'Failure!'; v_setlist DBMS_DATA_MINING.SETTING_LIST; BEGIN v_setlist('ALGO_NAME') := 'ALGO_KMEANS'; v_setlist('PREP_AUTO') := 'ON'; v_setlist('KMNS_DISTANCE') := 'KMNS_EUCLIDEAN'; v_setlist('KMNS_DETAILS') := 'KMNS_DETAILS_ALL'; v_setlist('KMNS_ITERATIONS') := '3'; v_setlist('KMNS_NUM_BINS') := '10’; . . . Analyzing Near-Real-Time Data with OML4SQL (2) . . . DBMS_DATA_MINING.CREATE_MODEL2( model_name => 'OML4_SIMIOT_CLUSTERING', mining_function => 'CLUSTERING', data_query => 'SELECT * FROM simiot.solar_superusers', set_list => v_setlist, case_id_column_name => 'SM_ID'); EXCEPTION WHEN OTHERS THEN SQLERRNUM := SQLCODE; SQLERRMSG := SQLERRM; vcReturnValue := 'Error creating new OML4SQL model: ' || SQLERRNUM || ' - ' || SQLERRMSG; DBMS_APPLICATION_INFO.SET_MODULE(NULL,NULL); DBMS_OUTPUT.PUT_LINE(vcReturnValue); END; / … and leverage the view we just created as the source for each Smart Meter’s related attributes
- 25. Analyzing Near-Real-Time Data with OML4SQL (3) Here’s an example of using a Radar Area graph to show the top 15 business types that are the most egregious outliers in terms of solar power used (aka distance from centroid)
- 26. So … Where Are the Solar-Powered Superstars? … Oracle’s powerful Spatial and Graph toolsets and free APEX plug-ins make short work of identifying which businesses are taking maximum advantage of solar power With a relatively simple SQL query against our SmartMeter attributes and their most recent meter readings …
- 27. Plans for Future Experimentation Simulating intense direct streaming from IoT devices using Oracle Streaming Services (OSS) Testing with extreme workloads against an Autonomous Transaction Processing (ATP) instance Incorporating a new workload generator for more complex IoT streams (details coming soon!)
- 28. Useful Resources and Documentation City of Chicago Reference Information: https://data.cityofchicago.org/Community-Economic-Development/Business-Licenses-Current-Active/uupf-x98q/data Architecture of the Memoptimized Rowstore: https://docs.oracle.com/en/database/oracle/oracle-database/19/tgdba/tuning-system-global- area.html#GUID-9752E93D-55A7-4584-B09B-9623B33B5CCF Todd Sharp’s Blog Posts on Oracle Streaming Service (OSS): https://blogs.oracle.com/developers/back-to-the-database-part-1-preparing-to-persist-data-from-a-stream https://blogs.oracle.com/developers/back-to-the-database-part-2-persisting-data-from-a-stream https://blogs.oracle.com/developers/back-to-the-database-part-3-publishing-database-changes-to-a-stream
- 29. Articles on the Future of IoT … and Our Planet IoT and the Potential to Save the World https://www.iotforall.com/iot-potential-to-save-the-world What is IIoT? Industrial Internet of Things Explained https://www.hologram.io/blog/what-is-iiot IoT For All at CES: AI, IoT, and Robots https://www.iotforall.com/ai-iot-aiot-ces How Smart Devices with Edge Analytics are Helping Business https://www.iotforall.com/how-smart-devices-with-edge-analytics-are-helping-business How 100% Clean Energy Could Power Our Cities And Towns https://grist.org/energy/how-100-clean-energy-could-power-our-cities-and-towns American suburbs are about to look more like European cities https://www.fastcompany.com/90576432/american-suburbs-are-about-to-look-more-like-european-cities Miami pilots e-cargo bikes to reduce congestion, pollution https://www.smartcitiesdive.com/news/miami-e-cargo-bike-pilot-dhl-city-congestion-pollution/578115/ You say old coal plant, I say green hydrogen https://www.utilitydive.com/news/you-say-old-coal-plant-i-say-green-hydrogen/588499/
Editor's Notes
- #6: Directly from Oracle documentation: The ingested data is batched in the large pool and is not immediately written to the database. Thus, the ingest process is very fast. Very large volumes of data can be ingested efficiently without having to process individual rows. However, if the database goes down before the ingested data is written out to the database files, it is possible to lose data. Fast ingest is very different from normal Oracle Database transaction processing where data is logged and never lost once "written" to the database (i.e. committed). In order to achieve the maximum ingest throughput, the normal Oracle transaction mechanisms are bypassed, and it is the responsibility of the application to check to see that all data was indeed written to the database. Special APIs have been added that can be called to check if the data has been written to the database. The commit operation has no meaning in the context of fast ingest, because it is not a transaction in the traditional Oracle sense. There is no ability to rollback the inserts. You also cannot query the data until it has been flushed from the fast ingest buffers to disk. You can see some administrative information about the fast ingest buffers by querying the view V$MEMOPTIMIZE_WRITE_AREA. You can also use the packages DBMS_MEMOPTIMIZE and DBMS_MEMOPTIMIZE_ADMIN to perform functions like flushing fast ingest data from the large pool and determining the sequence id of data that has been written. Index operations and constraint checking is done only when the data is written from the fast ingest area in the large pool to disk. If primary key violations occur when the background processes write data to disk, then the database will not write those rows to the database. Assuming (for most applications but not all) that all inserted data needs to be written to the database, it is critical that the application insert process checks to see that the inserted data has actually been written to the database before destroying that data. Only when that confirmation has occurred can the data be deleted from the inserting process.
- #11: From 19c documentation: Tables with the following characteristics cannot use fast ingest: Tables with: disk compression in-memory compression column default vales encryption functional indexes domain indexes bitmap indexes bitmap join indexes ref types varray types OID$ types unused columns virtual columns LOBs triggers binary columns foreign keys row archival invisible columns Temporary tables Nested tables Index organized tables External tables Materialized views with on-demand refresh Sub-partitioning is not supported. The following partitioning types are not supported. REFERENCE SYSTEM INTERVAL AUTOLIST
- #12: Directly from Oracle 19c documentation: The ingested data is batched in the large pool and is not immediately written to the database. Thus, the ingest process is very fast. Very large volumes of data can be ingested efficiently without having to process individual rows. However, if the database goes down before the ingested data is written out to the database files, it is possible to lose data. Fast ingest is very different from normal Oracle Database transaction processing where data is logged and never lost once "written" to the database (i.e. committed). In order to achieve the maximum ingest throughput, the normal Oracle transaction mechanisms are bypassed, and it is the responsibility of the application to check to see that all data was indeed written to the database. Special APIs have been added that can be called to check if the data has been written to the database. The commit operation has no meaning in the context of fast ingest, because it is not a transaction in the traditional Oracle sense. There is no ability to rollback the inserts. You also cannot query the data until it has been flushed from the fast ingest buffers to disk. You can see some administrative information about the fast ingest buffers by querying the view V$MEMOPTIMIZE_WRITE_AREA. Parent-child transactions must be synchronized to avoid errors. For example, foreign key inserts and updates of rows inserted into the large pool can return errors, if the parent data is not yet written to disk. Index operations and constraint checking is done only when the data is written from the fast ingest area in the large pool to disk. If primary key violations occur when the background processes write data to disk, then the database will not write those rows to the database. Assuming (for most applications but not all) that all inserted data needs to be written to the database, it is critical that the application insert process checks to see that the inserted data has actually been written to the database before destroying that data. Only when that confirmation has occurred can the data be deleted from the inserting process.
- #13: Notes from Kam Discussion: > Do FORALL bulk INSERTs cloud the effectiveness of MORS vs. single-statement INSERTs? It’s really designed for single-row INSERTs from multiple sessions. > What are the thresholds that MORS uses to decide to write transactions to the database? None, really. When a reserved MORS buffer is filled up, it’s marked for writing and then it is written via W00x processes (really via a job that’s tasked to the W00X process).