Oracle 19c Automatic Indexing: Data Skew Part II (Everything’s Alright) September 14, 2020
Posted by Richard Foote in 19c, 19c New Features, Automatic Indexing, Automatic Table Statistics, Autonomous Transaction Processing, Data Skew, Exadata, High Frequency Statistics Collection, Histograms, Oracle, Oracle Cost Based Optimizer, Oracle General, Oracle Indexes, Oracle Statistics, Performance Tuning.2 comments
In my previous post, I discussed an example with data skew, in which the Automatic Indexing process created a new index, but somehow the CBO when using the index estimated the correct cardinality estimate even though no histograms were explicitly calculated.
In this post I’ll answer HOW this achieved by the CBO.
Get some idea on the answer by now looking at the column details:
SQL> select column_name, num_buckets, histogram from user_tab_cols where table_name='BOWIE_SKEW'; COLUMN_NAME NUM_BUCKETS HISTOGRAM --------------- ----------- --------------- ID 1 NONE CODE 10 FREQUENCY NAME 1 NONE
We can see that there is now indeed an histogram on the column. When and how were these histograms collected?
The answer lies with a new Oracle Database 19c feature called “High-Frequency Automatic Statistics Collection“, which is available on Exadata environments. As I’m running all these demos on the Oracle Autonomous Transaction Processing Cloud environment which runs on an Exadata platform, this feature is enabled by default.
To highlight the capabilities of this features more fully, I’m going to setup a slightly different demo with three tables:
SQL> create table bowie1 (id number, code number, name varchar2(42)); <= Stale with no stats Table created. SQL> insert into bowie1 select rownum, mod(rownum, 100)+1, 'David Bowie' from dual connect by level <= 100000; 100000 rows created. SQL> commit; Commit complete.
Table BOWIE1 has no statistics collected on it.
SQL> create table bowie2 (id number, code number, name varchar2(42)); Table created. SQL> insert into bowie2 select rownum, mod(rownum, 100)+1, 'David Bowie' from dual connect by level <= 100000; 100000 rows created. SQL> commit; Commit complete. SQL> exec dbms_stats.gather_table_stats(ownname=>null, tabname=>'BOWIE2'); PL/SQL procedure successfully completed. SQL> insert into bowie2 select rownum+100000, mod(rownum, 100)+1, 'Ziggy Stardust' from dual connect by level <= 50000; 50000 rows created. SQL> commit; Commit complete.
BOWIE2 table has new rows added after statistics have been collected and so has “stale” outdated stats.
SQL> create table bowie3 (id number, code number, name varchar2(42));
Table created.
SQL> insert into bowie3 select rownum, 10, 'DAVID BOWIE' from dual connect by level <=1000000;
1000000 rows created.
SQL> update bowie3 set code = 9 where mod(id,3) = 0;
333333 rows updated.
SQL> update bowie3 set code = 1 where mod(id,2) = 0 and id between 1 and 20000;
10000 rows updated.
SQL> update bowie3 set code = 2 where mod(id,2) = 0 and id between 30001 and 40000;
5000 rows updated.
SQL> update bowie3 set code = 3 where mod(id,100) = 0 and id between 300001 and 400000;
1000 rows updated.
SQL> update bowie3 set code = 4 where mod(id,100) = 0 and id between 400001 and 500000;
1000 rows updated.
SQL> update bowie3 set code = 5 where mod(id,100) = 0 and id between 600001 and 700000;
1000 rows updated.
SQL> update bowie3 set code = 6 where mod(id,1000) = 0 and id between 700001 and 800000;
100 rows updated.
SQL> update bowie3 set code = 7 where mod(id,1000) = 0 and id between 800001 and 900000;
100 rows updated.
SQL> update bowie3 set code = 8 where mod(id,1000) = 0 and id between 900001 and 1000000;
100 rows updated.
SQL> commit;
Commit complete.
SQL> exec dbms_stats.gather_table_stats(ownname=>null, tabname=>'bowie3', estimate_percent=>100, method_opt=>'FOR ALL COLUMNS SIZE 1');
PL/SQL procedure successfully completed.
SQL> select code, count(*) from bowie3 group by code order by code;
CODE COUNT(*)
---------- ----------
1 10000
2 5000
3 1000
4 1000
5 1000
6 100
7 100
8 100
9 327235
10 654465
The BOWIE3 table is as my previous example, with data skew but with NO histograms collected. I’m now going to run a query on BOWIE3 where the CBO gets the cardinality estimate hopelessly wrong because of the missing histogram on the CODE column:
SQL> select * from bowie3 where code=7;
100 rows selected.
Execution Plan
----------------------------------------------------------
Plan hash value: 2517725203
----------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 100K| 1953K| 974 (2)| 00:00:01 |
|* 1 | TABLE ACCESS FULL| BOWIE3 | 100K| 1953K| 974 (2)| 00:00:01 |
----------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter("CODE"=7)
If we look at the current statistics on these tables:
SQL> select table_name, num_rows, stale_stats, notes from user_tab_statistics
where table_name in ('BOWIE1', 'BOWIE2', 'BOWIE3');
TABLE_NAME NUM_ROWS STALE_S NOTES
--------------- ---------- ------- ------------------------------
BOWIE1
BOWIE2 100000 YES
BOWIE3 1000000 NO
BOWIE2 150000 STATS_ON_CONVENTIONAL_DML
We can see that BOWIE1 has indeed no statistics.
BOWIE2 is marked as having state statistics, although thanks to another Oracle Database 19c feature called “Real-Time Statistics Collection“, does have some additional statistics captured (such as NUM_ROWS) when the additional rows were inserted. I’ll discuss this feature more fully in a later blog article.
BOWIE3 is considered fine in that it does have statistics which are NOT stale, BUT…
SQL> select column_name, num_buckets, histogram from user_tab_col_statistics where table_name='BOWIE3'; COLUMN_NAME NUM_BUCKETS HISTOGRAM --------------- ----------- --------------- ID 1 NONE CODE 1 NONE NAME 1 NONE
We don’t currently have any histograms even though a simple single table query was previously run based on a CODE predicate which had hopelessly inaccurate cardinality estimates.
If we wait approximately 15 minutes (default) for the High-Frequency Automatic Statistics Collection process to run and look at these column statistics again:
SQL> select table_name, num_rows, stale_stats from user_tab_statistics
where table_name in ('BOWIE1', 'BOWIE2', 'BOWIE3');
TABLE_NAME NUM_ROWS STALE_S
--------------- ---------- -------
BOWIE1 100000 NO
BOWIE2 150000 NO
BOWIE3 1000000 NO
SQL> select column_name, num_buckets, histogram from user_tab_col_statistics where table_name='BOWIE3';
COLUMN_NAME NUM_BUCKETS HISTOGRAM
--------------- ----------- ---------------
ID 1 NONE
CODE 10 FREQUENCY
NAME 1 NONE
We now notice that:
BOWIE1 now has statistics captured, as the High-Frequency Automatic Statistics Collection process looks for tables with missing statistics.
BOWIE2 now has fully up to date statistics, as the High-Frequency Automatic Statistics Collection process looks for tables with stale statistics.
BOWIE3 now has histograms on the CODE columns, as the High-Frequency Automatic Statistics Collection process looks out for missing histograms if queries have been subsequently run with poor cardinality estimates.
Having more accurate, appropriate and up to date statistics all supports the CBO in making much better decisions in relation to the use of any newly created Automatic Indexes.
You can configure High-Frequency Automatic Statistics Collection in the following manner:
SQL> EXEC DBMS_STATS.SET_GLOBAL_PREFS('AUTO_TASK_STATUS','ON');
PL/SQL procedure successfully completed.
This turns the feature ON/OFF. It’s OFF by default on standard Exadata environments but ON by default in Autonomous Database environment.
SQL> EXEC DBMS_STATS.SET_GLOBAL_PREFS('AUTO_TASK_MAX_RUN_TIME','900');
PL/SQL procedure successfully completed.
This configures how long to allow the process to run (default is 3600 seconds/60 minutes).
SQL> EXEC DBMS_STATS.SET_GLOBAL_PREFS('AUTO_TASK_INTERVAL','900');
PL/SQL procedure successfully completed.
This configures the interval between the process running (default is every 900 seconds/15 minutes).
In my next post, I’ll look at a slightly more complex data skew example with Automatic Indexing, where both selective and unselective SQL predicates are invoked…
