Monday, March 19, 2018

SQL statement parsing stages



SQL Parsing Flow Diagram:
SQL Parsing Flow Diagram 
NOTES
1. A cursor is an address on the client that points to the memory location of a SQL statement on the server. Multiple-client cursors may point at the same address on the server.
2. Remember that 'Client' and 'Server' sides may reside on the same machine - in which case Client/Server is a logical distinction.
3. If a cursor is open, then the statement will be in the sql_area, so no parsing is necessary. This is why locks may remain when a client is terminated abnormally (such as a PC Client being turned off without closing open cursors).
4. SESSION_CACHED_CURSORS is the initialisation parameter that specifies how many cursors to hold open for a particular session.
The open cursor request will still be sent to the server but it will not be executed once a matching cursor is found in the session cursor cache.
5. HOLD_CURSOR is an precompiler parameter that specifies that an individual cursor should be held open. See:

Change value of parameter to all connected sessions


The ALTER SYSTEM SET clause for this parameter is only valid with the SCOPE=SPFILE option, so you can not change the parameter value with this dynamically.
You can change the value for all other sessions, e.g as below:
begin
for c in (select sid, serial# from v$session) loop
   dbms_system.set_int_param_in_session(c.sid,c.serial#,'session_cached_cursors', 100);
end loop;
end;
/

Wednesday, February 21, 2018

Histogram: Oracle 12c

Histogram are ways to store detailed information about column data for CBO optimizer to interpret the optimal access path for the query.

If there are no Histogram or table statistics are gathered with method_opt=>'FOR ALL COLUMNS SIZE 1', the optimizer checks the dictionary for:

1. High/Low (USER_TAB_COL_STATISTICS.LOW_VALUE and HIGH_VALUE) of column value,

2. Number of Distinct value (NDV),

3. The number of NULL and number of rows.



CBO generates the column selectivity of 1/NDV.



if there are a non-uniform distribution of data in the column, then CBO using default way of column selectivity cause performance degradation.



In terms of implementation, we could choose to store every distinct value together with the number of rows for that value.



For a small number of values this is efficient and 'width balanced' histograms are used.

For a higher number of distinct values, we should use Height Balanced Histogram





If the number of distinct values is less than or equal to the number of histogram buckets specified (up to 254) then a Frequency Histogram is created.

If the number of distinct values is greater than the number of histogram buckets specified, a Height Balanced Histogram is created.



Frequency Histograms:

These use buckets to record the row count for each distinct value.



Height Balanced Histograms:

These are implemented by dividing the data up into different 'buckets' where

each bucket contains the same number of values. The highest value in each bucket (or END_POINT) is recorded together with the lowest value in the "zero" bucket.



Once the data is recorded in buckets we recognize 2 types of data value - Non-popular values and popular values.



Non-popular values - are those that do not occur multiple times as endpoints.

Popular values - occur multiple times as end points.

We can use Popular and Non-Popular Values to provide users with various statistics. Since we know how many values there are in a bucket we can use this information to estimate the number of rows in total that is covered by Popular and Non-Popular values.



The selectivity for popular values can be obtained by calculation the proportion of bucket endpoints filled by that popular value.

The selectivity for nonpopular values can now be calculated as 1/number non-popular bucket endpoints, so we can now be more accurate about selectivities than the original 1/NDV because we have removed the popular values from the equation.



ALL_TAB_HISTOGRAMS describes histograms on tables and views accessible to the current user.
The ALL_TAB_HISTOGRAMS view contains a one-bucket histogram, which in fact signifies "No histogram" to the Oracle Database software. 

Therefore, it should not be queried to indicate the presence or absence of a histogram on a particular column.  Instead, query the value of column HISTOGRAM in the ALL_TAB_COL_STATISTICS view.


Density is a column statistic and provides selectivity estimates for
equi-join predicates (e.g. and A.COL1 = B.COL1) and equality predicates
(e.g. COL1 = 'Y').

The density is expressed as a decimal number between 0 and 1.
Values close to 1 indicate that this column is unselective
Values close to 0 indicate that this column is highly selective

The more selective a column, the less rows are likely to be returned by a
query referencing this column in its predicate list.

The column selectivity is part of the equation used to decide on the best
path for a query to take to retrieve the data required in the most effective
manner and hence impacts the final cost value for the query.

Wednesday, January 24, 2018

Smart Scan

What is Smart Scan ?
The data search and retrieval processing can be offloaded to the Exadata Storage Servers. This feature is called Smart Scan. Using this Smart Scan, Oracle Database can optimize the performance of operations that perform table and index scans by performing the scans inside Exadata Storage Server, rather than transporting all the data to the database server.

Smart Scan capabilities includes :-

1) Predicate Filtering

2) Column filtering

3) Join Processing



What is predicate filtering and what all conditional operators are supported by predicate filtering ?
 Exadata Storage Server enables predicate filtering for table scans. Rather than returning all the rows for the database to evaluate, Exadata Storage Server returns only the rows that match the filter condition.

 In addition, many common SQL functions can be evaluated by Exadata Storage Server during predicate filtering.

The definitive list of which functions are offloadable for your particular version is contained in V$SQLFN_METADATA.
   SQL> select * from v$sqlfn_metadata where offloadable = 'YES';

A list of conditional operators that are supported by predicate filtering include =, !=, <, >, <=, >=, IS [NOT] NULL, LIKE, [NOT] BETWEEN, [NOT] IN, EXISTS, IS OF type, NOT, AND, OR.



What is Bloom Filter ?
Offloaded joins are accomplished by creating what is called a bloom filter.

A Bloom filter, conceived by Burton Howard Bloom in 1970, is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set. The properties of a Bloom filter make is a very efficient way of determining which values are not in a set. This is very useful for processing join conditions where a significant proportion of the data does not fulfill the join criteria.

Oracle Database 10g Release 2 first used Bloom filters to optimize parallel join operations. When two tables are joined via a hash join, the first table (typically the smaller table) is scanned and the rows that satisfy the WHERE clause predicates (for that table) are used to create a hash table. During the hash table creation, a Bloom filter bit string is also created based on the join column. The bit string is then sent as an additional predicate to the second table scan. After the WHERE clause predicates relating to the second table are applied, the resulting rows are tested using the Bloom filter. Any rows rejected by the Bloom filter must fail the join criteria and are discarded. Any rows that match using the Bloom filter are sent to the hash join.

With Exadata, the Bloom filter is passed to the storage servers as an additional predicate. Processing the Bloom filter inside Exadata Storage Server can reduce the amount of data transported to the database server to process a join, which in turn can speed up query performance.



Will bloom filter work with all type of joins?
No, bloom filter works only with hash joins.



Does Smart Scan work with encrypted and compressed data ?
Yes, smart scan works with encrypted and compressed data.

Exadata Storage Server performs Smart Scans on encrypted tablespaces and encrypted columns. For encrypted tablespaces, Exadata Storage Server can decrypt blocks and return the decrypted blocks to Oracle Database, or it can perform row and column filtering on encrypted data. Significant CPU savings can be made within the database server by offloading the CPU-intensive decryption task to Exadata cells.

Smart Scan works in conjunction with Exadata Hybrid Columnar Compression so that column projection and row filtering can be executed along with decompression at the storage level to save CPU cycles on the database servers.



What are prerequisites for Smart Scan to occur ?
There are 4 basic requirements that must be met for Smart Scans to occur :
 1) There must be a full scan of an object; that is, full table scans, fast full index scans and fast full bitmap index scans.

 2)The scan must use Oracle’s Direct Path Read mechanism. Direct path reads are generally used by Oracle when reading directly into PGA memory (as opposed to into the buffer cache).
      - Direct-path reads are automatically used for parallel queries
      - Direct-path reads may be used for serial queries
             o Not used by default for serial small table scans
             o Use _serial_direct_read=TRUE to force direct path reads

3) Each segment being scanned must be on a disk group that is completely stored on Exadata cells. The disk group must also have the disk group attribute settings as below :-
        'compatible.rdbms' = 11.2.0.0.0' (or later)
        'compatible.asm' = 11.2.0.0.0' (or later)
        ' cell.smart_scan_capable' = TRUE

4) The CELL_OFFLOAD_PROCESSING initialization parameter enables or disables Smart Scan.

   The default value of the parameter is TRUE, meaning that Smart Scan is enabled by default. If it is set to FALSE, Smart Scan is disabled and the database uses Exadata storage to serve data blocks similar to traditional storage.



What are the situations that prevent Smart Scan to happen?
Scan on a clustered table
Scan on an index-organized table
Fast full scan on a compressed index
Fast full scan on a reverse key indexes
The table has row-level dependency tracking enabled
The ORA_ROWSCN pseudocolumn is being fetched
The optimizer wants the scan to return rows in ROWID order
The command is CREATE INDEX using NOSORT
A LOB or LONG column is being selected or queried
A SELECT .. VERSIONS flashback query is being executed
To evaluate a predicate based on a virtual column
More than 255 columns are referenced in the query
The data is encrypted and cell-based decryption is disabled
If table has CACHE property
If _serial_direct_read is turned off (NEVER)
If the partitioned object's size is less than _small_table_threshold.
Offloading is not done on serial DMLs.
Serial Direct read is not applied for these cases (and thus no smart scan) :
     - sql from a table function
     - sql from dbms_sql
     - sql from plsql trigger
Smart scan is not enabled for the sql in plsql when plsql package is called by 'CALL plsql‘
Does not work on serial queries issued from shared servers
The SQL in question must not be quarantined.
High CPU usage on the Storage Cells


What are Storage Indexes ?
Storage Indexes is a very useful Exadata feature which is transparent to database and are maintained automatically.
They are not indexes that are stored in the database like Oracle’s traditional B-Tree or bitmapped indexes. They are not capable of identifying a set of records that has a certain value in a given column. Rather, they are a feature of the storage server software that is designed to eliminate disk I/O.
They work by storing minimum and maximum column values for disk storage units, which are 1 Megabyte (MB) by default and are called region indexes.
Because SQL predicates are passed to the storage servers when Smart Scans are performed, the storage software can check the predicates against the Storage Index metadata (maximum and minimum values) before doing the requested I/O. Any storage region that cannot possibly have a matching row is skipped.

Since the indexes are in-memory on the cell server, if a cell server is restarted the SIs are lost and must be rebuilt.
They are generally created during the first smart scan that references a given column after a storage server has been restarted.
They can also be created when a table is created via a CREATE TABLE AS SELECT statement, or during other direct-path loads.



Under what conditions Storage Index would be used ?
In order for a storage index to be used, a query must include or make use of all the following :

1. Storage Indexes can only be used with statements that do Smart Scans. The main requirements are that the optimizer must choose a full scan and that the I/O must be done via the direct path read mechanism.
2. In order for a statement to use a Storage Index, there must be a WHERE clause with at least one predicate.
3. Storage Indexes can be used with the following set of operators: =, <, >, BETWEEN, >=, <=, IN, IS NULL, IS NOT NULL


What conditions prevent the use of Storage Indexes ?
1. Storage Indexes are not created on CLOBs.
2. Storage Indexes do not work with predicates that use the != comparison operator.
3. Storage Indexes do not work on predicates that use the % wildcard.
4. Storage Indexes are created and maintained for eight-columns per table.
5. Customer had set either of the following init.ora parameter

"_smu_debug_mode=134217728"

or alternatively

"_enable_minscn_cr"=false





What are the statistics related to Storage Indexes ?
There is only one database statistic related to storage indexes. The statistic, 'Cell Physical IO Bytes
Saved by Storage Index', keeps track of the accumulated I/O that has been avoided by the use of Storage Indexes.
Since the statistic is cumulative, it must be checked before and after a given SQL statement in order to
determine whether Storage Indexes were used on that particular statement.

Run the following query to check the statistics for Storage Indexes :
select name, value from v$sysstat where name like '%storage%';


How to diagnose Smart Scan causing wrong results ?
Refer Note 1260804.1 - Exadata: How to diagnose smart scan and wrong results.

Friday, October 20, 2017

12.1 Datapump Expdp Impdp Enhancements


Data Pump uses a multiprocess architecture in which a master control process dispatches job items to one or more worker processes. These worker processes may use PX processes to move data. 
Parallelism in this context refers to the total number of active worker processes and PX processes that are able to operate in parallel. 
The default PARALLEL value is one. If Data Pump decides to use PX processes to load or unload a table, then the PX processes are counted against the degree of parallelism used by Data Pump, but the worker process that starts the PX processes is not since it is idle while the PX processes are executing the query. Also, if a worker process has been started but it is idle, it is not counted against the limit specified by the PARALLEL parameter.

Datapump Job, Master Control Process creates a pool of data pump worker processes.
The degree of parallelism can be increased or decreased during execution of export/ import job.

The decrease in parallelism reduces the number of parallel workers. It may take a while to reduce the workers jobs.

The increase in parallelism takes affects immediately; 


Data Pump Import processes the database objects in the following order:
  1. The first worker begins to load all the metadata: the tablespaces, schemas, etc., until all the tables are created.
  2. Once the tables are created, the first worker starts loading data instead of metadata and the rest of the workers start loading data too.
  3. Once the table data is loaded, the first worker returns to loading metadata again. The rest of the workers are idle until the first worker loads all the metadata up to package bodies.
  4. Multiple workers load package bodies in parallel.
  5. One worker loads metadata up to and including secondary tables.
  6. Multiple workers load secondary table data.
  7. One worker loads the remaining metadata.

Bug 24423416 : IMPDP FOR SCHEMA_EXPORT/PACKAGE/PACKAGE_BODY TAKES HOURS
DataPump Import Is Extremely Slow When Importing Package Bodies (Doc ID 1452917.1)

If there are multiple partitions and subpartitions for a table, it takes high time to import the partition and subpartition segment information.
Here are things to perform speeding up the progress.
SQL> ALTER SYSTEM FLUSH SHARED_POOL;
SQL> ALTER SYSTEM SET STREAMS_POOL_SIZE=0;
SQL> ALTER SYSTEM SET STREAMS_POOL_SIZE=300M;


Use the parameter METRICS=Y to include additional logging information about the number of objects and the time it took to process them in the log file. With METRICS, additional information can be obtained about the number of objects that were processed and the time it took for processing them.

Tracing can be enabled by specifying an 7 digit hexadecimal mask in the TRACE parameter of expdp or impdp.TRACE does not add anything to the output of DataPump, it creates additional trace files in ADR_HOME location.
  10300 SHDW: To trace the Shadow process
  20300 KUPV: To trace Fixed table
  40300 'div' To trace Process services
  80300 KUPM: To trace Master Control Process
 100300 KUPF: To trace File Manager
 200300 KUPC: To trace Queue services
 400300 KUPW: To trace Worker process(es)        
 800300 KUPD: To trace Data Package
1000300 META: To trace Metadata Package
1FF0300 'all' To trace all components, full tracing
 eg:
expdp ..... EXCLUDE=statistics PARALLEL=16 TRACE=1FF0300




Tuesday, October 10, 2017

Analyzing B-Tree Indexes

Analyzing B-Tree Indexes

CREATE TABLE index_log (
 owner          VARCHAR2(30),
 index_name     VARCHAR2(30),
 last_inspected DATE,
 leaf_blocks    NUMBER,    
 target_size    NUMBER,
 idx_layout     CLOB);

ALTER TABLE index_log ADD CONSTRAINT pk_index_log PRIMARY KEY (owner,index_name);

CREATE TABLE index_hist (
 owner          VARCHAR2(30),
 index_name     VARCHAR2(30),
 inspected_date DATE,
 leaf_blocks    NUMBER,    
 target_size    NUMBER,
 idx_layout     VARCHAR2(4000));

ALTER TABLE index_hist ADD CONSTRAINT pk_index_hist PRIMARY KEY  (owner,index_name,inspected_date);

--
-- Variables:
--  vMinBlks: Specifies the minimum number of leaf blocks for scanning the index
--            Indexes below this number will not be scanned/reported on
--  vScaleFactor: The scaling factor, defines the threshold of the estimated leaf block count 
--                to be smaller than the supplied fraction of the current size. 
--  vTargetUse : Supplied percentage utilisation. For example 90% equates to the default pctfree 10 
--  vHistRet : Defines the number of records to keep in the INDEX_HIST table for each index entry
--

CREATE OR REPLACE PACKAGE index_util AUTHID CURRENT_USER IS
vMinBlks     CONSTANT POSITIVE := 1000;
vScaleFactor CONSTANT NUMBER := 0.6;
vTargetUse   CONSTANT POSITIVE := 90;  -- equates to pctfree 10  
vHistRet     CONSTANT POSITIVE := 10;  -- (#) records to keep in index_hist
 procedure inspect_schema (aSchemaName IN VARCHAR2);
 procedure inspect_index (aIndexOwner IN VARCHAR2, aIndexName IN VARCHAR2, aTableOwner IN VARCHAR2, aTableName IN VARCHAR2, aLeafBlocks IN NUMBER);
END index_util; 
/

CREATE OR REPLACE PACKAGE BODY index_util IS
procedure inspect_schema (aSchemaName IN VARCHAR2) IS
 begin
 FOR r IN (select table_owner, table_name, owner index_owner, index_name, leaf_blocks 
           from dba_indexes  
           where owner = upper(aSchemaname)
             and index_type in ('NORMAL','NORMAL/REV','FUNCTION-BASED NORMAL')
             and partitioned = 'NO'  
             and temporary = 'N'  
             and dropped = 'NO'  
             and status = 'VALID'  
             and last_analyzed is not null  
           order by owner, table_name, index_name) LOOP

   IF r.leaf_blocks > vMinBlks THEN
   inspect_index (r.index_owner, r.index_name, r.table_owner, r.table_name, r.leaf_blocks);
   END IF;
  END LOOP;
 commit;
end inspect_schema;
procedure inspect_index (aIndexOwner IN VARCHAR2, aIndexName IN VARCHAR2, aTableOwner IN VARCHAR2, aTableName IN VARCHAR2, aLeafBlocks IN NUMBER) IS
 vLeafEstimate number;  
 vBlockSize    number;
 vOverhead     number := 192; -- leaf block "lost" space in index_stats 
 vIdxObjID     number;
 vSqlStr       VARCHAR2(4000);
 vIndxLyt      CLOB;
 vCnt          number := 0;
  TYPE IdxRec IS RECORD (rows_per_block number, cnt_blocks number);
  TYPE IdxTab IS TABLE OF IdxRec;
  l_data IdxTab;
begin  
  select a.block_size into vBlockSize from dba_tablespaces a,dba_indexes b where b.index_name=aIndexName and b.owner=aIndexOwner and a.tablespacE_name=b.tablespace_name;
 select round (100 / vTargetUse *       -- assumed packing efficiency
              (ind.num_rows * (tab.rowid_length + ind.uniq_ind + 4) + sum((tc.avg_col_len) * (tab.num_rows) )  -- column data bytes  
              ) / (vBlockSize - vOverhead)  
              ) index_leaf_estimate  
   into vLeafEstimate  
 from (select  /*+ no_merge */ table_name, num_rows, decode(partitioned,'YES',10,6) rowid_length  
       from dba_tables
       where table_name  = aTableName  
         and owner       = aTableOwner) tab,  
      (select  /*+ no_merge */ index_name, index_type, num_rows, decode(uniqueness,'UNIQUE',0,1) uniq_ind  
       from dba_indexes  
       where table_owner = aTableOwner  
         and table_name  = aTableName  
         and owner       = aIndexOwner  
         and index_name  = aIndexName) ind,  
      (select  /*+ no_merge */ column_name  
       from dba_ind_columns  
       where table_owner = aTableOwner  
         and table_name  = aTableName 
         and index_owner = aIndexOwner   
         and index_name  = aIndexName) ic,  
      (select  /*+ no_merge */ column_name, avg_col_len  
       from dba_tab_cols  
       where owner = aTableOwner  
         and table_name  = aTableName) tc  
 where tc.column_name = ic.column_name  
 group by ind.num_rows, ind.uniq_ind, tab.rowid_length; 

 IF vLeafEstimate < vScaleFactor * aLeafBlocks THEN
  select object_id into vIdxObjID
  from dba_objects  
  where owner = aIndexOwner
    and object_name = aIndexName;
   vSqlStr := 'SELECT rows_per_block, count(*) blocks FROM (SELECT /*+ cursor_sharing_exact ' ||
             'dynamic_sampling(0) no_monitoring no_expand index_ffs(' || aTableName || 
             ',' || aIndexName || ') noparallel_index(' || aTableName || 
             ',' || aIndexName || ') */ sys_op_lbid(' || vIdxObjID || 
             ', ''L'', ' || aTableName || '.rowid) block_id, ' || 
             'COUNT(*) rows_per_block FROM ' || aTableOwner || '.' || aTableName || ' GROUP BY sys_op_lbid(' || 
             vIdxObjID || ', ''L'', ' || aTableName || '.rowid)) group by rows_per_block order by rows_per_block';
   execute immediate vSqlStr BULK COLLECT INTO l_data;
  vIndxLyt := '';

   FOR i IN l_data.FIRST..l_data.LAST LOOP
    vIndxLyt := vIndxLyt || l_data(i).rows_per_block || ' - ' || l_data(i).cnt_blocks || chr(10);
   END LOOP;

   select count(*) into vCnt from index_log where owner = aIndexOwner and index_name = aIndexName;

   IF vCnt = 0   
    THEN insert into index_log values (aIndexOwner, aIndexName, sysdate, aLeafBlocks, round(vLeafEstimate,2), vIndxLyt);
    ELSE vCnt := 0;

         select count(*) into vCnt from index_hist where owner = aIndexOwner and index_name = aIndexName;

         IF vCnt >= vHistRet THEN
           delete from index_hist
           where owner = aIndexOwner 
             and index_name = aIndexName 
             and inspected_date = (select MIN(inspected_date) 
                                   from index_hist
                                   where owner = aIndexOwner 
                                     and index_name = aIndexName);
         END IF;

          insert into index_hist select * from index_log where owner = aIndexOwner and index_name = aIndexName;

         update index_log  
         set last_inspected = sysdate,
             leaf_blocks = aLeafBlocks, 
             target_size = round(vLeafEstimate,2),
             idx_layout = vIndxLyt
        where owner = aIndexOwner and index_name = aIndexName;

   END IF;
  END IF;
 END inspect_index;
END index_util;
/