As the son of a former Marine Chief Warrant Officer (CW2) I have not only seen the occasional riot act being doled out to my brother; I have been on the receiving end of a few myself.  I have discovered that in order for your data quality initiatives to be successful you need someone who is willing to play the role of drill sergeant on occasion. There are going to be occasions where someone needs to take control when a business process or system provides the latitude to introduce bad data into the environment. This person is going to need to motivated and passionate about rooting out bad data and the processes that allow it to be introduced.

Recently I received an email from a business analyst inquiring about a discrepancy in the between the operational system and the analytical application that his team had recently distributed across the enterprise. The discrepancy was that a name change in the operational system was not being reflected in the analytical application. At the surface, this seemed fairly benign. We took a look at the table responsible for this particular entity the table below depicts what was discovered.

Since this table behaves as a Type 4 slowly changing dimension, an update of the last name and user id would have simply move the original record to an audit table and introduced a new record with the correct last name. Instead, a new record was created not only introducing a new surrogate key for the same person but the unique constraint on the employee id was bypassed by inserting a leading zero into the field.  This had to have been done by a user who was not properly trained on handling employee name changes in the operational system. A well written email can prevent this from happening again. The user could even be instructed to go back in to the system, delete the new record, and update the existing record properly. But that was not the case.

The users responsible for managing user accounts in the operational system have been instructed to create a new user account when someone submits a name change. These instructions also explicitly state to insert a leading zero for the employee id to bypass the uniqueness constraint placed on the field. What happens when an employee gets married then divorced and no longer wishes to retain her now ex-husband’s last name or marries again? Will they add a second leading zero to the employee id?

It was upon hearing the details of the business process that I could feel the increase in my blood pressure. This operational system has been live enterprise wide for less than six months and the business processes that support it are already showing signs of failure. History did not provide the learning experience that you would have hoped; it will take someone to put on their drill instructor uniform and break the privates in their platoon of their bad habits. Only then can they be built back up with understanding that they are to protect the enterprises data like it was their homeland.

In my previous blog post I showed how to build an Object Catalog of your Teradata data warehouse. Trouble is even on moderately sized systems your Object catalog could result in tens of thousands of rows. One of my object summaries is over thirty thousand rows. This can be overwhelming if you hand it off to management since they prefer a thirty thousand foot view.

So in order to provide a higher level overview of your Teradata system you can create a view on top of your object catalog that summarizes information at the database or user level providing information such perm space allocation/consumption, average table perm space consumption, average number of rows per table, number of tables with/without stats, and a count of each object type in the database.

The following view could be extended fairly easily to include additional metrics on such as the largest and smallest tables in both row count or perm space consumption. For my needs, I didn’t need to include those items.  The SQL can be found below:

REPLACE VIEW SysDBA.Database_Object_Overview AS
SELECT C.DatabaseName
     , CASE WHEN RowType = 'u' THEN 'User'
            WHEN RowType = 'D' THEN 'Database'
            ELSE NULL
       END DatabaseType
     , DatabaseAvailablePerm
     , SUM(TableSize) AS DatabaseSize
     , AVG(TableSize) AS AvgTableSize
     , AVG(TableRows) AS AvgTableRows
     , SUM(CASE WHEN TableMissingStats = 'Y' THEN 1 ELSE 0 END) TablesMissingStats
     , SUM(CASE WHEN TableMissingStats = 'N' THEN 1 ELSE 0 END) TablesWithStats
     , TablesWithStats/(NULLIFZERO(TableCount) * 1.000) AS PercentTableWithStats
     , COUNT(TableName) AS ObjectCount
     , SUM(CASE WHEN TableKind = 'T' THEN 1 ELSE 0 END) AS TableCount
     , SUM(CASE WHEN TableKind = 'V' THEN 1 ELSE 0 END) AS ViewCount
     , SUM(CASE WHEN TableKind = 'M' THEN 1 ELSE 0 END) AS MacroCount
     , SUM(CASE WHEN TableKind = 'P' THEN 1 ELSE 0 END) AS ProcedureCount
     , SUM(CASE WHEN TableKind = 'E' THEN 1 ELSE 0 END) AS ExternalCount
     , SUM(CASE WHEN TableKind = 'F' THEN 1 ELSE 0 END) AS FunctionCount
     , SUM(CASE WHEN TableKind = 'I' THEN 1 ELSE 0 END) AS JoinIndexCount
     , SUM(CASE WHEN TableKind = 'U' THEN 1 ELSE 0 END) AS UserDefinedTypeCount
     , SUM(CASE WHEN TableKind = 'S' THEN 1 ELSE 0 END) AS OLAPFunctionCount
     , SUM(CASE WHEN TableKind = 'R' THEN 1 ELSE 0 END) AS TableFunctionCount
     , SUM(CASE WHEN TableKind = 'G' THEN 1 ELSE 0 END) AS TriggerCount
     , SUM(CASE WHEN TableKind = 'D' THEN 1 ELSE 0 END) AS JARCount
     , SUM(CASE WHEN TableKind = 'Q' THEN 1 ELSE 0 END) AS QueueTableCount
     , SUM(CASE WHEN TableKind = 'N' THEN 1 ELSE 0 END) AS HashIndexCount
     , SUM(CASE WHEN TableKind = 'J' THEN 1 ELSE 0 END) AS JournalCount
     , SUM(CASE WHEN TableKind = 'H' THEN 1 ELSE 0 END) AS InstanceOrMethodCount
     , SUM(CASE WHEN TableKind = 'B' THEN 1 ELSE 0 END) AS CombinedAggOLAPCount
     , SUM(CASE WHEN TableKind = 'A' THEN 1 ELSE 0 END) AS AggregateFunctionCount
     , SUM(CASE WHEN TableKind = 'X' THEN 1 ELSE 0 END) AS AuthorizationCount
FROM SysDBA.Object_Catalog C
   , DBC.Dbase D
   , (SELECT DatabaseName
           , SUM(MaxPerm) AS DatabaseAvailablePerm
      FROM DBC.DiskSpace
      GROUP BY 1
     ) S
WHERE C.DatabaseName = D.DatabaseName
  AND S.DatabaseName = D.DatabaseName
GROUP BY 1,2,3
;

Recently I was asked to provide an inventory of a production Teradata system that included a list of all user created objects and as much information about the objects that I could gather included perm space usage, row counts, and object type. Two of the three could easily be gathered using the data dictionary tables in DBC. However, the row count requirement made me pause for a moment and weigh the impact this could have. The production system that I was working on had hundreds of databases and thousands of tables.

I knew writing an SQL script that would build the SELECT queries that would perform a COUNT(*) against every table and capture that in a flat file or table on the system was trivial. However, when you have a system with thousands or tens of thousands of objects it would take quite some time to run all of those statements even if many of the tables are relatively small. It just didn’t make good sense to take that approach.

Then I realized that nearly every table had some level of statistics collected against it and maintained at least semi-regularly depending on the DBA that was responsible for the database. The trouble is that Teradata’s HELP STATISTICS command only shows you the number of unique values for each set of columns containing statistics. But then I remembered that the view I used as part of my stats maintenance routine did have a column for the number of rows. The views that I use as the basis for my stats maintenance routine were written by Dieter Noeth and is available for the general public to use courtesy of the TeradataForum. For the sake of brevity and maintaining a single source for the code I am not going to post the SQL for the views here but you can find them here in the Attachment Library over at the TeradataForum. (It should be noted that there are two different sets of views that support both the 32 bit and 64 bit versions of Teradata. Make sure you use the correct set of views for your installation.)

In order to keep things organized there are three views used to gather the information instead of creating a bunch of derived tables. The view Stats_Data is responsible for collapsing the number of fields into a comma delimited list for the first 16 columns and then extracting the pertinent binary data for the statistics collected on a given set of columns. The second view Stats_Basics_[32/64]bit extrapolates the binary data from the Stats_Data view into individual binary fields that require further refining. The final view Stats_Details composes the binary data into its final numeric and temporal values from which additional reporting and analysis can be done.

With these three views in place the Object_Catalog view could easily be constructed by simply bringing together a view additional data dictionary tables and the Stats_Details view. The Object_Catalog view displays the following attributes about the objects in each database on the system:

• Database Name – self-explanatory
• Object Name – self-explanatory
• Column Count – This is a count of the number of fields for the object in TVFields.
• Table Rows – The number of rows on the table when the stats were last collected. NULL if stats are not collected.
• Table Size – The current perm space being consumed by the table.
• Table Missing Stats – Indicator field for if stats exist on the table. NULL for non-table objects.
• Table Kind – A single character field categorizing the object type. (e.g. T, V, M, etc.)
• Table Kind Description – A description of the object type (e.g. Table, View, Macro, etc.)

The SQL for the Object_Catalog view can be found below:

REPLACE VIEW SysDBA.Object_Catalog AS
SELECT DISTINCT
       T.DatabaseName
     , T.ObjectName
     , DT4.ColumnCount
     , CAST(DT2.NumRows AS BIGINT) AS TableRows
     , DT3.TableSize
     , CASE WHEN T.Tablekind = 'T'
             AND TableRows IS NULL
            THEN 'y'
            WHEN T.Tablekind = 'T'
             AND Tablerows > 0
            THEN 'n'
            ELSE NULL
       END AS TableMissingStats
     , T.TableKind
     , CASE WHEN T.TableKind = 'A' THEN 'Aggregate function'
            WHEN T.TableKind = 'B' THEN 'Combined aggregate and ordered analytical function'
            WHEN T.TableKind = 'D' THEN 'JAR'
            WHEN T.TableKind = 'E' THEN 'External Stored Procedure'
            WHEN T.TableKind = 'F' THEN 'Standard function'
            WHEN T.TableKind = 'G' THEN 'Trigger'
            WHEN T.TableKind = 'H' THEN 'Instance or Constructor Method'
            WHEN T.TableKind = 'I' THEN 'Join Index'
            WHEN T.TableKind = 'J' THEN 'Journal'
            WHEN T.TableKind = 'M' THEN 'Macro'
            WHEN T.TableKind = 'N' THEN 'Hash Index'
            WHEN T.TableKind = 'P' THEN 'SQL Procedure'
            WHEN T.TableKind = 'Q' THEN 'Queue Table'
            WHEN T.TableKind = 'R' THEN 'Table function'
            WHEN T.TableKind = 'S' THEN 'Ordered analytical function'
            WHEN T.TableKind = 'T' THEN 'Table'
            WHEN T.TableKind = 'U' THEN 'User-defined data type'
            WHEN T.TableKind = 'V' THEN 'View'
            WHEN T.TableKind = 'X' THEN 'Authorization'
            ELSE NULL
       END AS TableKindDescription
FROM DBC.TABLES T
 LEFT JOIN
    (SELECT DatabaseName
          , TableName
          , NumRows
     FROM SysDBA.Stats_Details
     QUALIFY CollectTimeStamp
           = MAX(CollectTimestamp)
             OVER(PARTITION BY DatabaseName, TableName)
         AND ROW_NUMBER()
             OVER(PARTITION BY DatabaseName, TableName
                  ORDER BY CollectTimestamp DESC)
           = 1
    )     DT2
        ON T.DatabaseName = DT2.DatabaseName
       AND T.TableName    = DT2.TableName
LEFT JOIN
    (SELECT DatabaseName
          , TableName
          , SUM(CurrentPerm) AS TableSize
     FROM DBC.TableSize
     GROUP BY 1,2
    ) DT3
       ON T.DatabaseName = DT3.DatabaseName
      AND T.TableName    = DT3.TableName
LEFT JOIN
    (SELECT D.DATABASENAME
          , T.TVMNAME        AS TableName
          , COUNT(C.FieldId) AS ColumnCount
     FROM DBC.TVM T
        , DBC.TVFields C
        , DBC.Dbase D
     WHERE T.TVMId = C.TableId
       AND T.DatabaseId = D.DatabaseId
     GROUP BY 1,2
    ) DT4
      ON T.DatabaseName = DT4.DatabaseName
     AND T.TableName    = DT4.TableName
;