SQL Archives - Page 2 of 5

Data histogram equalization

The Document Attachments were stored in our AX 2012 database, and were occupying several hundred GBs. As part of the migration to D365FO we had to export these files in bulk. The most efficient way is to run the batch task in parallel. But this data is tilted in a way that if you would group the tasks based on their creation date, it the first tasks would barely have any records to process while the last ones would go on forever. The solution for this is called data histogram equalization.

It would be a big challenge to code this in X++, but SQL Server has a function for doing this exactly: NTILE.

The following direct query is able to process the data histogram, and then return 10 buckets of Record Identifier ranges of roughly equal size:

WITH documents (Bucket, RecId)
AS (
       SELECT NTILE(10) OVER( ORDER BY DocuRef.RecId) AS Bucket
              ,DocuRef.RECID
    FROM docuRef
        INNER JOIN docuType
            ON  (docuType.dataAreaId = docuRef.RefCompanyId OR DOCUREF.REFCOMPANYID = '')
                AND docuType.TypeId = docuRef.TypeId
        INNER JOIN docuValue
            ON  docuValue.RecId          = docuRef.ValueRecId                
       WHERE  docuType.TypeGroup         = 1 -- DocuTypeGroup::File
        AND docuType.FilePlace           = 0 -- DocuFilePlace::Archive
        AND docuType.ArchivePath  <> ''
              AND docuValue.Path                = ''
       )
 
SELECT Bucket
       ,count(*) AS Count
       ,(SELECT MIN(RecId) FROM documents D WHERE D.Bucket = documents.Bucket) AS RecId_From
       ,(SELECT MAX(RecId) FROM documents D WHERE D.Bucket = documents.Bucket) AS RecId_To
       FROM documents
       GROUP BY Bucket

Here are the results for spawning 10 batch job tasks to do parallel execution based on the RecId surrogate key index, with ~57230 rows in each bucket. This allows you to evenly distribute the load for data processing in parallel.

If we would export our document attachments sequentially, it would take roughly 40 hours total. By utilizing the data histogram equalization we could get it down to be under 3 hours. That is amazing!

It is a great way to split ranges of data for parallel processing, to keep in mind for the future.

You can learn more about the functionality here:

https://www.sqlitetutorial.net/sqlite-window-functions/sqlite-ntile/

By DAXRunBase|2022-03-31T16:42:48+02:00March 31st, 2022|Categories: AX 2012, Dynamics 365 for Finance and Operations, MSDyn365FO|Tags: #MSDyn365FO, AOT, AX 2012, batch, metadata, SQL, X++|0 Comments

What are the SQL Server Jobs which looks like a GUID?

If you have ever encountered SQL Server Jobs which looks like a GUID value, you might have wondered what they are used for.

Our AX 2012 deployment is using a SQL Server Reporting Services instance, which is used for two purposes:

AX 2012 SSRS reports
Custom reporting from our Data warehouse

These jobs are actually periodically generated SSRS reports scheduled by our staff. Each job belongs to a different person and report.

By just having a look at the job itself does not reveal what is actually being executed, but the following T-SQL script can extract the details when you run it against the ReportServer database:

SELECT
c.Name AS ReportName,
S.ExtensionSettings,
'Next Run Date' = CASE next_run_date
WHEN 0 THEN null
ELSE
substring(convert(varchar(15),next_run_date),1,4) + '/' +
substring(convert(varchar(15),next_run_date),5,2) + '/' +
substring(convert(varchar(15),next_run_date),7,2)
END,
'Next Run Time' = isnull(CASE len(next_run_time)
WHEN 3 THEN cast('00:0'
+ Left(right(next_run_time,3),1)
+':' + right(next_run_time,2) as char (8))
WHEN 4 THEN cast('00:'
+ Left(right(next_run_time,4),2)
+':' + right(next_run_time,2) as char (8))
WHEN 5 THEN cast('0' + Left(right(next_run_time,5),1)
+':' + Left(right(next_run_time,4),2)
+':' + right(next_run_time,2) as char (8))
WHEN 6 THEN cast(Left(right(next_run_time,6),2)
+':' + Left(right(next_run_time,4),2)
+':' + right(next_run_time,2) as char (8))
END,'NA'),
Convert(XML,[ExtensionSettings]).value('(//ParameterValue/Value[../Name="TO"])[1]','nvarchar(100)') as [To]
,Convert(XML,[ExtensionSettings]).value('(//ParameterValue/Value[../Name="CC"])[1]','nvarchar(100)') as [CC]
,Convert(XML,[ExtensionSettings]).value('(//ParameterValue/Value[../Name="BCC"])[1]','nvarchar(100)') as [BCC]
,Convert(XML,[ExtensionSettings]).value('(//ParameterValue/Value[../Name="RenderFormat"])[1]','nvarchar(50)') as [Render Format]
,Convert(XML,[ExtensionSettings]).value('(//ParameterValue/Value[../Name="Subject"])[1]','nvarchar(50)') as [Subject]
/*,Convert(XML,[Parameters]).value('(//ParameterValue/Value[../Name="StartDateMacro"])[1]','nvarchar(50)') as [Start Date]
,Convert(XML,[Parameters]).value('(//ParameterValue/Value[../Name="EndDateMacro"])[1]','nvarchar(50)') as [End Date]
,Convert(XML,[Parameters]).value('(//ParameterValue/Value[../Name="Currency"])[1]','nvarchar(50)') as [Currency]*/
,[LastStatus]
,[EventType]
,[LastRunTime]
,[DeliveryExtension]
,[Version]
FROM 
 dbo.[Catalog] c
INNER JOIN dbo.[Subscriptions] S ON c.ItemID = S.Report_OID
INNER JOIN dbo.ReportSchedule R ON S.SubscriptionID = R.SubscriptionID
INNER JOIN msdb.dbo.sysjobs J ON Convert(nvarchar(128),R.ScheduleID) = J.name
INNER JOIN msdb.dbo.sysjobschedules JS ON J.job_id = JS.job_id

Now we have the details of:

what reports were scheduled
the report format
execution status
schedule and last successful run time.

This should give you a better overview about what are the SQL Server Jobs which looks like a GUID value in your database instance.

By DAXRunBase|2022-03-31T11:46:55+02:00March 31st, 2022|Categories: AX 2012, Microsoft Dynamics AX|Tags: AX 2012, SQL|0 Comments

Number sequence consumption monitoring

Once a company has been live for a while and they are doing Business As Usual, they often forget about maintaining one critical area. The system is using Number Sequences as identifiers, for which we have fixed, allocated range of values. Typical example is a general financial journal with a format like JNL18-###### with values between 000001 and 999999. Number Sequence consumption monitoring is essential!

If the sequence reaches the maximum value, you are no longer able to create new financial journals. Sometimes it is obvious when a number is about to run out, but in many cases AX has it hidden on a transactional level that is not necessarily visible for the users. When the sequence runs out, it can cause serious issues:

database locking
error messages
rolled back transactions.

Proactive monitoring is key to a healthy ERP system on many levels. Number sequence consumption monitoring is no exception. We are running a Transact-SQL script that keeps tracking of the number sequence utilization, and sends out an e-mail with entries reaching a set threshold. We are running the job based on a weekly schedule. We include any sequences that have used up at least 70% of their available range.

(more…)

By DAXRunBase|2022-03-29T18:35:13+02:00March 31st, 2020|Categories: AX 2012|Tags: AX 2012, database, Monitoring, Number sequence, SQL|0 Comments

Archiving SQL database backups using Azure blob storage

It is a good practice to keep multiple copies of our most precious data. By using on-premises SQL Server databases for AX 2012 or Dynamics 365 Finance and Operations, archiving SQL database backups to offsite-locations are a must. I have built automation for archiving SQL database backups using Azure Blob Storage.

Overview of the processes

Maintenance regime

Our maintenance regime looks like the following:

1x Weekly Full backup
6x Daily Differential backup
15 minute Transactional log backups

They are captured locally on the primary SQL instance, to keep the timestamps for last successful backups in our AlwaysOn cluster. Then we move the files to a shared network storage, which is visible to both High Availability sites, in case there is an outage and we need to a fail over and restore data.

In case of a natural disaster due to the close geographical proximity of the sites we needed an additional layer of safety.

Archiving offsite

Every night we are running a PowerShell script that uses the AzCopy utility. It is uploading our backup files on a Microsoft Azure cloud storage account.

You are paying for the network usage (IO) and the size occupied on the disks, so it is a good idea to have some sort of housekeeping. Our solution was to use an Azure RunBook to determine what to keep and what to delete. The current setup is to have 1 full backup file for a year available every quarter (4x 400 GB), and keep all full / differential / transactional files for the last month (4x 400 GB + 26x 10 GB).

This puts the total size occupied around 4 TB, and costs about 35 GBP a month using a cold storage. This price could go up if you also want to utilize a hot tier storage for the latest backup set. That is useful if you want to come back from the cloud on a short notice.

(more…)

By DAXRunBase|2020-03-23T13:39:50+01:00March 19th, 2020|Categories: AX 2012, Dynamics 365 for Finance and Operations, MSDyn365FO|Tags: #MSDyn365FO, archive, Azure, backup, Blob Storage, database, Dynamics 365, GitHub, PowerShell, SQL|0 Comments

Trace database cleanup in an efficient way

The Trace parser is an excellent tool for troubleshooting business functionality within Microsoft Dynamics AX 2012 and Dynamics 365 for Finance and Operations. Your Trace database can quickly grow large and it does affect the tools’ performance adversely.

Removing the old trace collections one-by-one is time consuming. You could utilize a stored procedure for cleaning up your AX trace database efficiently in SQL:

-- Replace AXTrace with your database name
USE [AXTrace]
GO
 
-- Truncate transaction log to reduce size
DBCC SHRINKFILE (N'AXTrace_log' , 0, TRUNCATEONLY)
GO
 
-- Iterate through the list of traces in the database
DECLARE cur CURSOR FOR 
       SELECT [TraceId] FROM [dbo].[Traces]
 
DECLARE @Id AS int
 
OPEN cur;
FETCH NEXT FROM cur into @Id;
 
-- Remove all traces in the database with the DeleteTrace stored procedure
WHILE (@@FETCH_STATUS = 0)
BEGIN
       EXEC [dbo].[DeleteTrace] @TraceId = @Id
       FETCH NEXT FROM cur into @Id;
END
 
CLOSE cur;
DEALLOCATE cur;
 
-- Truncate transaction log to reduce size
DBCC SHRINKFILE (N'AXTrace_log' , 0, TRUNCATEONLY)
GO

By DAXRunBase|2020-03-23T13:21:53+01:00March 31st, 2019|Categories: AX 2012, Dynamics 365 for Finance and Operations|Tags: AX 2012, cleanup, database, SQL, trace|0 Comments