wiseDATAman

T-SQL Tuesday #186 - Managing Agent Jobs

Tue, 13 May 2025 21:52:33 +0000

For T-SQL Tuesday #186, Andy Levy asks, “How do you manage and/or monitor your SQL Server Agent jobs?”

This is a great opportunity for me to discuss how DBA Dash can help monitor SQL Agent jobs. DBA Dash is a free and open-source monitoring tool for SQL Server, created by me. It’s used to monitor thousands of SQL Server instances within Trimble alone, and it’s gaining popularity in the SQL Server community.

Built-in alerts for job failures

Before we discuss DBA Dash, what can we do within SQL Server without any 3rd party tools?

SQL Agent jobs have native alert notifications. You can get an email whenever a job fails, succeeds, or completes.

Issues with built-in alerts:

If a frequently executed job starts failing, this can generate a lot of noise.
A workaround is to set a retry policy and retry interval for your job steps.
A lack of a failure notification doesn’t mean that everything is in a good state.
You should validate the status of your SQL agent jobs in combination with alerts. Job activity monitor in SSMS could be used for this, but it doesn’t scale if you have many SQL instances to monitor.
It requires database mail. There are no modern notification channels like Slack, Google chat, PagerDuty etc.

DBA Dash

DBA Dash includes several useful features to help you monitor your SQL Agent jobs.

Summary page

First up is the summary page. This is a single-page report that can tell you if you have failing jobs across your whole SQL Server estate. It can also tell you if your SQL Server agent is stopped on any of those instances. The summary page includes many other things you should include as part of your daily checks, in addition to monitoring agent jobs.

From the summary page, you can drill down and see specific jobs that are failing across your instances. You can click to view the failed steps and messages.

Thresholds and exclusions can be configured at the root, instance or job level.

Alerts

DBA Dash has an AgentJob alert rule that you can use for instant notifications. You can target a specific server, a group of servers (using tags), and filter by category and job name. Your notification options currently include Email, Slack, PagerDuty and Webhook (e.g. Google Chat).

This provides more modern notification options over built-in alerts and allows alerts to be configured centrally. You can control when you are notified and by what mechanism, depending on the priority of the alert.

Job Performance

You might also be interested in tracking job performance over time. DBA Dash allows you to do this centrally. The msdb database maintains a history of job execution, but the DBA Dash repository database enables you to store the execution metrics for a long time with minimal storage cost. You can also visualize the job performance over time very easily.

Timeline

The timeline view is a great way to see which jobs ran when and how long they ran for. This is useful to help plan job schedules, among other things.

Running Jobs

Want to know what jobs are running right now across all your instances?

Schema Tracking

DBA Dash scripts your SQL agent jobs, tracking when changes occur. This can be used to help answer what changed when and provides access to previous versions of jobs, should you need to roll back.

If you can source control your SQL agent jobs, that is also a great idea. DBA Dash can complement that.

This feature is also used to provide a SQL agent job DIFF tool so you can compare jobs between SQL instances.

More to come

As I’m writing this, I’m working on a Job Info page to provide job details on a single page (Steps, schedules, and other metadata). I’m using ScriptDom to parse this information, and with the DDL tracking, I can display this info for past versions of a job. I can also use this to display an alternative DIFF format.

DBA Dash includes a Running Query capture for performance monitoring (similar to sp_WhoIsActive or sp_BlitzWho). This will show you the job name if a query is associated with a running job. In the next version of DBA Dash, this will be a link, providing you with immediate access to the job details and execution stats.

~~These features should make their way into the next version of DBA Dash, 3.23. Or they might already be available by the time you read this.~~

Edit: 3.23 is here now! 🚀

Thanks for reading!

Removing large backup history with sp_delete_backuphistory

Sun, 04 Feb 2024 15:04:57 +0000

Problem

In SQL Server, the sp_delete_backuphistory stored procedure is is the standard and supported method to purge old data from the backup and restore history tables in the msdb database:

backupfile
backupfilegroup
backupmediafamily
backupmediaset
backupset
restorefile
restorefilegroup
restorehistory

Recently, I encountered an issue running sp_delete_backuphistory on servers that hosted a large number of databases with frequent log backup & restore operations. The clean up task hadn’t been scheduled and the history tables had grown very large over several months. The msdb databases was also hosted on a volume with limited IOPs.

Attempting to run sp_delete_backuphistory under these conditions you will likely encounter these issues:

The operation will take a long time
Log backup and restore operations are blocked during the process
Your msdb log file will grow large if you try to remove too much data at once
If you kill off the process at any point, the whole thing will rollback.

The sp_delete_backuphistory stored procedure gets the job done in most cases but it’s not very well optimized if you have a large amount of history to clear.

Optimized Solution for Backup History Cleanup

To clear a large amount of history without resorting to schema modifications or unsupported methods we can take an incremental approach to removing data.

The script below will check how much retention we currently have. It will then call sp_delete_backuphistory in a loop, removing data 1hr (configurable) at a time until we hit the target retention. We can also specify a maximum runtime for the process.

This approach won’t necessarily speed things up but it provides the following benefits:

Data is cleared as a series of smaller transactions rather than a single large transaction
Smaller transactions allow for log truncation and prevent the log file from growing.
We can interrupt the process without having to rollback all the progress that has been made. (just the current batch)
Log backups/restores might still be blocked, but only for the duration it takes to complete a single loop - not for the entire time it takes to clear the history. This can still impact RPO/RTO though.
We can control how long the process runs for. We can keep it running outside business hours or allow for breaks to prevent log backup/restores from falling too far behind.
The script will print progress information.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76


/********************* Params *******************/
DECLARE @RetentionDays INT = 30 /* How long should we keep the data for in days */
DECLARE @Mins INT = 60 /* Delete data X minutes at a time. Small interval keeps transaction short. Less blocking and log usage*/
DECLARE @MaxRuntimeMins INT = 60 /* Stop processing if runtime exceeds this threshold */
/************************************************/

DECLARE @TargetDate DATETIME
DECLARE @oldest_date DATETIME
DECLARE @Message NVARCHAR(4000)
DECLARE @t1 DATETIME
DECLARE @Duration INT
DECLARE @TotalDuration BIGINT=0
DECLARE @IterationCount INT=0
DECLARE @StopAt DATETIME

/* The target date we want to get to based on the specified @RetentionDays */
SET @TargetDate = DATEADD(dd,-@RetentionDays,GETDATE())
/* When should we stop processing batches based on @MaxRuntimeMins */
SET @StopAt = DATEADD(mi,@MaxRuntimeMins,GETDATE())

/* Get the oldest backup finish date from table. We will start removing data older than this point + @Mins minutes */
SELECT @oldest_date = MIN(backup_finish_date)
FROM msdb.dbo.backupset

PRINT CONCAT('Oldest Date: ',@oldest_date)
PRINT CONCAT('Target Date: ', @TargetDate)
PRINT CONCAT('Max Run Time (Mins): ',@MaxRuntimeMins,', Processing will stop after ',@StopAt)

/* Process the delete operation in small chunks until we reach the target date for our retention */
 WHILE 1=1
 BEGIN
	/* Increment date by @Mins minutes */
	SET @oldest_date = DATEADD(mi,@Mins,@oldest_date)
	/* Ensure that we don't overshoot our target date */
	IF @oldest_date > @TargetDate
		SET @oldest_date = @TargetDate	
		
	SET @IterationCount = @IterationCount + 1
	/* 
	Print message using RAISERROR WITH NOWAIT so message is printed immediately. 
	For keeping track of progress 
	*/
	SET @Message = CONCAT(CONVERT(VARCHAR,GETDATE(),114),' Start iteration ',@IterationCount,': Removing backup history before ',@oldest_date)
	RAISERROR(@Message,0,1) WITH NOWAIT
	SET @t1 = SYSUTCDATETIME()
	/* Remove the history up to @oldest_date (@Mins minutes worth of history) */
	EXEC msdb.dbo.sp_delete_backuphistory @oldest_date = @oldest_date;

	/* Print how long this iteration took & keep track of total duration and iteration count */
	SET @Duration = DATEDIFF(ms,@t1,SYSUTCDATETIME())
	SET @TotalDuration = @TotalDuration + @Duration 
	SET @Message = CONCAT(CONVERT(VARCHAR,GETDATE(),114),' Iteration ', @IterationCount,' completed in (hh:mi:ss:mmm): ',CONVERT(VARCHAR,DATEADD(ms,@Duration,0),114))
	
	RAISERROR(@Message,0,1) WITH NOWAIT

	/* Stop once we hit our target date for retention or we have ran for longer than @MaxRuntimeMins */
	IF @oldest_date>=@TargetDate 
		OR GETDATE() > @StopAt
		BREAK

 END

 /* 
	Print overall stats 
	Accounting for possibility that total duration could span > 1 day
 */
SET @Message = CONCAT('Completed ',
				@IterationCount,
				' iterations in ',
				CAST(NULLIF(DATEDIFF(d,0,DATEADD(s,@TotalDuration/1000,0)),0) AS NVARCHAR(4000)) + ' Days ',
				LEFT(CONVERT(VARCHAR,DATEADD(s,@TotalDuration/1000,0),114),8),
				' (hh:mi:ss), Avg Duration: '
				,CONVERT(VARCHAR,DATEADD(ms,@TotalDuration/NULLIF(@IterationCount,0),0),114), 
				' (hh:mi:ss:mmm). Reached target date: ',CASE WHEN @oldest_date>=@TargetDate THEN 'YES' ELSE 'NO' END )

RAISERROR(@Message,0,1) WITH NOWAIT

Links:

These related links might be interesting:

https://www.brentozar.com/archive/2018/02/the-annals-of-hilariously-bad-code-part-1/
https://www.brentozar.com/archive/2018/02/annals-hilariously-bad-code-part-2/
https://dba.stackexchange.com/questions/233375/how-to-minimize-plan-ahead-the-effect-of-running-sp-delete-backuphistory-measur

T-SQL Tuesday #156 Production Code

Tue, 08 Nov 2022 08:31:01 +0000

Introduction

The invitation from Tom Zíka asks “Which quality makes code production grade?”.

Reliability

Code that runs in production should be reliable. It should produce consistent results, handle concurrency and deal with error situations.

Performance & Scalability

Production code needs to be able to scale. Scalability should be considered in the context of your own applications and their expected growth. Not everyone has to deal with the same scalability challenges as Google, Facebook or Twitter.

Performance is also important. It doesn’t matter how awesome your app is if it can’t return data fast enough for end users.

Fast and efficient code can save $$$ in running costs and keep end users happy.

Security

Production code needs to be secure, following industry best practices. A security incident can cause severe reputational damage and financial impact.

Maintainability

Production code should be maintainable. Avoiding duplication and striving for simplicity.

Reviewed

Production code will often go through a code review by another developer before it’s accepted into the codebase.

Tested

Production code will go through a series of automated and human tests before it gets to production.

Accurate?

Production code doesn’t always have to produce accurate results. Accuracy can be very important in some situations. In others, you might be able to trade some accuracy for speed. e.g. Caching data, eventual consistency.
The number of active users on a website doesn’t need to be 100% accurate and can be out of date by the time it’s printed.

Documented

Production code should have comments and documentation.

Other

You might have policies for coding standards to follow. This might include things like naming conventions, code formatting and other best practices.

Does it need to be perfect?

Production code doesn’t have to be perfect. Perfect code doesn’t really exist. The code we write today will be viewed through a different lens tomorrow with the benefit of hindsight, experience and new improved ways of doing things.

The code we ship to production needs to be fit for purpose. Shipping code that works and delivers value in a timely manner is often more important than spending months trying to find the most optimal solution. There is a balance to be struck.

Architectural decisions made early in a project can have long-lasting consequences. Making good decisions early in a project can be cheaper than fixing them later.

You can’t always predict how users will use a new feature and how to optimize it. Growing pains are inevitable for successful applications. It’s important to monitor and be proactive in fixing problems.

Not production grade

So what are some examples of code that shouldn’t be in production?

Code that uses undocumented/unsupported features or behaviour
Untested code
Insecure code. e.g. Building dynamic SQL in a way that is prone to SQL injection.

Summary

Production code can vary in quality for a variety of reasons. The experience level of the developer writing the code, time pressures, and level of fatigue all impact code quality.

It’s not unusual for code written as a quick PoC to find its way into production. Code can also start to get messy over time as you start to shoehorn new features into an existing codebase.

If your code is delivering value to the end-user or business you can feel happy about it. Over time we all accumulate technical debt and need to allocate time to reduce it.

Speaking at Data Relay

Thu, 13 Oct 2022 12:17:23 +0000

Data Relay is a free community conference in the UK for the Microsoft data platform. I submitted a session for DBA Dash which was accepted and I delivered my session on the 4th October 2022 in Leeds.

This was the first time I’ve done something like this and it was outside of my comfort zone. I was excited to show off DBA Dash though. My plan was to demo how to use the tool to solve performance issues.

I created demos in my Lab environment and restored the repository database to my laptop to perform the demos.

To help prepare for speaking at data relay I recorded myself. This allowed my to practice my presentation and get an idea how long it would take to deliver. The actual session at data relay wasn’t recorded but I’ve uploaded my practice session to YouTube. It’s a bit rough around the edges, but hopefully you will find the video useful. 😊

https://youtu.be/pGro__GkeDM

Thank you to @MrKeithAtherton for providing the photo from the event!

What's new in 2.23.0

Mon, 10 Oct 2022 15:54:37 +0000

OS Loaded Modules Reporting

Some modules such as antivirus software are known to cause issues when loaded into SQL Server address space. These problems might vary from performance issues to weird bugs and corruption. DBA Dash can now highlight where you have problematic modules loaded.

New website for DBA Dash!

Sun, 02 Oct 2022 13:43:57 +0000

DBA Dash has a new website. The purpose of this site is to provide a friendly landing page for new users of DBA Dash and to provide a better documentation experience.

DBA Dash remains open source and the GitHub page isn’t going anywhere. GitHub is still the place to log issues, create discussions and contribute to the project.

I hope you like the new website and find the documentation easier to navigate. I want to improve the documentation experience over time.

The dbadash.com website was created with Doks. Doks is a theme for Hugo, a static website generator. This the first time I’ve used a static site generator. There is a bit of a learning curve compared to something like WordPress. Static websites are fast, easy to source control and cheaper to host though.

T-SQL Tuesday #154 Invitation – SQL Server 2022

Tue, 13 Sep 2022 08:03:38 +0000

Introduction

This is my first time participating in T-SQL Tuesday. The Invitation from Glenn Berry asks to write about what you’ve been doing with SQL 2022.

Getting DBA Dash ready for SQL 2022

I was eager to test DBA Dash with SQL 2022 and started testing with the first public release (CTP 2.0). I added a SQL 2022 instance to my lab environment, created using AutomatedLab.

Unlike my other lab machines, I ran through the installer for 2022 manually to get up and running quickly and test the new installer.

DBA Dash only needed some minor modifications to work with SQL 2022. The first and most important step was to add version 16 to the hardcoded list of supported versions and also update some static data for SQL Server 2022. I hardcode the list of supported versions to give me the opportunity to test new versions first and fix any issues.

This initial support for SQL 2022 was added in May. I’ve been running DBA Dash in my lab environment since then and it works well.

DBA Dash - support for new features in SQL 2022

QAT_DEFLATE Backup Compression

Version 2.22.0 adds support for capturing the compression algorithm used on SQL 2022.

Glenn Berry has some good articles on this new backup compression algorithm here and here. I’m getting similar results to Glenn with my own testing using QAT_DEFLATE (software mode) - faster backups, higher compression and lower resource utilization.

sys.databases

SQL 2022 adds some new columns to sys.databases:

is_data_retention_on
is_ledger_on
is_change_feed_enabled

I’m planning to add these to the DBA Dash collection. Only is_ledger_on is currently documented.

Other

On the to-do list is to test DBA Dash with contained availability groups. DBA Dash has monitoring for agent jobs and availability groups so it’s likely some changes will be required in this area.

What I’m excited about in SQL 2022

The intelligent query processing features in SQL 2022 and tempdb concurrency enhancements are very interesting. There is also some really useful language enhancements like GENERATE_SERIES, DATE_BUCKET, GREATEST and LEAST that I can see myself using. Null handling also gets easier with IS [NOT] DISTINCT FROM. I can’t use any of this for DBA Dash as I want to keep the repository compatible with SQL 2016 for now. The language enhancements will be useful for some of the SaaS databases I manage though once we upgrade.

The ability to failover back and forth between Azure managed instance and SQL 2022 is also a game changer. It certainly takes some of the risk out of the process if you are considering using managed instances.

The feature I’m most excited about? Backup/Restore to S3. If you host SQL instances on AWS you have to figure out how to get your backups to S3. There isn’t a standard way to do this. Many people backup to an EBS volume first then push the files to S3 which isn’t ideal. EBS storage is expensive and the two step process to get the backups to S3 adds time and complexity to the process. The restore process is also slower and more complex as you need to pull the backups from S3 first before you can restore them.

The ability to backup directly to S3 is a huge win. It also has value outside of AWS as there are a number of other S3 compatible storage providers.

September Free Community Tools Awareness Month

Thu, 08 Sep 2022 21:05:28 +0000

I’m writing this post to join in with September Free Community Tools Awareness Month. There are many awesome free community tools. I’ve chosen to blog about 3 of them that I think are interesting any maybe less well known in the database community. I’m also taking the opportunity to shamelessly plug my own tool, DBA Dash.

DBA Dash

DBA Dash is a monitoring tool for SQL Server. It’s something I use every day and it provides value in several different ways. It supports everything from SQL 2005 to SQL 2022 and Azure DB.

🏎️If you have a performance issue, DBA Dash can help you get to the root cause. You can troubleshoot an issue that is occurring right now or do a postmortem for an issue that occurred weeks ago.

🏥It’s also a tool for health checks. Do you have missing backups, agent job failures, or corruption? Is log shipping, availability groups, or mirroring broken? Are your servers running out of disk space? DBA Dash checks all of these and more. Recently added is a check for identity columns running out of values.

⚙️DBA Dash captures configuration data for your whole SQL Server estate (config settings, trace flags, tempdb). Use the configuration data to check for configuration settings that are different between your SQL instances - great for validating the config of new SQL instances. Check the patch level of your SQL instances. Keep track of when updates were installed and configuration changes made.

l created this tool and I’m also it’s #1 user. The tool is totally free without restrictions or limitations - even better it’s open source. Get it on GitHub.

This video will give you a quick overview of the tool.

https://www.youtube.com/watch?v=X7e4zElOQ3c

_Note: Th_is video was created in January when the tool first launched. Many new features and enhancements have been made since the video was created.

AutomatedLab

AutomatedLab enables you to quickly set up lab environments on Hyper-V and Azure. I use this to create lab environments for DBA Dash on Hyper-V.

The lab environments enable me to test DBA Dash with SQL versions from 2005 to 2022 as well as availability groups, and different configurations. I also use the lab environment as a general playground for learning and experimentation - something useful to have for any DBA.

The power of AutomatedLab is that I can re-create my lab by running a PowerShell script and I can tear it down just as easily. Infrastructure as code!

dbatools is also a key component to setting up my lab environment and it’s one of my favourite community tools. Use this for automating anything relating to SQL Server.

Pssdiag/SqlNexus

This is a monitoring tool for SQL Server, but unlike DBA Dash it’s not a tool for 24/7 monitoring. I don’t use this tool very often but it can be useful if you are dealing with a particularly tricky SQL issue.

I’ve been a DBA for 17 years and I’ve had a small number of edge case issues that required a phone call to Microsoft product support. During these engagements I’ve been asked to run a pssdiag session and upload it for their offline analysis. You can create your own pssdiag session with SqlDiag Manager and analyse it with SqlNexus.

I would run this tool for short periods of time while you are experiencing particularly tricky issues and use other tools like DBA Dash for your regular monitoring.

This is a Microsoft tool, but like the other tools on the list it’s also open source!

DBA Dash – What’s new in 2.22.0? 🎉

Thu, 08 Sep 2022 10:34:52 +0000

Identity Columns collection

If your table runs out of identity values, you might see an error message similar to the one below:

Msg 8115, Level 16, State 1, Line 7
Arithmetic overflow error converting IDENTITY to data type int.

DBA Dash now has monitoring for identity values. You can see when you are about to run out of identity values which allows you time to plan and fix these problems before they cause errors in your database.

Identity Columns on DBA Dash Summary page

Identity Columns drilldown

More information here.

Backup Compression Algorithms for SQL 2022

SQL 2022 has the option to use Intel QAT backup compression. In software mode this is supported on standard and enterprise editions. It’s significantly faster, produces higher levels of compression and uses less resources. Hardware mode is just supported on enterprise edition and can significantly offload the CPU cost of compressing your backups.

Glenn Berry has some good articles on this here and here.

So what’s this got to do with DBA Dash? DBA Dash now captures the compression algorithm used from msdb.dbo.backupset for your latest backups. You can now where QAT_DEFLATE compression is in use.

New columns added to capture compression algorithm used for last backup

DBA Dash also tracks sys.configurations settings in use across all your SQL instances, including the new configuration options used to enable QAT_DEFLATE compression. You can see the configuration setting in use and also track when settings have been changed.

'#Tracking changes to sys.configurations

Download

Download Now

DBA Dash - What's new in 2.21.0? 🚀

Tue, 09 Aug 2022 21:16:25 +0000

Intro

The first public release of DBA Dash was in January 2022 and version 2.21.0 is the 24th release.🎉 A lot has changed with many new features added as well as bug fixes. I haven’t really blogged much about the changes, but I thought it might be good to capture what its new for significant releases. So here is what is new and cool in 2.21.0!

Note: You can see the full change log here.

Metrics tab

The metrics tab was previously a wall of numbers - the Max, Min, Avg, Total & Current value of all the collected performance counters. At the click of a button you could see a chart for any counter over time for the selected period. It was limited to showing 1 chart at a time.

In 2.17.0 it was possible to configure threshold values for your counters - improving on the wall of numbers by highlighting counters that need your attention.

Now in 2.21.0 when you click view to view a chart for a particular counter, it adds a new chart. You can view charts for multiple counters simultaneously. Also, you can add other charts that are not os performance counters - CPU, Blocking, IO, Object Execution and Waits. This provides additional context for your performance counters. You can re-order the charts and remove them at the click of a button.

Once you’ve created a useful dashboard you can save and re-load it at the click of a button. You can save the view for yourself or share it with your team.

See it in action here:

DBA Dash - Metrics tab

Tip: You can easily configure which performance counters are collected. Also, you can collect any metric you like that can be queried with T-SQL. See here for details on how to customize the collection.

Performance Summary tab

The performance summary tab now has saved views the same as the metrics tab. You could save the layout previously but it wasn’t possible to save multiple versions or to share those with the rest of the team.