Contents
It is past midnight, a table is locked in production and complaints are piling up at the service desk. The fix is one line of SQL. The normal approval flow asks for two people, and both are asleep. At this point it looks like there are two options: hold the process, or skip it. In most organisations the second is chosen, and a ticket is opened retroactively the next morning.
The real problem is not the work done that night. The problem is that the organisation has no pre defined path for an emergency. When there is no path, everyone opens their own, and that one leaves no record.
The Decision at Midnight
A retroactive ticket has a cost, and that cost shows up months later rather than during the incident. At audit time the questions asked about that change are: when was the change made, when was the ticket opened, why is there a gap between them, who was authorised at that moment, and was the work genuinely urgent?
None of these questions are technical. They are all governance questions, and their answers lie not in the incident itself but in the preparation done before it.
Emergency Change Is Not an Approval Bypass
The first model that comes to mind for emergency change is "turn the controls off". That is not the right model. The ITIL definition of an emergency change does not remove control either; it describes assessment and authorisation carried out in an accelerated form. Information security has similar reasoning: emergency access is a path that is recorded and reviewed afterwards.
The right mental model sits between the two:
Fast, but controlled. The emergency path is shorter than the normal one. It is not unrecorded.
To make the difference concrete:
| Step | Normal change | Emergency change | Uncontrolled intervention |
|---|---|---|---|
| Request record | Yes | Yes, justification required | Later, by hand |
| Script validation | Yes | Yes, the same rules | No |
| Approval step | The full policy | Shortened, at least one locked step | None |
| Audit record | Yes | Yes, and marked as an emergency | Only in the database log |
Designing the Emergency Path
There are four questions to answer when designing the emergency path. All four should be answered before the incident; a rule debated during an incident is not a rule.
1. Is the emergency path a separate policy?
It should be. Editing the normal policy during an incident means changing a control without control. The emergency path is defined in advance as its own policy, with the servers and request types it applies to written down, and during the incident it is only selected.
2. How many approvals remain?
It should not be zero. The point of the emergency path is to reduce four approvals to one, not to reduce one to zero. The remaining step should be marked as non skippable, otherwise the application will pass it too when conditions look right, and you are left with an unapproved production change.
Who closes that single remaining step also needs thought. In organisations with an on call rota the step is tied to the on call role. Without a rota the emergency path does not work in practice and the team bypasses the process anyway.
3. Do validation and risk assessment remain?
They should, because they cost nothing. Parsing the script and running the rule checks takes seconds; the time lost in an emergency is waiting for approval, not validation. In a fix written at midnight the chance of an UPDATE without a filter is higher than on a normal day. That is exactly the moment the control is needed most.
4. Are a justification and an incident reference mandatory?
They should be. Whoever declares the emergency should write why it is urgent and which incident it belongs to. This field does two jobs: it makes the post incident review possible, and it stops the emergency path from turning into a shortcut. Even a free text field is a deterrent.
Who Can Declare an Emergency?
This is the most skipped question in emergency change design. Declaring an emergency is not an operation, it is an authority. If anyone can call something urgent, the emergency path becomes the shortcut, and within a few months the normal path is abandoned.
The practical approach is to tie the declaration to a role level. A requester may see an urgent need, but the authority to declare an emergency sits with a team lead or the on call manager. That threshold is tuned to the size of the organisation: lower in small teams, higher in large ones.
How SQL Change Guard applies this model. Emergency policies are defined separately from normal ones and are matched by server, environment and request type. The emergency option only appears when two conditions hold together: a matching emergency policy must exist, and that policy must contain at least one locked approval step. A locked step is never skipped. The authority to declare an emergency is tied to a role threshold managed by a parameter. When the request is saved, a justification and an incident ticket are asked for; validation, risk assessment and audit recording work in the emergency path exactly as they do in the normal one.
What Has to Happen Afterwards
Whether the emergency path counts as controlled depends on a review taking place afterwards. That review does not have to be long. It has to be regular.
- Was the change genuinely urgent, or could it have been planned?
- Who declared the emergency, and were they entitled to?
- Was the remaining approval step actually closed, and by whom?
- Is the executed script still on record, and is it the same as the approved content?
- If a temporary fix was applied, was a request opened through the normal path for the permanent one?
The fourth item is usually the hardest, because scripts get edited quickly during an emergency night. Recording a content digest of the script at execution time makes that question answerable later; we covered it in was the SQL that ran the SQL that was approved.
Measuring the Health of the Emergency Path
A well designed emergency path is measurable. Three numbers are enough, and all three come out of the organisation's own records.
- The share of emergency changes in the total. If this ratio is rising, the problem is not in the emergencies but in the slowness of the normal path. The fix is to speed up the normal path, not to restrict the emergency one.
- Repeated emergency changes on the same system. If one system needs urgent intervention every month there is a structural problem there, and the emergency path is hiding it.
- The number of emergency changes never reviewed. Anything other than zero here weakens the claim that the emergency path is controlled.
Frequently Asked Questions
Are emergency change and break glass access the same thing?
They are not. Break glass is the temporary opening of a privilege nobody normally holds, and it belongs to access management. An emergency change is the shortening of the approval path for a piece of work, and it belongs to change management. Both can exist in one organisation: work done in a session opened through emergency access should still be tied to an emergency change request.
Can approval be removed entirely for an emergency change?
Technically it can, but then what you have is not an emergency change, it is a recorded uncontrolled intervention. From an audit perspective the difference between a single approval and no approval is large. Keeping at least one locked step is recommended.
How do we stop the emergency path from being abused?
Three levers work: tie the declaration to a role threshold, make the justification and incident ticket fields mandatory, and report the share of emergency changes regularly. Making it visible works better than forbidding it.
Is there time to prepare a rollback plan for an emergency change?
Generating a rollback plan is faster than writing one by hand, but not every change can be fully undone. For statements that change data, going back usually depends on a backup and a time window. On an emergency night the most realistic approach is to know the scope of the rollback plan in advance and to accept what cannot be undone. We went into this in SQL rollback strategy.
Build the emergency path with your own policy
Bring your current approval flow and we will design together how the emergency path is shortened, which step stays locked and how the record is kept.
Book a Demo →