Slaying the SCOM Auditability Dragon
In
the library of boring topics, software configuration auditing must surely rank
close to the top. While end user
applications such as messaging and database servers spark swarms of such
products, SCOM and its System Center siblings have yet to attract major
interest. The reasons for this are open
to debate, but there can be no doubt as to the fact that Microsoft’s
infrastructure products are rife with configuration interfaces and rich APIs
that make it all too easy for configuration drift to rear its ugly head. A
typical SCOM deployment will have over 10,000 rules and another 10,000 monitors. In fact, the job boards over the recent past have
been thick with openings for SCOM engineers (a search for “SCOM” on Dice
returns 253 postings), many of which are to reign in existing deployments
sinking under their own clumsy weight.
So
how much should you worry about “what changed” in your SCOM deployment? Well, the truth is that, in the absence of
tight change controls, SCOM deployments evolve like any other complex system in
your data center: instead of being a
ready tool in helping detect and solve problems, it becomes part of the noise
overwhelming administrators. It is
another example that even the monitor needs monitoring, and IT managers pay a
price for ignoring that reality.
Enter
The Dude
These
questions of SCOM configuration controls were the subject of a recent conversation
I had with a very capable IT engineer, whom I enjoy calling the “Dude” on
account of his unshakable confidence and periodic exhibits of great flair. The specific point of our debate was the use of
naming conventions for SCOM authoring. (Don’t say you weren’t forewarned some
dull topics were afoot!) Dude had
inherited a SCOM deployment that was perfectly devoid of documentation,
courtesy of his predecessors, all reputedly SCOM experts. Dude had earned his SCOM spurs in this school
of hard knocks: deciphering the meaning
of each SCOM alert, and investigating how to ensure SCOM alerted when serious problems.
From
his first tentative overrides to his eventual mastery of the Authoring pane, he
carried on his company’s tradition of documenting “nada”. And “nada” means no run guide, no change log,
heck not even a lazy description sprinkled here and there. His disregard for documentation in any form was
an article of simple faith: why waste
time on documentation when you will always remember the changes you made, and
if somehow your memory fails you, then it’s no problem to figure it out when
the need arises. Such self-assurance was
so charming – ah, how good it is to work in IT when you’re outside the scope of
any quality controls, audits or best practices reviews.
Some
more context would be helpful. Dude was
responsible for only one SCOM management, so the question of code consistency across
more than one system never crossed his mind.
Further, since Dude was the only SCOM administrator, it was actually
possible, assuming a heroic memory, that he might remember all his overrides
and customizations. Lastly, since it was
not a customer-facing application, Dude’s managers accepted the frequent
problems with SCOM as inevitable annoyances.
They never challenged Dude to set any quality goals or continuous
improvement plan. It was ‘the best of
times, it was the worst of times.
Under
the Covers
Microsoft’s
Management Pack Authoring Guide for SCOM gives a clear explanation why the
key attribute of every element in a MP is the ID field. As with SCSM, SCOM constructs a class
hierarchy ordering all the elements in every imported MP, and maintaining this
in memory is one of the key roles of the RMS.
When your primary tool for creating new rules and monitors is the Ops
console, SCOM conceals the ID field, automatically constructing one on the fly from
the element’s type and a GUID-like string of numbers. All the author controls in the console are
the Display Name and the Description. Nice
and easy, but not the best design for configuration auditing.
The
uniqueness function provided by a SCOM element’s ID is similar to the function
of a hostname in a DNS namespace, in that the hostname must be unique within a
DNS zone. Further, the ID of the MP anchors
the namespace in SCOM as the name of the zone does in DNS. The display name, on the other hand, is like
the comment field in that, just as you can give many hosts in a zone the same
comment (or no comment at all), you can give the same display name to any
number of SCOM elements – think “VIP Application Service Down Monitor.”
Restoring
Order
The
Authoring Console is actually the intended tool for extending SCOM with custom
classes in enterprise deployments. The
Authoring Guide recommends that you standardize your IDs just as Microsoft does
in its own MPs. And how does that
work? Well, as I explained to Dude, the
basic process is:
1. Create a concisely
named MP, using the model evident in all the SCOM system MPs and most of
Microsoft’s application MPs
2. create the element
in the Ops console as usual and host it in the well named MP
3. export the MP to an
XML file
4. open the file in an
editor and locate the console-generated ID
5. replace all
occurrences of that ID with a new ID comprised of the following parts, each
delimited by a period
a.
the
ID of the MP
b.
the
element type (such as “Group”, “Rule”, or “Monitor”)
c.
one
or more descriptors that indicate the essence of the element (such as AppLog.Error1102
for a rule that alerts when an Error event occurs in the Application log with
an ID of 1102)
d.
Re-import
the MP
And
what have you accomplished? Your new IDs
now reflect a path in a clean amespace from a root or branch to a leaf node in
terms that are easy to follow. These IDs
will reinforce the design integrity of the deployment and protect the relevance
of the documentation. When you export
your MPs to a common area, you can search across multiple files for elements
targeting the same class or object, which can answer many questions such as
identifying redundancies. If you ever
run the Alert report, your IDs will no longer read like the outputs from a
runaway random generator.
Y.A.M.
-- Young Admin Myopia – and the Upside of Planning
At
this point, Dude was rolling his eyes and groaning in disbelief. “Why all this fuss and bother if it produces
no visible benefit in the console?” He
might well have added, “And why spend my good time assisting future
administrators if it’s not in the job description?” Well, a good retort might follow the logic Robert
Duvall gave Sean Penn in “Colors” on the question of running down a hill vs.
walking down, but your mileage may vary.
Well,
returning to the question we started with, if you ever hope to implement change
control audits for your SCOM configuration, especially if you have multiple
administrators who make frequent changes, it’s simply unrealistic to try to do it
with manual tools such as screenshots and spreadsheets. There are just too many elements to track without
an automated tool.
In
the meanwhile, hopefully this article has persuaded you to take the extra
effort to convert your GUID-like IDs to a more meaningful namespace style. Here are some explicit examples of the two
styles we’ve been discussing.
Class
ID:
Before:
UINameSpace172861e718614744a224992d5237de31.Group
After:
MyCompany.Messaging.Monitoring.MyAppServers.Group
Rule
ID:
Before: MomUIGeneratedRule3d232ca92a3a4e9e9c53e70c6838439b
After: MyCompany.Messaging.Monitoring.Alert.AppLog.Error1102
Rule
Property Override ID:
Before: OverrideForRuleMomUIGeneratedRule083aa6fe88f34aedb0c871e3da8843a1ForContextUINameSpace59a7638242334435824fcf4ebbf3450bGroup75fba1242f0d4f038b8590e169a123d0
After: MyCompany.Messaging.Monitoring.Alert.AppLog.Error1102.Override.Interval.MyAppServers
SCOM
has been around for a while, but good practices are never too late to start. I hope this article has shown you that these
default IDs are not beyond your control, and they’re worth controlling.
