Videos of serverrooms and progress report

So I havent posted about our Datacenter move in a while.

Well we are starting to pick up the pace now, all the new servers at the new location have been set up, and various other teams are now installing their stuff on there.

We are now getting into decomissioning more and more servers from the old server rooms. Here are two videos of rooms as they were 2 weeks ago.

 


Old Datacenter 1 from Robert Kloosterhuis on Vimeo.
(I focus on one of the comm pc’s for a while, as the mouse was moving just a minute or so before I filmed this, someone was working on the box remotely, and I was hoping to capture it )


Old Datacenter 2 from Robert Kloosterhuis on Vimeo.

The pics below show some of the servers, or lack thereof (gaps), as we are starting to take the first ones out

IMG_3431

IMG_3430

IMG_3429

IMG_3428
Ready to be disposed of.

IMG_3437
I had Mustafa looking at ways to make some ghost-based images of some of the servers, just in case something happened during the move. He has a version of Bart’s modular Boot disk running there, allowing very easy to configure dos-based network booting.

In clearing out the office, I also found some interesting items 😉

IMG_3438

IMG_3436
This nicely complements my collection 😉
All I miss now is the NT4 and 2008 resource kits!

IMG_3439

 


Friendfeed Sysadmin Room, Twhirl FF support, and Pretend-Sysadmins

So not long after I complained that there where so few System Administerors on the social media scene, Friendfeed introduces “Rooms” and immediately Adnan takes the initiative and makes a Sysadmin Room.

The uptake was pretty damn fast, probably partly due to Adnans blog being part of the Planetsysadmin collective. (why am I not on there yet?! )

So, lets hope admins posts “adminy” and interesting there. Adnan and me are off to a good start, at least.

Meanwhile, I realised I now had a treasure trove of Sysadmins to add to Twitter, which has been a really succesfull strategy so far.

I was immediately struck my a number of things though. Why is it, that I wanted to follow these guys on Twitter, and not just solely on Friendfeed, where they already where?

Well the awnser to that is very very simple: Twhirl. Or more to the point, their completely crappy implementation of Friendfreed support, at least at the moment.

I am reffering mostly to the lack of any kind of filtering, the fact that FF and Twitter are still two seperate streams, double Twitter posts, and the lack of FF comment collapsing/expanding.

But to get back to my original action, the adding of the Sysadmins, as it turns out, many people that added themselves to the Sysadmin Room, turn out not to be Sysadmins at all. Rather any are developers, or web-entrepeneurs. At least that is how they discribe themselves on Twitter. I have filtered who I add to Twitter accordingly 😉

 

 

 

 

 


Scheduled reboot batch job, unexpected “access denied” and how to handle security

So here is something silly I was running up against. In the end its super simple, but its not obvious, and not easy to google for.

I want to equip the new servers are are installing with a standard weekly reboot schedule.

I created a batch file that launched shutdown.exe with some fancy parameters, and set this up as a scheduled task for each server.
I created a special domain account called sa-scheduledreboot with normal user rights, and rights to access the share, and of course the famous “log on as a batch job” privilege, granted to each server via Group Policy.

But dispite this, rather textbook, rights scenario,  I was continuously getting “Could not Start”

However, if I ran the command using Runas, using the credentials of the sa-scheduledreboot account, it would work fine.

The Scheduled Task eventlog showed the following:

“Task Scheduler Service”
5.2.3790.3959 (srv03_sp2_rtm.070216-1710)
“Sheduled Reboot.job” (Reboot.cmd) 5/13/2008 5:43:54 PM ** ERROR **
    Unable to start task.
    The specific error is:
    0x80070005: Access is denied.
    Try using the Task page Browse button to locate the application.

I spent all several hours trying to find out where the “access denied” came from. Eventually, I stumbled apon this:

http://support.microsoft.com/kb/867466/en-us

as it turns out:

In Windows Server 2003, the Users group does not have Read and Execute permissions to the command processor (Cmd.exe). By default, the Cmd.exe program has the following permissions settings:
•    The Interactive implicit group and the Service implicit group have Read and Execute permissions.

Note On a member server, the TelnetClients group also has Read and Execute permissions. On a domain controller, the Batch implicit group also has Read and Execute permissions.
•    The Administrators group and the System implicit group have Full Control permissions.

One of those quirky things you just have to know.

The way I have solved this, is that I have created a special Domain Local security group called RG_command_processor_execute  (RG stands for Resource Group)

This group will allow me to control this specific privilege, and assign it to accounts, usually service accounts, that require the access to cmd.exe to run batch files.

I have added sa-scheduledreboot to this group.

I dont want to mess around on each individual server, so I have made it standard that -all- security settings, including changes to default ACL’s, should happen via Group Policy.

For this we use the File System section of the Security Settings part of a Group Policy Object.
We can add files and folders here, and define how their ACL should look.

The tricky bit is that you have to remember that this Group Policy setting overrides and replaces the original ACL on the object.

Thats a bit annoying, cause it means I have to replicate its current ACL’s, including any special permissions assigned to implicit security groups. 

The KB article shows two ways to do this.
The first is to add the account or group directly to cmd.exe. ACL
the second is to add the BATCH group to the cmd.exe ACL

The second option is interesting, because the BATCH built-in group implicitly includes all batch files that run on the system.

The way that would go would be:

sa-scheduledreboot –>member of–> RG_command_processor_execute –>member of–> %hostname%/BATCH –>applied to–> (ACL of) cmd.exe

This looked like a good option for a while, until I realized it was perhaps a bit broad. (all batch files, including those run by rogue processes? )

And since it only applies to batch files, if I ever needed to grant anything other than a batch file (say, a resident program or agent), that right, I would have to assign the group directly anyway.

So I decided to add the group directly to the resource, which also makes it easier to see what the ACL change is for, for anyone examining the GPO.

sa-scheduledreboot –>member of–> RG_command_processor_execute –>applied to–> (ACL of) cmd.exe

The scheduled reboot command works fine now. And I am confident I did not assign any more rights that I absolutely needed to to get it to work. (In contrast, the previous reboot account had domain admin rights).

The only thing I need to do now, is to remove many other rights from the sa-scheduledreboot service account.
Its currently a member of Domain Users, and that grants a load of rights this account certainly does not need. I will look more closely into that at a later time, as my solution will have to cover many service accounts, not just this one.

By giving out the exact rights needed in a very granular way for each service account I need, I can far more easily restrict ALL service accounts in other ways, all at once, making them useless to use for any other purpose than what they where intended for.

Documenting this is gong to be a challenge.

I need to document exactly what I am doing in the GPO that assigns the rights to these servers, and why each option was chosen the way it was.

I need to document the exact rights of the sa-scheduledreboot

And if I develop a blanket method to restrict ALL servuice accounts in other, general ways, I need to document that too!

I better get to it!


Loic le Meur responds to my post

I pinged Loic le Meur, creater of Seesmic and owner of Twhirl on Twitter to draw his attention to my post on some of the issues i have with Seesmic.

He responded as follows:

“you have great points and we are working exactly in the spirit you expect”

I will take this to mean that we can expect groups/filter function in the Twhirl client or the Seesmic service in the near future. I cant wait 🙂