Runaway Script Killer

[Steven]

Hello Guys and Gals,
We have a question for all of you to answer, and your feedback is important to us. As many of you know, we really do our best to ensure that all of our servers are running at optimal levels, and site performance is always fast as can be. We have come accross some run away scripts here are there, like most hosting providers have. But rather than stay passive, we want to know how you all feel in regards to a script running on the server that kills runaway scripts. This script seems like it would really do us a good service because we can set intervals of how long we want scripts to run before they are auto terminated, we can put some scripts on a non-terminating section, so we have flexibility with it. We just want to guage interest, in how many of you would like to see us implement something like this?

Thanks
-Steven

[SmackDaddy]

What sort of scripts are identified by the "Script Killahs"? (Which I presume are elite Ninja service script killers which takes no prisoners?)

:hammer

[Aaron]

Smackers, I have to say I like your old avatar better. What is that one, a cereal box you found at Goldy's Adult Superstore? :guess

[pete3005]

Steven,

Sounds like a great idea, runaway scripts are the usual source of most server load spikes.

The reason why I am here is because you don't sit passively like most hosts and you do care that load isn't sky high, much appreciated.

How will your script killer handle cron jobs for database back-ups?

I really see this as a good thing as long as it doesn't interfere with innocent scripts. If any script of mine is 'running away' I would be happy to have it terminated. However will there be a log so you can notify people when their script is a 'server eater'?

I appreciate your stance on keeping a lid on server load, to that end I think your idea is well worth trying and I fully endorse it.

Pete

[SmackDaddy]

Originally posted by Aaron
Smackers, I have to say I like your old avatar better. What is that one, a cereal box you found at Goldy's Adult Superstore? :guess

Well....I changed it since my nipples aren't sweaty anymore....hope you like the new me! :D :guess

[Steven]

Pete,
I believe that there are certain scripts, cronjobs, etc that we can setup as being "overlooked" by the script killer, thus allowing them to run.

-Steven

[SmackDaddy]

Originally posted by Steven
Pete,
I believe that there are certain scripts, cronjobs, etc that we can setup as being "overlooked" by the script killer, thus allowing them to run.

-Steven

So what would be identified? Any of the php chat scripts included with cpanel? Or just Cron jobs, and custom scripts?

[TheComputerGuy]

I think this is a great idea. Man when CPU usage reaches 67.8 on a server...and the server is not responding you know your screwed...I been there...this is a great idea!

[mort]

Good idea, although I would agree that some type of notification that the script was terminated would be benificial.

If this is implemented, then I'd appreciate knowing the criteria under which the killer would kick in, ie, if cpu usage is greater than... etc. Just so a can keep my customers informed.

[SmackDaddy]

Originally posted by SmackDaddy
So what would be identified? Any of the php chat scripts included with cpanel? Or just Cron jobs, and custom scripts?

I hope Steven doesn't have me on ignore since I joke alot....but.....this'll be the third time I have asked this question....(see above quote)....

[Steven]

Ok here is the deal, we have been testing the scripts on the servers without it killing anything for almost 2 days. We feel fairly confident that this script will be a great addition to the arsenal of our monitoring servers.

Mick,
Everytime a script would get terminated, we get an email. We in turn will notify our client of this. As far as the settings, we are still tweaking them, but it should be something like 80% CPU and or 80% memory for a specified amount of time.

Smackers,
Sorry I missed your posts, the script auto kills runaways or other scripts utilizing way more than their allocated resources. Granted we don't have this issue often, but in the event that we do, we now have a viable solution. The script allows us to exclude certain scripts, cpanel functions which we have done. Cron jobs don't take a long time to execute, and the php chats, really aren't intensive. This script is really geared towards runaway scripts and server intensive scripts using 80% CPU or 80% memory for a specified amount of time.

We are sending out an email later tonight detailing some of our new ideas, new implementations, etc.

-Steven

[harmonic]

Yes, this would definanltly be awesome.

On my old host, I got real PO'd because my site was going slow for days... After I sent an email, the host looked into it and found someone using a poorly coded perl script that was taking up as much resources as possible... They fixed it, but my site was slow for days.

Plus, everything I use is PHP and takes less than a second to execute ;)

[barrywien]

Hi
Just wondering if this email went out or not.

I have not had an email from IH for about...well since I joined to be honest. Is there any way I can check you have the correct email address on record for me?

[Steven]

Hi Barry,
That is very odd, we must not have the correct email info for you on file. We send out newsletters at least once or twice a week. I will send you an email shortly with the email we have on file for your account.

-Steven

[Steven]

Ok, here is the story with the script killer. It wasn't working as we had hoped so we contracted the developer to customize the script for our use. It is now complete. We can setup the script to kill based on CPU usage and time, Memory usage and time, and now just by time alone. We anticipate that things will continue to run as smoothly as ever.

-Steven

[psyco]

About this script killer, How can I set a script to be ignored by it?

Paul Sykes

www.fnf-web.com

[mort]

Paul
Have you had a scripted killed? If so according to steven's earlier post, you should have received an email, which I would expect to have the correct procedure for contact.

If you are antisipating that a script will be killed, the best thing to do, is drop in a helpdesk ticket, giving details of the script, expected running time, etc. If insiderhosting consider that the script won't be detremental to other's on the server, they will add it to the ignore list.

[Tal]

Originally posted by mort
Paul
Have you had a scripted killed? If so according to steven's earlier post, you should have received an email, which I would expect to have the correct procedure for contact.

If you are antisipating that a script will be killed, the best thing to do, is drop in a helpdesk ticket, giving details of the script, expected running time, etc. If insiderhosting consider that the script won't be detremental to other's on the server, they will add it to the ignore list.

I could not have said it better myself

[psyco]

No, I am planning ahead. as my customers are going to be coming to me with issues along these lines, and I currently have one customer running a series of scripts, knowing this information would be useful.

also being able to set a script to be ignored (for example by setting its file name to something) would be a useful idea.

so maybe a .cgi script comes under the normal rules (ie it can be killed) but once it is debugged, and it may need to run for extended periods, switching it to .pl would cause the script killer software to ignore it.

hence the question.

Psyco

[Sardtok]

I'm not sure if that's such a good idea, considering how fast support is here I think we might as well make an application for it, or simply e-mail support if we need a script to be ignored by the script killer.
Script kiling is a great idea.
I was with a host once and I had a runaway script (at least they claimed it did and I had been running it for almost a month with no problems or anything, I've also seen other people's run-away scripts ruin the execution of "good" scripts), I didn't get a warning mail or anything, they just shut the entire site down (couldn't they just have removed the script or disabled it or what ever), and they didn't tell me what had happened or which script it was, they just removed all files from my dir and put a text file saying something along the lines of: "Your script was buggy weeeeeeh."
Ok so it was a free host, but would have been nice if they hadn't removed my entire site, or if they had contacted me (not much I could've done anyway as I was on vacation).
Anyway,
script killer, YAY, at least if we get notifications about buggy scripts.

Currently I'm running a very small set of simple php include scripts + phpBB so shouldn't be any big problems with what I've currently got.

[VxJasonxV]

Keep in mind everyone.
These scrips have to TAX the server's processing abilities for a period of time.

All php routines on phpBB run for less than 2-3 seconds. And they do a lot of page generation. Especially with some of the themes we use :).

You'd have to have screwed up a script (that runs forever) for the script to be killed. Or doing super-computer calculations :D.

(Not really...but it's not a common thing for a script to run over script killing parameters. Not common at ALL. Am I right?)

[Sardtok]

Well,
if a script gets caught by the script killer, there is most likely a bug with a loop somewhere (locking up in a loop is no fun).
Or there could be some kind of security loophole that lets a user of the webpage screw up the script somehow, eternal looping again for instance.
Also if the script invokes an external (by that I mean local) command, and waits for return codes or what ever it is possible for it to screw up.

There are usually more ways to screw up a script/program than one would think there is.

Oh, almost forgot.
I don't think it's common for scripts to run over script killing parameters, no.
It actually shouldn't be able to unless there are any security issues with the script killing program, which I suspect you need root access to deal with anyway???

[VxJasonxV]

Thank you for repeating what I just said :D.

[Sardtok]

Actually it was meant as an answer.
You say: Am I right?
And I say:
No I don't think it is common.
The no in the beginning and end of the sentence means it's an answer to a question.
w00t,
I don't know why I said that though,
w00t w00t.
AAAAH
Everquest is burning it's way from the ancient memories part of my brain to all other parts.
w00t w00t w00t w00t w00t AAAAH ;)

Well, I still don't know why I started saying w00t. ;)

[TheVoidz]

Its a good idea to have a script killer, just make sure all imporant scripts prolly all owned by root are in a protected part. I remeber PHW not doing this, and it was killing the site back ups. So the back up never finished so the crom job started it again and so on... got messy.

[nathanpalmer]

Maybe it's just me.. but I don't think you should be able to specify a script as *excluded* from the script killer. I don't think any script is important enough to take all a servers resources..

If the script needs to be run, then the site should be put onto a dedicated server or the owner should take a close look at what the script is doing so they can optimize it.

[pete3005]

I agree Nathan, good post :)

[Simon]

What about the PHP ini settings max_execution_time and memory_limit - surely these would deal with most runaway PHP scripts. I guess the author could change the values for individual scripts though and still cause problems.

< Simon >

[VxJasonxV]

It's not just php scripts.

[Simon]

Yeah, I understand that - just hadn't seen it mentioned.

Does Perl have any way of limiting execution time or memory usage?! (I'm a PHP/ASP geek)

< Simon >

[Simon]

--- [ Alert: 6-month-old thread ] ---

Steven,

I haven't heard much about this lately - did the script killer work out? How has it been going for the last few months - has it been effective?

< Simon >