Welcome Back!

Welcome back sorry for the problems but most should be fixed.... well it should be. Sorry if you were
here earlier and posting didn't work, I jumped the gun and didn't fully test it.

This is the new static IP for nepadigital , the only thing I need to do now is wait for the new DNS entry to
propagate so nepadigtal.com correctly resolves to this IP. Once that is complete I'll move this out of
the temporary directory back the neapdigital.com/bb .. Once that's done its completely back to normal.
DNS usually takes about 1 to 3 days and i have no control over it.

This forum is completely functional as far I know but I have not checked for every possible issue,
if you come across anything let me know. Specifically I need the exact error text you encounter and what
you were doing when the error occurred.

So onto the story..This all started about the last two weeks in October, I was checking over my stats
and noticed a huge spike in the CPU. I wasn't keeping up on it because historically for the amount of traffic
I was generating it wasn't an issue. I was easily going to go over the limit but the numbers didn't add up
according to previous stats. For example:

In Feb. of 2007 I served 167,743 pages, the cpu usage was 43.85% .

In June of 2007 I served 77,962 pages and used 27.60% of my allotted CPU.

In July I served 84,482 pages only 10,000 more than in June but the CPU balloons to 55.40%.

In September 2007 I served 119,871 pages nearly 50,000 less than in In Feruary yet the CPU
usage is now at 105.67% more than double for February.

Obviously there is something wrong here, so I started going over my scripts to see what could
be the problem. The trouble is I didn't really add or change anything in those months so I didn't
know where to start. Removing some of the features on the forum didn't help at all. The biggest
issue is I didn't have the tools to determine what the problem is. I have one number and that's
the total percentage and it was only updated every 24 hours. So I'd make some changes and
wait. Add that to the fact traffic is increasing daily and I have some trouble on my hands.

At the end of October the CPU usage has reached 250% my allotted amount and I get this e-mail:

Dear Customer,

Over the past month, we have been monitoring the resource usage of your Web site on our Shared hosting server. We would like to think that, with the amount of resources your site is consuming, your business is doing very well these days. Unfortunately, the plan you are currently using no longer meets your site's requirements. We would like to invite you to review the current plans we offer, available at http://hosting.aplus.net/, and have you choose a more appropriate one for your site to avoid a disruption in service.

For your convenience, we have included a brief summary of your current plan as well as the suggested plan that we think will better suit your needs:

Server : pro10.abac.com
Registration # : XXXXXXXXXX
Current Plan : Solo XR
Username : XXXXXXXXXX
Overuse : 38min CPU time daily
Suggested Plan : ePro

So now I got a week to fix it, I have to give them credit for giving me some leeway.

LOL 16 minutes is my allotment, this BTW I've found out is ridiculously low by any standards. So I call up tech support
and explain the situation with the disparity in the stats. Absolutely no help what-so-ever, just some guy
sitting in front of knowledge base that has less knowledge than me giving me canned answers. I've
been a customer with this company for 5 years and have never had to deal with their
"tech support". There suggestion was I need to upgrade to a $39 a month plan (which BTW is outrageous for
what they are offering).

So in the early morning hours of Nov. 4th I do the last thing I can think of. I install phpbb3. If it's a problem
with the scripts this will undoubtedly fix it because there is no modifications. However I get hit by the
"perfect storm". It didn't occur to me that people would be clicking all over the place. Add to that Google
has a index update (see the first post for Nov 4)and now decides it wants
to be my best friend and just about doubles the referrals, between the two page views double . I'm
now serving 25,000+ pages a day and the meter hits 950% for the days allotment on Nov.5 .

Here's some stats, note that thes are just web stats and don't include CPU.


Even the highest plan they have will not cover this besides the $99 a month dedicated server, to give you
an idea a dedicated server could handle a thousand people online at once. So now I'm getting desperate
and I contact tech support again and they suggest I contact the Security Assurance Department because
they are the ones that handle over usage.

So I fire off a letter to the "Security Assurance Department" representative:

Server : pro10.abac.com
Registration # : xxxxxxxxx
Current Plan : Solo XR
Username : xxxxxxxxx
Overuse : 38min CPU time daily

I've been attempting to correct this myself but cannot get any answers from
your support staff so I'll explain it one more time. First I'm aware the
current plan is not going to meet my needs. However the CPU usage is
astronomical and doesn't make sense. I'll give you a time line of events and
you can confirm this yourself if you wish.

In Feb of 2007 I served 167,743 pages, the cpu usage was 43.85% .

In June of 2007 I served 77,962 and used 27.60% of my allotted CPU.

In July I switched to PHP5 in the hosting control panel, served 84,482
pages only 10,000 more than in June but the CPU balloons to 55.40%. This I
think is important and where things go wrong. I've since returned it PHP4
and that hasn't helped.

In September 2007 I served 119,871 pages nearly 50,000 less than in In
Feruary yet the CPU usage is now at 105.67%.

I must emphasize that there was no major changes that I can remember to the
scripts between Feb. and Sept. that I'm running. The only major change was
switching to PHP5 . Just to add I've been trying to fix this myself long
before this notice came, by removing features on my forum but all to no
avail. Historically the SQL queries were always a higher percentage than the
CPU right up until I switched to php5.

I've requested information more than once as to exactly what is using all
this CPU because I don't have shell access to get it myself. Without knowing
what the problem is I can't solve it.

As I said I'm willing to upgrade and it appears that I have too however I do
not simply throw money at problems. I attempt to solve them and in this case
it doesn't look like throwing money at it would even help. Something is
definitely wrong and I think it's wrong on your part considering all the
above information. I've been a customer for well over 5 years, if I cannot
get any assistance in solving this I'll be switching hosts.

And his response:

You can monitor your resource usage from your monitor tool in your
control panel. I've notice that you are exceeding your CPU time usage
and you mentioned that you haven't changed anything recently.The CPU
usage can't be explained only with a script issue it may be caused by
blogs, galleries etc. In this case will be wise to implement robots.txt
file in your hosting and by doing so prevent search engines crawlers to
go thru your pictures.If you have a lot of pictures or big pictures,
this could be the main reason for your CPU time over usage.Unfortunately
we can not go over and revise your page and how it is setup because this
will involve checking on the coding, the structure of the site and other
add ons such as forums, blogs or forms.This whole process will take time
and resources and unfortunately revising of the code is considered as
paid support.

Regards,
Aplus.Net Security Assurance Department

Now I'm convinced they are all idiots, robots.txt is in use and I know exactly how many pages
Google crawls on this site, 500-1000 a day, it's a lot but under normal circumstance only about 10%.
Other bots don't add to much


This isn't even a canned message because pictures don't eat CPU they eat bandwidth. The only way
images could eat CPU is if you're resizing many of them dynamically all the time. This forum does
that to create thumbs but only once per picture, just to add I resized 1000's of them at once server
side once on another site and the CPU wasn't affected much.

So it's back to tech support and finally I get them to have an admin look at the situation, I figure
someone has to know something there. I specifically request information on the the top scripts broken down
by percentage. I get this shortly after :

Dear customer,

We are writing you regarding TID 200817. Our administrators
explained that the reason for the abuse is in your php
scripts.

Sincerely Yours,
Customer Care Representative

No *censored* sherlock. I though maybe it was a mouse in the server... Believe it or not this gets better

Stay tuned for part II , same bat channel , same bat time.

And now the #%$#%#@ is down again.

Sorry again for the downtime but this is new territory for me, one of the issues is the DNS switch makes things all that much more complicated. Some people may be getting old host while other will be getting new host. You can even get two different hosts from seperate your work computer nad your home computer. Nothing I can do about it but everthing will smooth out in a day or two.

So how long is it a person can go without sleep?

Wow, The things you do for us just amazes me. I wish I was smart enough to help but I know my limits. I hope things get better for you at the new host so you can enjoy this as much as everyone else does. Thanks Coalman!

So back to my story...

So Nov. 6 rolls around, the notice for suspension was sent late afternoon on the last day in October so its
crunch time. Looking over the stats even if I purchase the new plan it's not going to cover it so it's pointless
to pursue it, so I figure I'll upgrade for the month which will give me some time to prepare to switch servers
in interim knowing very well that the new plan wasn't going to cover and I'd be going through this again
at the end of the month.

I call up sales and request the upgrade for 1 month, Jack the sales guy goes can you please hold.....
He comes back on the phone and informs me they can't do that because I pay for it yearly and if I want to upgrade
I need to do it for the entire length of the billing period even if I waived the prepaid balance. In other words I
can get the plan for one month if I'm new customer but since I'm a loyal customer that pays by the year I have no
choice but to upgrade the entire length of the plan. I suppose I could have cancelled and then got new plan but
that would have been pointless because again the higher plan wasn't going to cover it anyway and the whole
idea of dealing with them was to keep the site online without any interuptions.

So now I'm really steamed, so I lay into Jack. I tell him all the bullshit that has been going on and why I only
I only want the plan for month, I forward the above emails to him jack puts me on hold again.... He comes back
and says he'll see what he can do and will call me back and he "assures" me they won't cut the site off.
His exact word words. Jack never calls back.

On Nov.7 I get a billing notice, i check the bill and it's for the full amount of the entire length of the plan. I figure
figure no problem Jack's working on it and I'm still waiting for that phone call. Shortly after site goes down for
the first time because they suspended it for non-payment.

I call Jack and he informs me he sent a email, ohhhhhhh wait a minute it got bounced because the site was suspended
and not accepting any email. So he sent it after they switched the plan. He switches the site back on and resends the email:

The admin said you have a huge number of scripts running and is not possible
to locate which one generates the CPU usage.

I have upgraded your plan to ePro , as we orignally discussed, to avoid
suspending.

You might experiece downtime of several hours.

Jack

First of all I don't have a huge number of scripts running, i have a single forum running. A forum script
I will note that can be installed directly in the control panel. It's even advertised in the literature for the plan.

Is the suspense killing you yet?

Part 3...

I tell Jack I never wanted it for the whole length of the plan... again. He says he noted it to the
billing department but apprently they didn't follow through. He switches the site back on and says
he'll get back to me. I never here from Jack again.

I now have access again so I download the entire site to my computer. I keep frequent backups but
my last full backup of the files was some time ago. If in the event i lost these files at most I would
have lost some image files. Thankfully i have cable, my estimation is that it would take a week to
complete the same thing with dial-up. Most importantly I make a backup of the database, this
contains all the text, posts, members, configurations etc. Without this you have nothing but a
blank forum. So now I'm ready, at least I think I am and I start looking for another hosting company.

In the meantime I'm still getting notices for non-payment but Jack's working on it right? At the
very least hes going to let me know what the situation is. Friday the 8th comes and the site goes
down again. I call ...

They inform me it's because of non-payment, I explain that I only authorized it for one month. Further
I tell them I want it switched back to the original because i never authorized this and under no circumstances
do i want this plan upgraded. I will will remove the scripts and move to another company. To which they reply
it will take 12-24 hours to get it back online which is utter bullshit, they didn't have to do this during the upgarde.
It's simple on off button on the control panel, only reason needed to disable it is if you're moving it from one server
to another and that's only if you want to take the easy way out. Here's the button, I have my own now.



So I figure for this 20 hours I'll start preparing for this move....but...my database backup is incomplete. It got corrupted
some how during the transfer and I only have 90% and the the 10% of it that's missing is crucial I still have backup
that is good from a week ago but that's a week out of date. Now I'm getting worried...

Saturday morning comes around, it's 24 hours and still no site. So I call again, they hit the button for me... Gee thanks.

First thing I do is disable the forum so it's not jacking the CPU through the roof. I get a fresh and complete copy of the
database. I have a local server running here on my computer for testing purposes so I install the forum and restore the database.
I then ran a crawler across it which produced a few thousand static pages, these do not eat CPU just bandwidth , I upload them to
the site. At least it's still viewable. Pretty slick eh? Now I have some time to work with.

In the meantime i start researching new hosts, By Saturday I find what I hope is good host. All signs point to that they are. I'm
now running on a VPS which is like having your own server but you only get a piece of it I can do almost anything with it including
hosting other domains, game servers really anything. Before I had PHP, now I can install PHP The only thing that is metered is the
bandwidth and the disk space which is way above my needs. The best analogy i can give is if you're on a restricted AOL account
and go straight to broadband.

15 GB VPS Disk Space
300 GB Bandwidth
256 MB Guaranteed Ram
1024 MB Burstable Ram
Equal share cpu

Story isn't over yet....

So during the break here from my story if anyone is interested in knowing when the DNS changes,
if you look at the top of http://nepacrossroads.com/ the ending of title desciption has 4 periods.
these 4 periods will not be there if you are reaching the new domain.

I wouldappreciate some input on this so i can get the forum back to /bb.

You may ask why it's just not there now, the reason being I don't want the bots indexing the IP address.
i.e. http://www.nepadigital.com and 69.73.136.20 would be seen as two separate URLS' The correct way to
migrate is to set up the forum up on both sites and sync the databases but that impossible given the
circumstances at this point in time.

Looking at the forum I see 5 people online, most I would usally get is 10, maybe 15 tops.

Here's the processor load average:

Load average
Last 1 minute 0.07
Last 5 minutes 0.04
Last 15 minutes 0.05

...... really nothing more to say. I'll see what it does in the next few days. Quick guess tells me I would nee
a couple hundred people online to break a sweat.

What's the units? Percentage of CPU utilization?

Percentage I'm guessing, there is no perecent sign though. Must be percent because in the hosts forum the guy had to upgrade to dedicated server but his total most ever online users was 1000+ He was running phpbb2, phpbb3 needs less resources. It's optimized for large forums.

I seen 8 now here it is again:

Load average
Last 1 minute 0.09
Last 5 minutes 0.11
Last 15 minutes 0.09

Hmm apparently it's not a percentage, seemed low to me. I should have looked. It's a represenative number

Well, load is an average of the number of processes in the run queue over some length of time. It's affected by CPU utilization, I/O, etc...and can be kind of hard to get a handle on. Discussions regarding system load have a tendancy to turn into arguments icon_smile.gif

For all intents and purposes: a load 1.0 means that processes are having to wait. Why? Well, could be for a number of reasons...to many processes eating up too much CPU, excessive swap usage, etc. .

http://forums.theplanet.com/index.php?showtopic=30691

Even if 1 is the limit before it starts to bog down that's still 70 -80 people online. It's plesk control panel if you want to look, there's nothing in the help file or the documentation in the manual. There's other tools, but i have to install. Haven't gotten that far yet and don't want to break anything yet. Maybe I'll go ask in their forum.

OK getting a handle on this, I have two control panels. This second one gives a graph.


Note that I was clicking all over the place and recreated the search words index to get the graph to move.
The search index is by far the largest table in the database.The first number under the graph is the last
minute, then 5 minutes, then 15 minutes. It appears the percent is based on an average of the
three numbers. With 1 being 100% but my limit is 200%... lol.. that's what it says. The plan calls for
a equal share of the CPU however I'd imagine the 200% is to allow for over usage when others are not using
it. I'm still sharing a server but with lot less people.

Here's one with indexing still going on, this is probably about 2 minutes since
the first image was taken.


Here's the next one a few minutes after the search index has completed..



How's that for some funky math?

that has to be fustrating. you must be a very paitent man. i would of trowen thing out the window. we all appreciate what your doing for us , thank you. i thought i was going to have to find ron's forum.

Thanks Coalman for all your hard work. It's been a difficult trip, all that moving around and squeezing through all those wires, but I think I'm OK. For a while I thought all was lost...Kinda felt like I really had caught a tit in the wringer...

One of my pet peeves is people you turn to for help who either aren't capable of doing their jobs or just don't care. I seem to encounter them all the time. I stuck with my ISP when I switched from dial-up to DSL because of all the help I got from them when the DSL wouldn't work. The problem was with the phone company, at the switching office, they didn't enable DSL on my phone line. My ISP even sent a tech to my house, we tested the line right at the interface and it wouldn't work. The phone co. still insisted in sending someone out to the house even though I told them the problem wasn't here. Then the guy they send out wanted me to get my DSL from the phone company. Never the less, he went to the switching office and corrected the problem.

I wonder...is this a labor of love or will you realize a financial gain from all this work? I would hope you do.

Thanks again.

Wood'nCoal ,thanks would have been a lot easier if i planned better, I think I mentioned that the correct way to do this is to set up two sites like i have now but you sync the databases together. If that was the case it would have been practically seamless. In hindsight I should have rented this new server beforehand. Live and learn.

I wonder...is this a labor of love or will you realize a financial gain from all this work? I would hope you do.

Both... I mentioned in another post I'm expanding this forum to include others and many more features. The idea is to attract more people, the ads are taking care of the server costs at the moment so it isn't costing me anything but time at the moment.