Raw Logs

[SarahJane]

Are raw access logs rolled over and zipped up and if so how often?

[Steven]

Hello SarahJane,
Raw logs are purged every month. You can download them all on the last day of every month if you desire.

-Steven

[SarahJane]

Assuming I need to see my access logs on a daily basis, is there anyway to enable this?

[Sardtok]

You download the log files whenever you want,
not only the last day of the month...
You can do this through your CPanel,
but I don't know if you can purge the files once a day,
I think that that must be setup explicitly by Steven (or someone else at InsiderHosting)... ;)

[SarahJane]

But after a few days the file will get to be very big and I will not be able to open it fully. By the end of the month trying to open it will crash my computer.
There may be ways around this that I don't know about.

[Sardtok]

Well,
it really depends on the program you use to read the file...
For instance Microsoft don't know what they are doing...
Wordpad buffers the entire file, and lags up the comp pretty badly...
Not sure if less is available on Windows...
I'll check, pretty sure more is though, but more is kinda useless, hehe ;)

Doesn't look like less is available...

What are you going to use this for?
(can you pipe something on windows like on linux, for instance the way you'd use grep for only finding relevant information)...

[SarahJane]

I am using Excel to open the files. It can handle about 65000 lines of data which is about 15mb of my present log files.

Other options I don't really know much about.

Thanks for your replies. :)

[Palmer]

I run into the same problem. Opening big files with WordPad takes forever and I have Pentium 4 running WinXP. I don't have Excel. What should I use?

[Sardtok]

Well Palmer,
it doesn't matter much how fast your cpu is (it matters, but it could just as well have been a 100 mhz for this)...
What matters is the read and transfer speed of your hardrive and EIDE controller (or SCSI controller or whatever type of controller you have your hard drive connected to) and the transfer speed to memory.
These programs buffer the entire file to memory, which is evil, and bad...
It's better to buffer for instance 1 Mb at a time, and read another one when required (not good for statistics and such in Excel though)...

Anyway,
if it was possible to pipe the filestream you could reduce the amount sent to the program.
Or you could use a program that splits files into files with a set number of lines.
I don't know of any such programs.
Programs for splitting files are quite common though and should be available somewhere.
I've never worked with piped streams so not sure if you can (or for that matter how) you do it on windows.
I've seen it done on Linux (which is the same on UNIX)...

I'll try to see if I can find a program that can pipe streams on windows,
that way you don't have to split the file... ;)

[EDIT]
Tried piping a text file to Excel using more,
doesn't look like Excel reads piped streams.
Anyway,
I could probably write a program myself that would split up files (Java though, don't remember much C, and haven't yet worked with C++),
but there must be a better way... :(

[Palmer]

Don't hurt yourself :)

The main reason I want to look is that AwStats does not do a good job of the actual page you received a referral from when it comes to forums.

My most referrals come from this link...
http://www.sitepointforums.com/showthread.php

Obviously not very helpful. In the logs it shows the actual page. I really don't need them that much, just wanting to know out of curiosity what people are saying about my sites :)

[Sardtok]

If you really want to bother Steven,
then you could ask him to grep for it... ;)
(grep is a GNU program made for searching files or piped streams using regular expressions, it will search for lines containing the regular expression, so nice for finding certain things in a log file, for instance you can grep for 404 and get all lines in an access log with the 404 error)... ;)

[Palmer]

I've bothered Steven enough this week. :) I eventually find the link to my site anyway, just would be faster with the ability to run a search on the log files.