Parsing a 700MB log file

Hi,

I have a log file which is 700MB in size and has lines and lines of information, separated by carriage returns, with the values in each line separated by commas.

I'm suppose to read through line by line of it before inserting the information in the database. Problem is, after I hit the upload button, I get a "The page cannot be displayed" page and got hang right at where the script parses through the file, while at other times my progress bar goes blinking and nothing happens.

I have these set in my PHP 4.3.6's ini file
max_execution_time = 180 ;
max_input_time = 120 ;
memory_limit = 8M
upload_max_filesize = 800M

What's the problem? Is it impossible to upload a 700MB file?

I have no idea but it strikes me as a bad idea. What kind of connection would you be uploading it over? For something like this you'd be better off doing it in c/c++ rather than php I think since it has good mysql access (I'm presuming thats your db) but won't cry when you show it the file.

-KM-

Alternatively, you could upload the log file by ftp (or whatever method you upload your php files with) and access it through the fopen() command. This way, you don't need to read the entire file at once, which is probably the problem, considering the 8MB memory limit to the php parser.
fgets() is ideal for log files, especially with ini_set("auto_detect_line_endings", "1");

good luck.

Yup, my db is MySQL and indeed I'm using fgets() to read the line and have set my ini_set("auto_detect_line_endings", "1");

kode_monkey, what do you mean what kind of connection I'll be uploading it over? CUrrently, I'm only testing it localhost. I'm not even attempting to insert the read line into db yet, I'm just merely trying to echo out all the read lines on screen, just for testing.

Itsacon, is it possible to increase my 8M memory limit of php parser without any performance degradation? Or is this recommended?

Basically, I'm trying to build a web-based reporting tool that imports log file to be analyzed. Hence, I would like to have the users upload the log files themselves using <input type=file> and not use other ftp programs to do it (which understandably is much easier)

Wouldn't really know about the mem limit, no doubt you can increase it, but it might increase the system load PHP generates.
I'd say make a benchmark program, and test the two settings.

As for uploading log files, my server logs each day in another file, so log files of 700 MB are rather uncommon (non-existent, in fact), so for now I'd decrease the size for a bit and get the program working. If all's fine, increase the size by small steps and see when it blows.
Don't forget to test on multiple systems, as this behaviour may vary between different OS.

Well uploading a file of that size (even over a lan) will take a non-negligable amount of time. You may want to run some tests to find out if the script is timing out before the upload completes. Since php was written for webpages I doubt anyone intended it to be used for something that large and thus you may need to tweek you settings some.

Thats not to say its a bad idea mind. I'm sure with a few tweeks here and there it'll work out nicely,

-KM-

Another possibility is to have people submit the files in a compressed format, and read them from PHP using the ZLib Library.

My experience is that text files (especially files with many identical parts, like log files) compress very well, so this may enable you to keep the logfile size limit a bit higher.

Hm...yeah, ok, will try out and see how. Thanks. Will keep you guys posted....