Transcription

1 Resolving Common Weblogic Stdout Log Rotation Problems on UNIX (Doc ID ) In this Document Purpose Troubleshooting Steps References APPLIES TO: Oracle WebLogic Server - Version 10.3 and later Generic UNIX PURPOSE The goal is to troubleshoot WebLogic stdout log rotation and truncation problems; these become significant when the stdout file starts growing beyond a manageable size. Specifically this note covers the usual problems, which are: 1) The stdout file is growing too big and cannot be truncated 2) The stdout file is growing too big and cannot be rotated 3) The stdout file is not growing too big but just isn't being rotated. TROUBLESHOOTING STEPS Review of the general principles ================================ a) stdout is a UNIX stream and weblogic cannot natively rotate it. b) Weblogic can however rotate its own server log files, and there are command-line flags / admin console GUI option / WLST methods to redirect stdout to the server log file - the idea is that then, if server log rotation is enabled, then stdout will also be rotated as it's going to be written to the same file. The standard methods include: i) Via the gui console: Admin server (or the Managed Server) -> Logging -> General -> Advanced Settings -> Redirect stdout logging enabled -> enable the checkbox. 1 of 5 12/02/ :57 PM

2 ii) Setting redirect-to-serverlog flags in a startup or environment script, for example Via JAVA_OPTIONS in commenv.sh : JAVA_OPTIONS=$JAVA_OPTIONS"-Dweblogic.log.RedirectStdoutToServerLogEnabled=true" Some logging flags can be used in addition to the above, and some are mutually exclusive. These can be used in addition to the above: -Dweblogic.StdoutDebugEnabled=true -Dweblogic.log.StdoutSeverityLevel=Info or -Dweblogic.log.StdoutSeverityLevel=Debug However the following are for naming specific files: -Dweblogic.stdout=/tmp/mystdout.log -Dweblogic.stderr=/tmp/mystderr.log or on Windows, for example: -Dweblogic.Stdout=C:\test\stdout.log So if these get added by mistake, they may stop the RedirectStdoutToServerLogEnabled option from working. iii) Via WLST. See Note How to Redirect WLST stdout to a Specified File iv) Via the linux logrotate system, ref Note : "WebLogic Server: Rotating stdout Log Files with UNIX OS Commands" The same 'logrotate' command is not available on Solaris, but there is a Solaris equivalent, called 'logadm,' described here in the documentation: /html/ /emcby.html NOTE: When using this logrotate system, be sure to use the 'copytruncate' option -- this truncates the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one. - It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. (Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost.) In other words, this says that some programs know how to cooperate with logrotate by closing the log file and creating a new one when commanded. Logrotate can do this by trapping sighup. c) The problem with the above standard methods is in how you start weblogic. Many choose to do: nohup startweblogic.sh > customlogfile.log 2>1& & this means that : i) the UNIX redirect will *override* any redirect options you have set either in the console or via the command line 2 of 5 12/02/ :57 PM

3 ii) even if you take the UNIX redirect away, then you will also get a nohup.out file generated, which will also grow potentially huge in size. For example : nohup startweblogic.sh The obvious workaround is to instead run it like this: startweblogic.sh..but then the drawback is that this needs to be run either from the system console, or from a secure terminal somewhere that is constantly left running, options which are not always feasible, because of remote security restrictions for console access. The rest of this document describes some workarounds to the above problem Known issues: Note : Log Rotation Does Not Happen When Rotated Log Directory And Current Log Are On Two Different Filesystems / mountpoints Note : WebLogic Server Log Rotation Not Working When Writing to NFS Mount -> both are fundamentally related to operating system limitations, so these are not bugs; for example the solution in the NFS note is simply to not use an NFS drive. The problem with logrotate There are some support notes that suggest that the solution to the stdout log rotation problem is to use the Linux utility 'logrotate' to rotate log files, which depending on certain factors, may work or may not, and may sometimes cause binary corruption of the out file andnyou may see the file still growing in size, and not getting truncated. (Ref: Note How to Rotate the Standard Output (STDOUT) File for WebLogic Server -- This contains basic /etc/logrotate.conf setup.) (If you are getting binary corruption, you can usually recreate the problem by trying to manually truncate the file: cat /dev/null > your_log_file; -> if the log file a) shows binary characters in b) is still huge in size then this is generally the same issue affecting 'logrotate.') Known workarounds/alternatives to the UNIX logrotate utility ) instead of a redirect, when starting weblogic you can pipe the stdout to the UNIX 'split' command, so do something like this: nohup startweblogic.sh split -l > what this does is separate the stdout into manageable line files. You can set this amount to anything you like, 1000 lines, lines etc. However for the problem of out files growing huge, seems a reasonably good value. In its basic form, split creates the chunks as files called xaa xab xac xad etc, increasing the letter on the suffix, however if you look at the man pages there are options to allow you to provide your own naming convention. To test the principle without risking a change to WebLogic, you can do the following. Create a test UNIX script called myloop containing: 3 of 5 12/02/ :57 PM

4 while [ 2 -gt 1 ] ; do echo "hello" done and then run the script like this:./myloop split -l from another terminal session, cd to the same directory and you will see the files being created: xaa xab etc. - The administrator can now then write a cron job to archive these files out as and when needed to save space. 2) The logrotate mechanism is not reliable if the stdout is created via a 'non append' redirect symbol: ">". However changing to ">>" should allow logrotate and truncates to work: nohup startweblogic.sh >> customlogfile.log 2>1 && This is because the file is created in "append" mode. The only drawback is that in some cases a new file may be required on each startup; in this case the administrator should simply move the file to archive before starting up WebLogic. 3) Many UNIXes have the /usr/sbin/rotatelogs utility; this is related to Apache log rotation. You can pipe the output of startweblogic to rotatelogs, which handles it all for you. This basic example rotates the file at 1 meg: nohup startweblogic.sh /usr/sbin/rotatelogs /path/to/your/log/out.log 1M & 4) If in spite of using >>, you find that logrotate/truncate is still not working, it could be caused by the 'noclobber' option set in the startup environment. See: /wiki/clobbering. If this is set, the logrotate will not work, and the solution is to unset it: set +o noclobber (set -o noclobber turns it on.) 5) Use a UNIX pipe: See Note : How To Rotate The nohup.out File By Size. This is a similar solution to the 'split' suggestion, but involves creating a named 'UNIX pipe,' and redirecting stdout to that on startup. You then use 'cat' to read from the pipe and redirect to a log file, and to stop the log file from being written you can kill the cat process, whereupon the log can be archived off or rotated manually, and then cat is restarted to continue to read from the pipe. 6) Check that you are not redirecting stdout to the actual server log file name on the command line; this will likely cause all sorts of problems. You will likely see problems regardless of whether you use > or >> So don't do either of these: startweblogic.sh > /path/user_projects/domains/doma/servera/logs/servera.log startweblogic.sh >> /path/user_projects/domains/doma/servera/logs/servera.log 7) For the extreme case where a quick solution to massive files is required, you can just destroy all stdout by starting up WebLogic like this: 4 of 5 12/02/ :57 PM

5 startweblogic.sh > /dev/null However no diagnostics are possible in the future, so this should be just a temporary measure, if used at all. Didn't find what you are looking for? 5 of 5 12/02/ :57 PM