<?xml version="1.0"?>

<!--

 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at
 
      https://www.apache.org/licenses/LICENSE-2.0
 
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.

-->

<document xmlns="http://maven.apache.org/XDOC/2.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/XDOC/2.0 https://maven.apache.org/xsd/xdoc-2.0.xsd">
 <properties>
  <title>Troubleshooting Guide</title>
  <author email="dev@commons.apache.org">Apache Commons Team</author>
 </properties>

 <body> 
<section name="Contents">
<ul>
<li>
<a href="#Contents">Contents</a>
</li>
<li>
<a href="#Using JCL Diagnostics">Using JCL Diagnostics</a>
<ul>
<li>
<a href="#When To Use Diagnostic Logging">When To Use Diagnostic Logging</a>
</li>
<li>
<a href="#How To Use Diagnostic logging">How To Use Diagnostic logging</a>
</li>
<li>
<a href="#OIDs">OIDs</a>
</li>
<li>
<a href="#Diagnostic Message Prefix">Diagnostic Message Prefix</a>
</li>
<li>
<a href="#ClassLoader Hierarchy Tree">ClassLoader Hierarchy Tree</a>
</li>
<li>
<a href="#LogFactory Class Bootstrap">LogFactory Class Bootstrap</a>
</li>
<li>
<a href="#Construction Of LogFactoryImpl Instances">Construction Of LogFactoryImpl Instances</a>
</li>
<li>
<a href="#Log Discovery Diagnostics">Log Discovery Diagnostics</a>
</li>
</ul>
</li>
<li>
<a href="#Containers With Custom LogFactory Implementations">Containers With Custom LogFactory Implementations</a>
<ul>
<li>
<a href="#The Incompatible LogFactory Issue">The Incompatible LogFactory Issue</a>
<ul>
<li>
<a href="#Symptoms">Symptoms</a>
</li>
<li>
<a href="#Explanation">Explanation</a>
</li>
<li>
<a href="#Fixes">Fixes</a>
</li>
</ul>
</li>
</ul>
</li>
<li>
<a href="#Containers With Custom ClassLoading Behaviour for Logging">Containers With Custom ClassLoading Behaviour for Logging</a>
<ul>
<li>
<a href="#Apache Tomcat">Apache Tomcat</a>
</li>
<li>
<a href="#JBoss Application Server">JBoss Application Server</a>
</li>
<li>
<a href="#Other Containers">Other Containers</a>
</li>
</ul>
</li>
</ul>
</section>
  <section name='Using JCL Diagnostics'>
    <p>
Diagnostics is a feature introduced in JCL 1.1 as an aid to debugging problems 
with JCL configurations. When diagnostics are switched on, messages are logged 
to a stream (specified by the user) by the two main classes involved in discovery
in JCL (<code>LogFactory</code> and <code>LogFactoryImpl</code>).
    </p>
    <p>
Diagnostics are intended to be used in conjunction with the source. The source
contains numerous and lengthy comments. Often these are intended to help explain
the meaning of the messages.
    </p>
    <subsection name='When To Use Diagnostic Logging'>
      <p>
Diagnostic logging is intended only to be used when debugging a problematic 
configuration. It <em>should</em> be switched off for production. 
      </p>
    </subsection>
    <subsection name='How To Use Diagnostic logging'>
      <p>
Diagnostic logging is controlled through the system property 
<code>org.apache.commons.logging.diagnostics.dest</code>. Setting the property value
to the special strings <code>STDOUT</code> or <code>STDERR</code> (case-sensitive) 
will output messages to <code>System.out</code> and <code>System.err</code> respectively.
Setting the property value to a valid file name will result in the messages being logged
to that file.
      </p>
    </subsection> 
    <subsection name='OIDs'>
      <p>
Diagnostics uses the concept of an Object ID (OID). This allows the identity of objects 
to be tracked without relying on useful <code>toString</code> implementations. 
These are of the form: 
      </p>
<code>
<em>classname</em>@<em>system identity hash code</em>
</code>
      <p>
The <em>system identity hash code</em> is found by calling <code>System.identityHashCode()</code> 
which should uniquely identify a particular instance. The classname is usually the fully qualified
class name though in a few cases, <code>org.apache.commons.logging.impl.LogFactoryImpl</code> may be
shortened to <code>LogFactoryImpl</code> to increase ease of reading. For example:
      </p>
<code>
sun.misc.Launcher$AppClassLoader@20120943
LogFactoryImpl@1671711
</code>
      <p>
OIDs are intended to be used to cross-reference. They allow particular instances of classloaders
and JCL classes to be tracked in different contexts. This plays a vital role in building
up the understanding of the classloader environment required to diagnose JCL problems. 
      </p> 
    </subsection>
    <subsection name='Diagnostic Message Prefix'>
      <p>
Each diagnostic message is prefixed with details of the relevant class in a standard format.
This takes the form:
    </p>
<code>
[<em>class-identifier</em> from <em>ClassLoader OID</em>]
</code>
    <p>
<em>ClassLoader OID</em> is the <a href='#OIDs'>OID</a> of a classloader which loaded 
the class issuing the message.
<em>class-identifier</em> identifies the object issuing the message. 
    </p>
    <p>
In the case of 
<code>LogFactory</code>, this is just <code>LogFactory</code>. For example (line split):
    </p>
<code>
[LogFactory 
   from sun.misc.Launcher$AppClassLoader@20120943] BOOTSTRAP COMPLETED
</code>
    <p>
In the case of 
<code>LogFactoryImpl</code>, the prefix is the instance OID. This can be cross referenced
to discover the details of the TCCL used to manage this instance. For example (line split):
    </p>
<code>
[LogFactoryImpl@1671711 
   from sun.misc.Launcher$AppClassLoader@20120943] Instance created.
</code>
    </subsection>
    <subsection name='ClassLoader Hierarchy Tree'>
      <p>
Understanding the relationships between classloaders is vital when debugging JCL.
At various points, JCL will print to the diagnostic log the hierarchy for important
classloaders. This is obtained by walking the tree using <code>getParent</code>.
Each classloader is represented (visually) by an OID (to allow cross referencing) 
and the relationship indicated in <code><em>child</em> --> <em>parent</em></code> fashion.
For example (line split for easy reading):
      </p>
<code>
ClassLoader tree:java.net.URLClassLoader@3526198  
      --> sun.misc.Launcher$AppClassLoader@20120943 (SYSTEM) 
      --> sun.misc.Launcher$ExtClassLoader@11126876 
      --> BOOT
</code>
      <p>
Represents a hierarchy with four elements ending in the boot classloader.
      </p>
    </subsection>
    <subsection name='LogFactory Class Bootstrap'>
      <p>
Whenever the <code>LogFactory</code> class is initialized, diagnostic messages about 
the classloader environment are logged. The content of each of these messages is prefixed by 
<code>[ENV]</code> to help distinguish them. The extension directories, application classpath, 
details of the classloader (including the <a href='#OIDs'>OID</a> and <code>toString</code> 
value) used to load <code>LogFactory</code> and the 
<a href='#ClassLoader%20Hierarchy%20Tree'>classloader tree</a> for that classloader 
are logged.
      </p>
      <p>
Many Sun classloaders have confusing <code>toString</code> values. For example, the OID may be
      </p>
<code>
sun.misc.Launcher$AppClassLoader@20120943
</code>
      <p>
with a <code>toString</code> value of
      </p>
<code>
sun.misc.Launcher$AppClassLoader@133056f
</code>
      <p>
Other classloader implementations may give very useful information (such as the local classpath).
      </p>
      <p>
Finally, once initialization is complete a <code>BOOTSTRAP COMPLETED</code> message is issued.
      </p>
    </subsection>
    <subsection name='Construction Of LogFactoryImpl Instances'>
      <p>
<code>LogFactoryImpl</code> is the standard and default <code>LogFactory</code> implementation.
This section obviously only applies to configurations using this implementation.
      </p>
      <p>
Before assigning a <code>Log</code> instance, <code>LogFactory</code> loads a 
<code>LogFactory</code> implementation. The content is prefixed by <code>[LOOKUP]</code>
for each diagnostic message logged by this process.
      </p> 
      <p>
The implementation used can vary per Thread context classloader (TCCL). If this is the first time 
that a Log has been requested for a particular TCCL a new instance will be created.
      </p>
      <p>
Information of particular interest is logged at this stage. Details of the TCCL are logged
allowing the <a href='#OIDs'>OID</a> later to be cross-referenced to the <code>toString</code> value 
and the <a href='#ClassLoader%20Hierarchy%20Tree'>classloader tree</a>. For example, the 
following log snippet details the TCCL (lines split):
      </p>
<code>
[LogFactory from sun.misc.Launcher$AppClassLoader@20120943] 
    [LOOKUP] LogFactory implementation requested for the first time for context 
        classloader java.net.URLClassLoader@3526198
[LogFactory from sun.misc.Launcher$AppClassLoader@20120943] 
    [LOOKUP] java.net.URLClassLoader@3526198 == 'java.net.URLClassLoader@35ce36'
[LogFactory from sun.misc.Launcher$AppClassLoader@20120943] 
    [LOOKUP] ClassLoader tree:java.net.URLClassLoader@3526198 
        --> sun.misc.Launcher$AppClassLoader@20120943 (SYSTEM)  
          --> sun.misc.Launcher$ExtClassLoader@11126876 
            --> BOOT
</code>
    </subsection>
    <subsection name='Log Discovery Diagnostics'>
      <p>
The standard <code>LogFactoryImpl</code> issues many diagnostic messages when discovering
the <code>Log</code> implementation to be used.
      </p>
      <p>
During discovery, environment variables are loaded and values set. This content is prefixed by 
<code>[ENV]</code> to make it easier to distinguish this material.
      </p>
      <p>
The possible messages issued during discovery are numerous. To understand them, the source
should be consulted. Attention should be paid to the classloader hierarchy trees for the
classloader used to load <code>LogFactory</code> and to the TCCL.
      </p>
    </subsection>
  </section>
  <section name='Containers With Custom LogFactory Implementations'>
    <p>
 Some containers use a custom <code>LogFactory</code> implementation to adapt JCL to their particular
 logging system. This has some important consequences for the deployment of applications using JCL within 
 these containers.
    </p>
    <p>
 Containers known to use this mechanism:
    </p>
    <ul>
      <li><a href='https://www.ibm.com/software/websphere/'>WebSphere Application Server</a> from
      <a href='https://www.ibm.com/software/websphere/'>IBM</a> (versions 5 and 6).</li>
    </ul>
    <p>
 Containers suspected to use this mechanism:
    </p>
    <ul>
      <li>WebSphere Application Server (other versions).</li>
    </ul>
    <p>
The Apache Commons team would be grateful if reports were posted to the development list
of other containers using a custom implementation.
    </p>
    <subsection name='The Incompatible LogFactory Issue'>
      <h4 id='Symptoms'>Symptoms</h4>
        <p>
  An exception is thrown by JCL with a message similar to:
        </p>
  <code>
  The chosen LogFactory implementation does not extend LogFactory. Please check your configuration. 
  (Caused by java.lang.ClassCastException: The application has specified that a custom LogFactory 
  implementation should be used but Class 'com.ibm.ws.commons.logging.TrLogFactory' cannot be converted 
  to 'org.apache.commons.logging.LogFactory'. The conflict is caused by the presence of multiple 
  LogFactory classes in incompatible classloaders. Background can be found in 
  https://commons.apache.org/logging/tech.html. If you have not explicitly specified a custom
  LogFactory then it is likely that the container has set one without your knowledge. 
  In this case, consider using the commons-logging-adapters.jar file or specifying the standard 
  LogFactory from the command line. Help can be found @https://commons.apache.org/logging.
  </code>
        <p>
  This is a WebSphere example so the name of the custom LogFactory is 
  <code>com.ibm.ws.commons.logging.TrLogFactory</code>. For other containers, this class name will
  differ.
        </p>
      <h4 id='Explanation'>Explanation</h4>
      <p>
 A custom <code>LogFactory</code> implementation can only be used if the implementation class loaded 
 dynamically at runtime can be cast to the <code>LogFactory</code> class that loaded it. There are 
 several ways in which this cast can fail. The most obvious is that the source code may not actually 
 extend <code>LogFactory</code>. The source may be compatible but if the <code>LogFactory</code> class
 against which the source is compiled is not binary compatible then the cast will also fail.
      </p>
      <p>
 There is also another more unusual way in which this cast can fail: even when the binary is compatible,
 the implementation class loaded at runtime may be linked to a different instance of the 
 <code>LogFactory</code> class. For more information, see the <a href='tech.html'>tech guide</a>.
      </p>
      <p>
 This situation may be encountered in containers which use a custom <code>LogFactory</code> implementation.
 The implementation will typically be provided in a shared, high level classloader together with JCL.
 When an application classloader contains <code>LogFactory</code>, the implementation will be loaded 
 from that higher level classloader. The implementation class will be linked to the <code>LogFactory</code>
 class loaded by the higher level classloader. Even if the 
 <code>LogFactory</code> implementations are binary compatible, since they are loaded by different classloaders
 the two <code>LogFactory</code> Class instances are not equal and so the cast must fail.
      </p>
      <p>
The policy adopted by JCL in this situation is to re-throw this exception. Additional information
is included in the message to help diagnosis. The reasoning behind this choice is that a 
particular <code>LogFactory</code> implementation has been actively specified and this
choice should not be ignored. This policy has unfortunate consequences when running in
containers which have custom implementations: the above runtime exception may be thrown
under certain classloading policies without the user knowingly specifying a custom 
implementation.
      </p>
      <h4 id='Fixes'>Fixes</h4>
        <p>
 There are various ways to fix this problem. Which fix is right depends on the circumstances.
        </p>
        <p>
 If you are happy using another classloading policy for the application, select a 
 classloading policy which ensures that <code>LogFactory</code> will be loaded from the
 shared classloader containing the custom implementation.
        </p>
        <p>
 If you want to bypass the container adaption mechanism then set the appropriate system property 
 to the default value when the container is started:
        </p>
 <code>
 -Dorg.apache.commons.logging.LogFactory=org.apache.commons.logging.impl.LogFactoryImpl
 </code>
      <p>
 If you want to continue to use the default container mechanism then:
      </p>
      <ul>
        <li>
 Find and replace the commons-logging implementation used by the container with
 the most modern release
        </li>
        <li>
 Replace the commons-logging jar in the application with the commons-logging-adapters jar.
 This will ensure that application classloader will delegate to it's parent when loading
 <code>LogFactory</code>.
        </li>
      </ul>
      <p>
 If you encounter difficulties when applying the fixes recommended, please turn on
 <a href='#Using JCL Diagnostics'>diagnostics</a> and consult the logs.
      </p>
    </subsection>
  </section>
  <section name='Containers With Custom ClassLoading Behaviour for Logging'>
    <p>
 Because commons-logging is such a fundamental library, some containers modify the way
 in which classloading behaves for commons-logging classes.
   </p>
   <subsection name="Apache Tomcat">
   <p>
 At the current date, Tomcat 5.5.16 is the current release. All releases from version
 4.1.x through 5.5.16 have a startup process that places jarfile 
 ${tomcat.home}/bin/commons-logging-api.jar in the system classpath and then
 prevents any webapp from overriding the classes in that jarfile. Effectively, all
 webapps behave as if "parent-first" classloading were enabled for those classes.
   </p>
   <p>
 This has some benefits; in particular it means that there are no problems in
 these Tomcat versions with having multiple copies of the commons-logging Log
 interface in the classpath (which avoids the "Log does not implement Log"
 problem described elsewhere).
   </p>
   <p>
 However it also means that no webapp can override the core commons-logging
 classes by including an updated commons-logging jarfile in WEB-INF/lib; any
 class already loaded via the container takes priority. In particular, as
 Tomcat bundles logging 1.0.4 only, the new diagnostics and memory-leak-prevention
 features of the 1.1 release will not be available unless the container's
 library version is updated.
   </p>
   <p>
 Because the commons-logging-api.jar in the container does not contain any
 log-library-adapter classes, updated behavior for these <i>will</i> be
 seen when logging 1.1 is bundled in WEB-INF/lib. In particular, the 
 support for Log4j's TRACE level will take effect without having to update
 the container.
   </p>
   <p>
 If you do wish to update Tomcat's version of commons-logging, then you
 <i>must</i> use the commons-logging-1.1-api jar only, not the full jar.
 Classes in the webapp cannot override classes loaded from the system
 classpath set up during Tomcat's startup process, and logging adapters
 can only see their matching concrete logging library if that library is
 available in the same classpath. Bundling the full commons-logging jarfile
 (with adapters) into the system classpath therefore means that logging
 libraries (eg Log4j) within WEB-INF/lib are not accessible.
   </p>
   <p>
 Note that the behavior described here only applies if the standard Tomcat
 startup process is run. When Tomcat is embedded in a larger
 framework (eg run embedded within an IDE) this may not apply.
   </p>
   </subsection>
   <subsection name="JBoss Application Server">
   <p>
   The JBoss Application Server can be configured to prevent deployed
   code from overriding classes higher in the hierarchy, effectively
   forcing "parent-first" behavior for selected classes. By default,
   commons-logging is in this list (at least for some JBoss versions
   starting with 4.0.2), and therefore including an updated version
   of commons-logging in WEB-INF/lib or similar will have no effect.
   See the JBoss classloading documentation for more details.
   </p>
   </subsection>
   <subsection name="Other Containers">
   <p>
   As more information becomes available on this topic, it may be added
   to the commons-logging wiki site.
   </p>
   </subsection>
  </section>
 </body>
</document>
