QPID 0.32 going out of memory

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

QPID 0.32 going out of memory

Akhil Samnotra

> Hi,
> We are using QPID 0.32 version .
> We have observed that the qpid broker is going out of memory and having memory leak even though we have increased the heap memory to 3 GB and that was observed using MAT TOOl. It was seen that the memory is consumed by Java io operation while creating the logs . These log objects  are consuming extremely memory. It can be seen in the 1 st screenshot that the string object for log is getting created and then the log is stored in the backup folder .
> I' m attaching the screen shots which I have taken using MAT tool .
>

>

>

>
> Kindly suggest me how can this be avoided .
> Thanks
>
> Akhil samnotra
>
> Sent from my iPhone



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: QPID 0.32 going out of memory

Oleksandr Rudyy
Hi Akhil,
It seems attachments have been stripped off. Could you please resend them
or raise a JIRA [1] and attach your screenshots to the JIRA?

Kind Regards,
Alex

[1] https://issues.apache.org/jira/projects/QPID
Reply | Threaded
Open this post in threaded view
|

Re: QPID 0.32 going out of memory

Oleksandr Rudyy
Hi Akhil,

The screenshots did not get through with your latest email. Could you
please try raising JIRA and attaching the screnshots to it?

Kind Regards,
Alex

On 17 June 2017 at 06:04, Akhil Samnotra <[hidden email]> wrote:

> hi ,
>
> please find the screenshots attached.
> thanks
>
> Akhil
>
> On Sat, Jun 17, 2017 at 1:52 AM, Oleksandr Rudyy <[hidden email]> wrote:
>
>> Hi Akhil,
>> It seems attachments have been stripped off. Could you please resend them
>> or raise a JIRA [1] and attach your screenshots to the JIRA?
>>
>> Kind Regards,
>> Alex
>>
>> [1] https://issues.apache.org/jira/projects/QPID
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
Reply | Threaded
Open this post in threaded view
|

Re: QPID 0.32 going out of memory

Rob Godfrey
On 19 June 2017 at 10:17, Oleksandr Rudyy <[hidden email]> wrote:

> Hi Akhil,
>
> The screenshots did not get through with your latest email. Could you
> please try raising JIRA and attaching the screnshots to it?
>
> Kind Regards,
> Alex
>
>
I believe Akhil has already raised a JIRA here: https://issues.apache.org/
jira/browse/QPID-7828 with the screenshots attached.

-- Rob


> On 17 June 2017 at 06:04, Akhil Samnotra <[hidden email]> wrote:
>
> > hi ,
> >
> > please find the screenshots attached.
> > thanks
> >
> > Akhil
> >
> > On Sat, Jun 17, 2017 at 1:52 AM, Oleksandr Rudyy <[hidden email]>
> wrote:
> >
> >> Hi Akhil,
> >> It seems attachments have been stripped off. Could you please resend
> them
> >> or raise a JIRA [1] and attach your screenshots to the JIRA?
> >>
> >> Kind Regards,
> >> Alex
> >>
> >> [1] https://issues.apache.org/jira/projects/QPID
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: QPID 0.32 going out of memory

Oleksandr Rudyy
In reply to this post by Oleksandr Rudyy
Hi Akhil,

Thanks for raising JIRA and attaching the screenshots from MAT.
As per screnshot [1] it seems that heap memory was consumed by java.io.File
objects. Judging by the file names, the file objects were created by log
appender whilst compression broker logs.

What type of file appender are you using with Qpid Broker? Is it
"org.apache.log4j.QpidCompositeRollingAppender"? You can check
${QPID_HOME}/etc/log4j.xml for logging configuration?
We are not aware about any defect in QpidCompositeRollingAppender which
would manifest in heap memory consumption by java.io.File objects.

The broker logging functionality in version 0.32 is based on log4j . It was
replaced with slf4j and logback in following  6.x versions.
You can consider migration to newer version of broker (6.1.3 at the moment
of writing this email). If migration of broker is not an option, you can
try replacing the log appender with another one.


Kind Regards,
Alex


[1] https://issues.apache.org/jira/browse/QPID-7828
[2] https://issues.apache.org/jira/secure/attachment/12873368/image1.PNG


On 19 June 2017 at 11:46, Akhil Samnotra <[hidden email]> wrote:

> HI ,
> I have even raised the jira but didnt got any resolution.
>
> Thanks
> Akhil
>
> On Mon, Jun 19, 2017 at 1:47 PM, Oleksandr Rudyy <[hidden email]> wrote:
>
> > Hi Akhil,
> >
> > The screenshots did not get through with your latest email. Could you
> > please try raising JIRA and attaching the screnshots to it?
> >
> > Kind Regards,
> > Alex
> >
> > On 17 June 2017 at 06:04, Akhil Samnotra <[hidden email]>
> wrote:
> >
> > > hi ,
> > >
> > > please find the screenshots attached.
> > > thanks
> > >
> > > Akhil
> > >
> > > On Sat, Jun 17, 2017 at 1:52 AM, Oleksandr Rudyy <[hidden email]>
> > wrote:
> > >
> > >> Hi Akhil,
> > >> It seems attachments have been stripped off. Could you please resend
> > them
> > >> or raise a JIRA [1] and attach your screenshots to the JIRA?
> > >>
> > >> Kind Regards,
> > >> Alex
> > >>
> > >> [1] https://issues.apache.org/jira/projects/QPID
> > >>
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: QPID 0.32 going out of memory

Oleksandr Rudyy
Akhil,

Appenders of type "org.apache.log4j.FileAppender" do not compress log files.
Thus, it means, that "ArchivingFileAppender" (of type
QpidCompositeRollingAppender ) is causing the issue. Potentially,  removal
or replacement of this appender with any suitable FileAppender (for
example, DailyRollingFileAppender or RollingFileAppender or any other log4j
file appender) might fix the issue.

> You mean to say that we should change Log4j and use
> slf4j instead.
> Yes presently we are using Log4j as appender and we have set the
> logging_level="info". Does changing it to only error will help?

sl4j is a logging framework used in newer versions of qpid brokers (6.0 and
above). You cannot use it with 0.32 version of Qpid Broker. In 6.x brokers
logging configuration is stored in broker own configuration.

"FileAppender" is a separate appender.  "FileAppender" and "Archiving
FileAppender" logs into different files separately from each other.
"FileAppender" does not compress its log files, whilst "Archiving
FileAppender" does it.
Changing log level on "FileAppender" to "error" will not help with the
issue.

Removal or replacement of "ArchivingFileAppender" (
QpidCompositeRollingAppender ) should fix the issue.

Kind Regards,
Alex


On 19 June 2017 at 14:13, Akhil Samnotra <[hidden email]> wrote:

> Also , we are using
> appender class="org.apache.log4j.QpidCompositeRollingAppender"
> name="ArchivingFileAppender">
>         <!-- Ensure that logs always have the DatePattern appended to the
> filename.
>              DEFAULT IF NOT CONFIGURED: true -->
>         <param name="StaticLogFileName" value="true"/>
>         <param name="file"
> value="${QPID_WORK}/log/${logprefix}qpidlog${logsuffix}.log"/>
>         <!-- Style of rolling to use, by:
>              File Size(1)
>              Date(2)
>              Both(3)
>
>              When Date (or Both) is enabled then the value of DatePattern
> will determine
>              when the new file is made. e.g. a DatePattern of
> "'.'yyyy-MM-dd-HH-mm"
>              which includes minutes will cause a new backup file to be made
> every minute.
>
>              DEFAULT IF NOT CONFIGURED: 3 -->
>         <param name="RollingStyle" value="3"/>
>         <!-- Set the count direction:
>              Negative numbers mean backups are numbered <latest>, .0, .1,
> .2,..., .n
>              0 means backup is DatePattern stamped and followed with a
> Positive number
>                  if the DatePattern stamp clashes with other existing
> backups.
>              Positive numbers mean backups are numbered 0, 1, 2, ..., n,
> <latest>
>
>              DEFAULT IF NOT CONFIGURED: -1 -->
>         <param name="CountDirection" value="0"/>
>         <!-- Maximum File Size:
>              DEFAULT IF NOT CONFIGURED: 10MB -->
>         <param name="MaxFileSize" value="10000"/>
>         <!-- Date Pattern:
>              DEFAULT IF NOT CONFIGURED: "'.'yyyy-MM-dd" -->
>         <param name="DatePattern" value="'.'yyyy-MM-dd-HH-mm"/>
>         <!-- Maximum number of backup files:
>               0 means no backups
>              -1 means infinite backups
>
>              DEFAULT IF NOT CONFIGURED: 0 -->
>         <param name="MaxSizeRollBackups" value="1000"/>
>         <!-- Compress(gzip) the backup files to the backup location:
>              DEFAULT IF NOT CONFIGURED: FALSE -->
>         <param name="CompressBackupFiles" value="true"/>
>         <!-- Compress the backup files using a second thread:
>              DEFAULT IF NOT CONFIGURED: FALSE -->
>         <param name="CompressAsync" value="true"/>
>         <!-- Backup Location:
>              DEFAULT IF NOT CONFIGURED: same dir as log file -->
>         <param name="backupFilesToPath" value="${QPID_WORK}/log/backup"/>
>
>         <layout class="org.apache.log4j.PatternLayout">
>             <param name="ConversionPattern" value="%d %-5p [%t] (%c{2}) -
> %m%n"/>
>         </layout>
>     </appender>
>
> We are presently using this configuration in Log4j.xml.
>
> Can you please help?
>
> Thanks
> Akhil Samnotra
>
>
> On Mon, Jun 19, 2017 at 6:40 PM, Akhil Samnotra <[hidden email]>
> wrote:
>
> > Hi ,
> > Thanks for the help . You mean to say that we should change Log4j and use
> > slf4j instead.
> > Yes presently we are using Log4j as appender and we have set the
> > logging_level="info". Does changing it to only error will help?
> >
> > This is the present configuration :
> >
> > <appender class="org.apache.log4j.FileAppender" name="FileAppender">
> >         <param name="File" value="${QPID_WORK}/backuplogs/${logprefix}
> > qpidmaster${logsuffix}.log"/>
> >         <param name="Append" value="true"/>
> >
> >         <layout class="org.apache.log4j.PatternLayout">
> >             <param name="ConversionPattern" value="%d %-5p [%t] (%c{2}) -
> > %m%n"/>
> >         </layout>
> >  </appender>
> >
> > Can we do anything in this configuration to get rid of the issue?
> >
> > Thanks
> > Akhil
> >
> > On Mon, Jun 19, 2017 at 6:31 PM, Oleksandr Rudyy <[hidden email]>
> wrote:
> >
> >> Hi Akhil,
> >>
> >> Thanks for raising JIRA and attaching the screenshots from MAT.
> >> As per screnshot [1] it seems that heap memory was consumed by
> >> java.io.File
> >> objects. Judging by the file names, the file objects were created by log
> >> appender whilst compression broker logs.
> >>
> >> What type of file appender are you using with Qpid Broker? Is it
> >> "org.apache.log4j.QpidCompositeRollingAppender"? You can check
> >> ${QPID_HOME}/etc/log4j.xml for logging configuration?
> >> We are not aware about any defect in QpidCompositeRollingAppender which
> >> would manifest in heap memory consumption by java.io.File objects.
> >>
> >> The broker logging functionality in version 0.32 is based on log4j . It
> >> was
> >> replaced with slf4j and logback in following  6.x versions.
> >> You can consider migration to newer version of broker (6.1.3 at the
> moment
> >> of writing this email). If migration of broker is not an option, you can
> >> try replacing the log appender with another one.
> >>
> >>
> >> Kind Regards,
> >> Alex
> >>
> >>
> >> [1] https://issues.apache.org/jira/browse/QPID-7828
> >> [2] https://issues.apache.org/jira/secure/attachment/
> 12873368/image1.PNG
> >>
> >>
> >> On 19 June 2017 at 11:46, Akhil Samnotra <[hidden email]>
> wrote:
> >>
> >> > HI ,
> >> > I have even raised the jira but didnt got any resolution.
> >> >
> >> > Thanks
> >> > Akhil
> >> >
> >> > On Mon, Jun 19, 2017 at 1:47 PM, Oleksandr Rudyy <[hidden email]>
> >> wrote:
> >> >
> >> > > Hi Akhil,
> >> > >
> >> > > The screenshots did not get through with your latest email. Could
> you
> >> > > please try raising JIRA and attaching the screnshots to it?
> >> > >
> >> > > Kind Regards,
> >> > > Alex
> >> > >
> >> > > On 17 June 2017 at 06:04, Akhil Samnotra <[hidden email]>
> >> > wrote:
> >> > >
> >> > > > hi ,
> >> > > >
> >> > > > please find the screenshots attached.
> >> > > > thanks
> >> > > >
> >> > > > Akhil
> >> > > >
> >> > > > On Sat, Jun 17, 2017 at 1:52 AM, Oleksandr Rudyy <
> [hidden email]>
> >> > > wrote:
> >> > > >
> >> > > >> Hi Akhil,
> >> > > >> It seems attachments have been stripped off. Could you please
> >> resend
> >> > > them
> >> > > >> or raise a JIRA [1] and attach your screenshots to the JIRA?
> >> > > >>
> >> > > >> Kind Regards,
> >> > > >> Alex
> >> > > >>
> >> > > >> [1] https://issues.apache.org/jira/projects/QPID
> >> > > >>
> >> > > >
> >> > > >
> >> > > >
> >> > > > ------------------------------------------------------------
> >> ---------
> >> > > > To unsubscribe, e-mail: [hidden email]
> >> > > > For additional commands, e-mail: [hidden email]
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: QPID 0.32 going out of memory

Keith Wall-2
Akhil,

You should be aware that the 0.32 version of the Qpid Java Broker was
affected by a number of CVEs.  See Apache Qpid's Security page for
information.

https://qpid.apache.org/components/java-broker/security.html

You ought to be planning to upgrade to a newer version.  You get get
the latest from here:

https://qpid.apache.org/components/java-broker/

Hope this helps

Keith.

On 19 June 2017 at 16:32, Oleksandr Rudyy <[hidden email]> wrote:

> Akhil,
>
> Appenders of type "org.apache.log4j.FileAppender" do not compress log files.
> Thus, it means, that "ArchivingFileAppender" (of type
> QpidCompositeRollingAppender ) is causing the issue. Potentially,  removal
> or replacement of this appender with any suitable FileAppender (for
> example, DailyRollingFileAppender or RollingFileAppender or any other log4j
> file appender) might fix the issue.
>
>> You mean to say that we should change Log4j and use
>> slf4j instead.
>> Yes presently we are using Log4j as appender and we have set the
>> logging_level="info". Does changing it to only error will help?
>
> sl4j is a logging framework used in newer versions of qpid brokers (6.0 and
> above). You cannot use it with 0.32 version of Qpid Broker. In 6.x brokers
> logging configuration is stored in broker own configuration.
>
> "FileAppender" is a separate appender.  "FileAppender" and "Archiving
> FileAppender" logs into different files separately from each other.
> "FileAppender" does not compress its log files, whilst "Archiving
> FileAppender" does it.
> Changing log level on "FileAppender" to "error" will not help with the
> issue.
>
> Removal or replacement of "ArchivingFileAppender" (
> QpidCompositeRollingAppender ) should fix the issue.
>
> Kind Regards,
> Alex
>
>
> On 19 June 2017 at 14:13, Akhil Samnotra <[hidden email]> wrote:
>
>> Also , we are using
>> appender class="org.apache.log4j.QpidCompositeRollingAppender"
>> name="ArchivingFileAppender">
>>         <!-- Ensure that logs always have the DatePattern appended to the
>> filename.
>>              DEFAULT IF NOT CONFIGURED: true -->
>>         <param name="StaticLogFileName" value="true"/>
>>         <param name="file"
>> value="${QPID_WORK}/log/${logprefix}qpidlog${logsuffix}.log"/>
>>         <!-- Style of rolling to use, by:
>>              File Size(1)
>>              Date(2)
>>              Both(3)
>>
>>              When Date (or Both) is enabled then the value of DatePattern
>> will determine
>>              when the new file is made. e.g. a DatePattern of
>> "'.'yyyy-MM-dd-HH-mm"
>>              which includes minutes will cause a new backup file to be made
>> every minute.
>>
>>              DEFAULT IF NOT CONFIGURED: 3 -->
>>         <param name="RollingStyle" value="3"/>
>>         <!-- Set the count direction:
>>              Negative numbers mean backups are numbered <latest>, .0, .1,
>> .2,..., .n
>>              0 means backup is DatePattern stamped and followed with a
>> Positive number
>>                  if the DatePattern stamp clashes with other existing
>> backups.
>>              Positive numbers mean backups are numbered 0, 1, 2, ..., n,
>> <latest>
>>
>>              DEFAULT IF NOT CONFIGURED: -1 -->
>>         <param name="CountDirection" value="0"/>
>>         <!-- Maximum File Size:
>>              DEFAULT IF NOT CONFIGURED: 10MB -->
>>         <param name="MaxFileSize" value="10000"/>
>>         <!-- Date Pattern:
>>              DEFAULT IF NOT CONFIGURED: "'.'yyyy-MM-dd" -->
>>         <param name="DatePattern" value="'.'yyyy-MM-dd-HH-mm"/>
>>         <!-- Maximum number of backup files:
>>               0 means no backups
>>              -1 means infinite backups
>>
>>              DEFAULT IF NOT CONFIGURED: 0 -->
>>         <param name="MaxSizeRollBackups" value="1000"/>
>>         <!-- Compress(gzip) the backup files to the backup location:
>>              DEFAULT IF NOT CONFIGURED: FALSE -->
>>         <param name="CompressBackupFiles" value="true"/>
>>         <!-- Compress the backup files using a second thread:
>>              DEFAULT IF NOT CONFIGURED: FALSE -->
>>         <param name="CompressAsync" value="true"/>
>>         <!-- Backup Location:
>>              DEFAULT IF NOT CONFIGURED: same dir as log file -->
>>         <param name="backupFilesToPath" value="${QPID_WORK}/log/backup"/>
>>
>>         <layout class="org.apache.log4j.PatternLayout">
>>             <param name="ConversionPattern" value="%d %-5p [%t] (%c{2}) -
>> %m%n"/>
>>         </layout>
>>     </appender>
>>
>> We are presently using this configuration in Log4j.xml.
>>
>> Can you please help?
>>
>> Thanks
>> Akhil Samnotra
>>
>>
>> On Mon, Jun 19, 2017 at 6:40 PM, Akhil Samnotra <[hidden email]>
>> wrote:
>>
>> > Hi ,
>> > Thanks for the help . You mean to say that we should change Log4j and use
>> > slf4j instead.
>> > Yes presently we are using Log4j as appender and we have set the
>> > logging_level="info". Does changing it to only error will help?
>> >
>> > This is the present configuration :
>> >
>> > <appender class="org.apache.log4j.FileAppender" name="FileAppender">
>> >         <param name="File" value="${QPID_WORK}/backuplogs/${logprefix}
>> > qpidmaster${logsuffix}.log"/>
>> >         <param name="Append" value="true"/>
>> >
>> >         <layout class="org.apache.log4j.PatternLayout">
>> >             <param name="ConversionPattern" value="%d %-5p [%t] (%c{2}) -
>> > %m%n"/>
>> >         </layout>
>> >  </appender>
>> >
>> > Can we do anything in this configuration to get rid of the issue?
>> >
>> > Thanks
>> > Akhil
>> >
>> > On Mon, Jun 19, 2017 at 6:31 PM, Oleksandr Rudyy <[hidden email]>
>> wrote:
>> >
>> >> Hi Akhil,
>> >>
>> >> Thanks for raising JIRA and attaching the screenshots from MAT.
>> >> As per screnshot [1] it seems that heap memory was consumed by
>> >> java.io.File
>> >> objects. Judging by the file names, the file objects were created by log
>> >> appender whilst compression broker logs.
>> >>
>> >> What type of file appender are you using with Qpid Broker? Is it
>> >> "org.apache.log4j.QpidCompositeRollingAppender"? You can check
>> >> ${QPID_HOME}/etc/log4j.xml for logging configuration?
>> >> We are not aware about any defect in QpidCompositeRollingAppender which
>> >> would manifest in heap memory consumption by java.io.File objects.
>> >>
>> >> The broker logging functionality in version 0.32 is based on log4j . It
>> >> was
>> >> replaced with slf4j and logback in following  6.x versions.
>> >> You can consider migration to newer version of broker (6.1.3 at the
>> moment
>> >> of writing this email). If migration of broker is not an option, you can
>> >> try replacing the log appender with another one.
>> >>
>> >>
>> >> Kind Regards,
>> >> Alex
>> >>
>> >>
>> >> [1] https://issues.apache.org/jira/browse/QPID-7828
>> >> [2] https://issues.apache.org/jira/secure/attachment/
>> 12873368/image1.PNG
>> >>
>> >>
>> >> On 19 June 2017 at 11:46, Akhil Samnotra <[hidden email]>
>> wrote:
>> >>
>> >> > HI ,
>> >> > I have even raised the jira but didnt got any resolution.
>> >> >
>> >> > Thanks
>> >> > Akhil
>> >> >
>> >> > On Mon, Jun 19, 2017 at 1:47 PM, Oleksandr Rudyy <[hidden email]>
>> >> wrote:
>> >> >
>> >> > > Hi Akhil,
>> >> > >
>> >> > > The screenshots did not get through with your latest email. Could
>> you
>> >> > > please try raising JIRA and attaching the screnshots to it?
>> >> > >
>> >> > > Kind Regards,
>> >> > > Alex
>> >> > >
>> >> > > On 17 June 2017 at 06:04, Akhil Samnotra <[hidden email]>
>> >> > wrote:
>> >> > >
>> >> > > > hi ,
>> >> > > >
>> >> > > > please find the screenshots attached.
>> >> > > > thanks
>> >> > > >
>> >> > > > Akhil
>> >> > > >
>> >> > > > On Sat, Jun 17, 2017 at 1:52 AM, Oleksandr Rudyy <
>> [hidden email]>
>> >> > > wrote:
>> >> > > >
>> >> > > >> Hi Akhil,
>> >> > > >> It seems attachments have been stripped off. Could you please
>> >> resend
>> >> > > them
>> >> > > >> or raise a JIRA [1] and attach your screenshots to the JIRA?
>> >> > > >>
>> >> > > >> Kind Regards,
>> >> > > >> Alex
>> >> > > >>
>> >> > > >> [1] https://issues.apache.org/jira/projects/QPID
>> >> > > >>
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > ------------------------------------------------------------
>> >> ---------
>> >> > > > To unsubscribe, e-mail: [hidden email]
>> >> > > > For additional commands, e-mail: [hidden email]
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]