this has started happening very recently. server gets to almost 18k online users. it works great for hours, suddenly it starts dropping connections.
there are many error entries in the error.log.
"IQ must be of type 'set' or 'get'" has started coming after disabling PEP through system property 'xmpp.pep.enabled' to 'false'. this error comes throughout the day but it works normally.
"closing connection due to error while processing the message" is the real one. when it starts coming it comes in numbers, and connected users falls to below 1k!
it takes hours to get back to normal. users keep trying to connect, it takes lot of attempts to get through.
Server details:
Openfire v4.0.2
Java version: 1.7.0_95
Memory: 6g out of 8g.
User and roster cache: 50mb(working very fine)
Plugins: REST API, Search, Subscription(for auto accepting)
Any help will be really appreciated. Thanks.
below is the part of error.log (when it just happened recently)
2016.08.02 12:01:27 org.jivesoftware.openfire.handler.IQHandler - Internal server error java.lang.IllegalArgumentException: IQ must be of type 'set' or 'get'. Original IQ: <iq type="result" id="41-5438968" from="dr837052579760263@example.com" to="dv162171057539782@example.com"><pubsub xmlns="http://jabber.org/protocol/pubsub"><subscription jid="dv162171057539782@example.com" subscription="subscribed"><subscribe-options/></subscription></pubsub></iq> at org.xmpp.packet.IQ.createResultIQ(IQ.java:384) at org.jivesoftware.openfire.pep.IQPEPHandler.handleIQ(IQPEPHandler.java:318) at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:66) at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:372) at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:115) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:78) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:69) at org.jivesoftware.openfire.pep.PEPService.send(PEPService.java:325) at org.jivesoftware.openfire.pubsub.NodeSubscription.sendSubscriptionState(NodeSubscription.java:781) at org.jivesoftware.openfire.pubsub.Node.createSubscription(Node.java:2100) at org.jivesoftware.openfire.pubsub.PubSubEngine.subscribeNode(PubSubEngine.java:629) at org.jivesoftware.openfire.pubsub.PubSubEngine.process(PubSubEngine.java:104) at org.jivesoftware.openfire.pep.PEPServiceManager.process(PEPServiceManager.java:215) at org.jivesoftware.openfire.pep.IQPEPHandler.createSubscriptionToPEPService(IQPEPHandler.java:494) at org.jivesoftware.openfire.pep.IQPEPHandler.subscribedToPresence(IQPEPHandler.java:557) at org.jivesoftware.openfire.user.PresenceEventDispatcher.subscribedToPresence(PresenceEventDispatcher.java:130) at org.jivesoftware.openfire.handler.PresenceSubscribeHandler.process(PresenceSubscribeHandler.java:215) at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:176) at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:80) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:88) at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java:360) at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaHandler.java:106) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:277) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:199) at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:181) at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:690) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:407) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:236) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:74) at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63) at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:769) at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:761) at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:703) at java.lang.Thread.run(Thread.java:745) 2016.08.02 12:01:47 org.jivesoftware.openfire.nio.ConnectionHandler - Closing connection due to error while processing message: <iq from="dv910366765752478@example.com/mobile" id="5nbeptqbif" type="get"><ping xmlns="urn:xmpp:ping"/></iq> java.lang.NullPointerException 2016.08.02 12:01:54 org.jivesoftware.openfire.nio.ConnectionHandler - Closing connection due to error while processing message: <presence from="dv1830022707230444@example.com" to="ctjbbymiguxs7mjk2rwcjg@example.com" type="subscribe"/> java.lang.NullPointerException 2016.08.02 12:01:54 org.jivesoftware.openfire.nio.ConnectionHandler - Closing connection due to error while processing message: <presence from="dv1830022707230444@example.com" to="ctjbbymiguxs7mjk2rwcjg@example.com" type="subscribed"/> java.lang.NullPointerException 2016.08.02 12:01:57 org.jivesoftware.openfire.nio.ConnectionHandler - Closing connection due to error while processing message: <presence from="dvtpkcjoi0eciqa4jdv@example.com" to="dv10206155956797886@example.com" type="subscribe"/> java.lang.NullPointerException 2016.08.02 12:01:57 org.jivesoftware.openfire.nio.ConnectionHandler - Closing connection due to error while processing message: <presence from="dvtpkcjoi0eciqa4jdv@example.com" to="dv10206155956797886@example.com" type="subscribed"/> java.lang.NullPointerException 2016.08.02 12:03:14 org.jivesoftware.openfire.nio.ConnectionHandler - Closing connection due to error while processing message: <message from="dv621319528038075@example.com/mobile" id="501d1e7a-498d-47c0-92f2-88d5194e86fb" type="chat" to="dvsupport@example.com"><body>Rahim</body><markable xmlns="urn:xmpp:chat-markers:0"/></message> java.lang.NullPointerException at org.jivesoftware.openfire.net.ClientStanzaHandler.processMessage(ClientStanzaHandler.java:112) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:232) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:199) at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:181) at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:690) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:407) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:236) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:74) at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63) at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:769) at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:761) at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:703) at java.lang.Thread.run(Thread.java:745)