Message processor runs infinitely without dequing messages - WSO2 ESB

This issue appeared when working with ESB message processors, I had configured the message processor to listen to a ActiveMQ queue which already had messages. I was using ESB 4.9

Issue

Here I face a issue where the Message processor keeps on sending the messages infinitely. The following errors were observed.

ERROR - ForwardingService BlockingMessageSender of message processor [InMEn_Proc2] failed to send message to the endpoint
[2015-07-21 21:55:38,062]  WARN - ForwardingService Failed to send the message through the fault sequence. Sequence name does not Exist.

The reason.

The reason for this issue is the ESB is following the guaranteed delivery precautions, ESB is waiting till the message delivered successfully.

How to get this fixed

The easiest way to fix this is make the ESB ignore the message after sending the message out. You can do this by setting the following properties.

<property name="FORCE_SC_ACCEPTED" value="true" scope="axis2" />
<property name="OUT_ONLY" value="true" />

You need to add these properties before storing the message in the store. Refer the following sample API.

<api xmlns="http://ws.apache.org/ns/synapse" name="api" context="/api">
   <resource methods="POST GET">
      <inSequence>
         <log level="custom">
            <property name="VALUE" value="********* IN GET RESOURCE *********"/>
         </log>
         <property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
         <property name="OUT_ONLY" value="true"/>
         <store messageStore="ActivMQ_Store2"/>
         <send>
            <endpoint key="POSTEP"/>
         </send>
      </inSequence>
   </resource>
</api>

Swagger Definition is not getting added when Creating an API with Publisher APIs in WSO2 API Manager 1.8

When creating a API with Publisher APIs the swagger definition may not get added, actually this is a known issue in Api Manager 1.8 and this is being fixed in Api Manager 1.9 release with the added capabilities of Swagger 2.0.

So How can you get this fixed. Actually its not straight forward to fix this issue, We can follow a workaround instead. The workaround is, after creating the API with Publisher APIs, you can add the Swagger definition separately.  Please follow the following commands to achieve this.

First Lets Create the API

curl -k -X POST -b cookies https://localhost:9443/publisher/site/blocks/item-add/ajax/add.jag -d 'action=addAPI&name=MyNewAPI6&visibility=public&version=1.0&description=My New API created and published by curl&endpointType=secured&http_checked=&https_checked=https&wsdl=&tags=automated published&tier=Gold&thumbUrl=https://pbs.twimg.com/profile_images/493735622587064320/z7qZUG0E_bigger.png&context=/apicurl6&tiersCollection=Gold,Silver,Bronze,Unlimited&resourceCount=5&resourceMethod-0=GET&resourceMethodAuthType-0=Application,Application User&resourceMethodThrottlingTier-0=Unlimited&uriTemplate-0=/assignments&resourceMethod-1=OPTIONS&resourceMethodAuthType-1=None&resourceMethodThrottlingTier-1=Unlimited&uriTemplate-1=/assignments&resourceMethod-2=GET&resourceMethodAuthType-2=Application,Application User&resourceMethodThrottlingTier-2=Unlimited&uriTemplate-2=/journeys&resourceMethod-3=OPTIONS&resourceMethodAuthType-3=None&resourceMethodThrottlingTier-3=Unlimited&uriTemplate-3=/journeys&resourceMethod-4=GET&resourceMethodAuthType-4=Application,Application User&resourceMethodThrottlingTier-4=Unlimited&uriTemplate-4=/info&resourceMethod-5=OPTIONS&resourceMethodAuthType-5=None&resourceMethodThrottlingTier-5=Unlimited&uriTemplate-5=/info' -d 'endpoint_config={"production_endpoints":{"url":"https://internal-mysampleapi.com:9443","config":null}, "sandbox_endpoints": {"url":"https://internal-mysampleapi.com:9443/mock","config":null},"endpoint_type":"http"}'

Now Lets add the Swagger Definition

curl -F name="MyNewAPI6" -F version="1.0" -F provider="admin" -F action="manage" -F swagger='{"apiVersion":"1.0","swaggerVersion":"1.2","authorizations":{"oauth2":{"scopes":[],"type":"oauth2"}},"apis":[{"index":0,"file":{"apiVersion":"1.0","basePath":"http://10.100.5.112:8280/newapi2/1.0","swaggerVersion":"1.2","resourcePath":"/assignments","apis":[{"index":0,"path":"/assignments","operations":[{"nickname":"get_assignments","method":"GET","parameters":[{"dataType":"String","description":"AccessToken","name":"Authorization","allowMultiple":false,"required":true,"paramType":"header"},{"description":"RequestBody","name":"body","allowMultiple":false,"required":true,"type":"string","paramType":"body"}]},{"nickname":"options_assignments","method":"OPTIONS","parameters":[{"dataType":"String","description":"Access Token","name":"Authorization","allowMultiple":false,"required":true,"paramType":"header"},{"description":"RequestBody","name":"body","allowMultiple":false,"required":true,"type":"string","paramType":"body"}]}]}]},"description":"","path":"/assignments"},{"index":1,"file":{"apiVersion":"1.0","basePath":"http://10.100.5.112:8280/newapi2/1.0","swaggerVersion":"1.2","resourcePath":"/info","apis":[{"index":0,"path":"/info","operations":[{"nickname":"get_info","method":"GET","parameters":[{"dataType":"String","description":"AccessToken","name":"Authorization","allowMultiple":false,"required":true,"paramType":"header"},{"description":"Request Body","name":"body","allowMultiple":false,"required":true,"type":"string","paramType":"body"}]},{"nickname":"options_info","method":"OPTIONS","parameters":[{"dataType":"String","description":"AccessToken","name":"Authorization","allowMultiple":false,"required":true,"paramType":"header"},{"description":"Request Body","name":"body","allowMultiple":false,"required":true,"type":"string","paramType":"body"}]}]}]},"description":"","path":"/info"},{"index":2,"file":{"apiVersion":"1.0","basePath":"http://10.100.5.112:8280/newapi2/1.0","swaggerVersion":"1.2","resourcePath":"/journeys","apis":[{"index":0,"path":"/journeys","operations":[{"nickname":"get_journeys","method":"GET","parameters":[{"dataType":"String","description":"AccessToken","name":"Authorization","allowMultiple":false,"required":true,"paramType":"header"},{"description":"Request Body","name":"body","allowMultiple":false,"required":true,"type":"string","paramType":"body"}]},{"nickname":"options_journeys","method":"OPTIONS","parameters":[{"dataType":"String","description":"AccessToken","name":"Authorization","allowMultiple":false,"required":true,"paramType":"header"},{"description":"Request Body","name":"body","allowMultiple":false,"required":true,"type":"string","paramType":"body"}]}]}]},"description":"","path":"/journeys"}],"info":{"title":"MyNewAPI6","termsOfServiceUrl":"","description":"","license":"","contact":"","licenseUrl":""}}' -k -X POST -b cookies https://localhost:9443/publisher/site/blocks/item-design/ajax/add.jag

Here you will have to create the swagger definition manually and parse the object according to the above format, this will add the swagger definition successfully

Another workaround is to create the API with normal Api creation EP. You can follow this log to read about this.

Error when Invoking a API - WSO2 API Manager - { Unexpected character '{' (code 123) in prolog; expected '<'}

Issue

When Invoking a API the following error was Observed.

 ERROR
{org.apache.synapse.transport.passthru.util.RelayUtils} - Error while building Passthrough stream {org.apache.synapse.transport.passthru.util.RelayUtils}
org.apache.axiom.om.OMException: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '{' (code 123) in prolog; expected '<'
at [row,col
{unknown-source}]: [1,1]
at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:296)
at org.apache.axiom.om.impl.llom.OMDocumentImpl.getOMDocumentElement(OMDocumentImpl.java:109)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.getDocumentElement(StAXOMBuilder.java:570)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.getDocumentElement(StAXOMBuilder.java:566)
at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:129)
at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:107)
at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:82)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:68)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:196)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:77)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.rest.Resource.process(Resource.java:297)
at org.apache.synapse.rest.API.process(API.java:297)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:83)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:51)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:220)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:492)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:170)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:228)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '{' (code 123) in prolog; expected '<'
at [row,col {unknown-source}
]: [1,1]
at com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:639)
at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2052)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1134)
at org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.next(XMLStreamReaderWrapper.java:225)
at org.apache.axiom.util.stax.dialect.DisallowDoctypeDeclStreamReaderWrapper.next(DisallowDoctypeDeclStreamReaderWrapper.java:34)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.parserNext(StAXOMBuilder.java:681)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:214)

What to Note.

This issue can only occur under following circumstances.

1. If you have enabled response caching this error can be seen.
2. If you have defined a custom mediation flow to the API. (In the IN or OUT sequence) this can occur.

The reason for this issue.

If you have enabled response caching or if you have any content aware mediators with  the API invocation flow the message will get built when it reaches the GW. So as per the stacktrace the GW fails to build the message successfully. Since the GW is unable to build the message it throws this error.

One of the reason for this is mismatching Content-type header with an mismatching content body. For e.g. Consider the following DEBUG log fragment.

[2015-05-15 16:56:04,827] DEBUG {org.apache.synapse.transport.http.wire} -  >> "Content-Type: text/html[\r][\n]" {org.apache.synapse.transport.http.wire}

[AM] [2015-05-15 16:56:04,830] DEBUG {org.apache.synapse.transport.http.wire} -  >> "{"Name":"sourish"}[\r][\n]" {org.apache.synapse.transport.http.wire}

As you can see above the Content-type header is sent as text/html, but the actual content is a Json {"Name":"sourish"}. So in this case when the GW is trying to build this message this error is thrown

So in order to avoid this issue you can follow one of the options.

1. You can change the back-end service so it sends the correct Content-Type with the response.
2. You can disable response caching and test the scenario.
3. You can add the following message builder and formatter, so the message will be built as plain text. You need to add these into axis2.xml under relevant Builder and Formatter sections. axis2.xml can be found at <AM_HOME>/repository/conf/axis2 directory.

<messageBuilder contentType="text/html" class="org.apache.axis2.format.PlainTextBuilder"/>

<messageFormatter contentType="text/html" class="org.apache.axis2.format.PlainTextFormatter"/>

Error when starting WSO2 servers after changing default ports

Error

The followign error might occur when starting WSO2 servers after changin there default ports.

ERROR

{org.apache.coyote.AbstractProtocol} - Failed to start end point associated with ProtocolHandler ["http-nio-80"] {org.apache.coyote.AbstractProtocol}

java.net.SocketException: Permission denied
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:470)
at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:617)
at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:444)
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1010)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.wso2.carbon.tomcat.internal.CarbonTomcat.startConnectors(CarbonTomcat.java:379)
at org.wso2.carbon.tomcat.ext.transport.ServletTransportManager.startTransports(ServletTransportManager.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

Possible Reason and Solution

You are getting this permission denied exception because the user you are running the instance does not have sufficient privileges (root permissions). All the ports below 1024 are called Privileged Ports and in Linux. Hence they are not allowed to be opened by any non-root user. It is a unix system security feature which prevents malicious users from setting up a malicious service on a well-known service port. To fix this you need to run the wso2carbon server as a root user. But before proceeding  make sure that you have setup the java environment ( setup PATH and JAVA_HOME environmental variables correctly) for the root user.

Error when Executing XPATH 2.0 functions. - WSO2 ESB

Error

When executing XPATH 2.0 functions within the mediation flow of the ESB you can come accross the following exception,

ERROR - SynapseXPath Evaluation of the XPath expression fn:tokenize(syn:get-property('axis2','ContentType'),';')[1] resulted in an error
org.jaxen.UnresolvableException: No Such Function {http://www.w3.org/2005/xpath-functions}:tokenize
 at org.jaxen.SimpleFunctionContext.getFunction(SimpleFunctionContext.java:127)
 at org.apache.synapse.util.xpath.SynapseXPathFunctionContext.getFunction(SynapseXPathFunctionContext.java:93)
 at org.jaxen.ContextSupport.getFunction(ContextSupport.java:242)
 at org.jaxen.Context.getFunction(Context.java:216)
 at org.jaxen.expr.DefaultFunctionCallExpr.evaluate(DefaultFunctionCallExpr.java:172)
 at org.jaxen.expr.DefaultFilterExpr.evaluate(DefaultFilterExpr.java:168)
 at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102)
 at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674)
 at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213)
 at org.jaxen.BaseXPath.evaluate(BaseXPath.java:172)
 at org.apache.synapse.util.xpath.SynapseXPath.stringValueOf(SynapseXPath.java:302)
 at org.apache.synapse.mediators.builtin.PropertyMediator.getResultValue(PropertyMediator.java:299)
 at org.apache.synapse.mediators.builtin.PropertyMediator.mediate(PropertyMediator.java:95)
 at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:71)
 at org.apache.synapse.mediators.filters.InMediator.mediate(InMediator.java:55)
 at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:71)
 at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:114)
 at org.apache.synapse.rest.Resource.process(Resource.java:306)
 at org.apache.synapse.rest.API.process(API.java:308)
 at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:76)
 at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:63)
 at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:182)
 at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:83)
 at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
 at org.apache.axis2.transport.http.util.RESTUtil.invokeAxisEngine(RESTUtil.java:144)
 at org.apache.axis2.transport.http.util.RESTUtil.processXMLRequest(RESTUtil.java:89)
 at org.apache.synapse.transport.nhttp.util.RESTUtil.processPOSTRequest(RESTUtil.java:195)
 at org.apache.synapse.transport.nhttp.ServerWorker.processEntityEnclosingMethod(ServerWorker.java:450)
 at org.apache.synapse.transport.nhttp.ServerWorker.run(ServerWorker.java:275)
 at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

Possible Solution

The above Error can occur if you haven't enabled Xpath 2.0 functionality in WSO2 ESB. So you can enable XAPTH 2.0 by following this post.

If you still get this error check whether you have defined the Namespaces correctly as well.

Error When Starting the Axis2 Server - Error: Could not find or load main class samples.util.SampleAxis2Server

The following error shows up sometimes when starting Axis2 Server.

Error: Could not find or load main class samples.util.SampleAxis2Server

One possible reason for this is permission issues. In-order to overcome this you can start the Axis2 server in the following manner.

                  sudo sh axis2server.sh                           

Or you can login as Super user and start the Axis Server.

WSO2 API Manager - Images are not visible in the Store and Publisher

The Issue.

The thumbnails and images that are uploaded are not shown in the publisher or store.

What to note.

In general store reources are accessed with a cotext with the store prefix as follows

http://localhost:9443/store/<ResourcePath>

But since publisher and store shares the common images which are stores in the registry, when retrieving these resources like thumbnails they are accessed directly fro the registry it self. So the Store and Publisher context it not amended to  the URL.

e.g :
In super Tenant Environment

http://localhost:9443/registry/resource/_system/governance/apimgt/applicationdata/icons/admin/testApi/1.0/icon

In Tenant environment

http://localhost:9443/t/test.com/t/test.com/registry/resource/_system/governance/apimgt/applicationdata/icons/admin-AT-test.com/Test/1.0/icon

Possible Reasons for this issue and possible remedies

Scenario 01

• As mentioned above one possible reason for this issue is when accessing through a reverse proxy there can be some complications and store context can be added by the proxy in some scenarios. Or the resource URL can be changed by the reverse proxy. You can inspect the element and see whats the URL is. Following is a issue I faced, note the URL of the thumbnail.

Then add map the relevent URL pattern from the reverse proxy. Sample Apache2 configs are given below, you can refer the following,

ProxyPass /publisher https://localhost:9443/publisher
ProxyPassReverse /publisher https://localhost:9443/publisher
ProxyPass /carbon https://localhost:9443/carbon
ProxyPassReverse /carbon https://localhost:9443/carbon
ProxyPass /store https://localhost:9443/store
ProxyPassReverse /store https://localhost:9443/store
ProxyPass /registry https://localhost:9443/registry
ProxyPassReverse /registry https://localhost:9443/registry
ProxyPass /t https://localhost:9443/t
ProxyPassReverse /t https://localhost:9443/t

• From API Manager 1.9 some reverse proxy related configs have been provided, you can refer the following public JIRA as well.

- https://wso2.org/jira/browse/APIMANAGER-3412

Scenario 02

• Another possible reason can be authorization issues, when accessing through a LB the domain you are accessing may not be added as a trusted domain, this can happen if there are multiple LBs in place etc. So in order to add the domain as a trusted domain try to access the resource directly via the browser, the browser will prompt and ask whether this needs to be added as a trusted domain and click on add, after adding the domain, you will be able to access these resources successfully.

Hope this will be useful to someone, If you came across some more scenarios please do share, so it may be useful to some else. :)