Error when Invoking a API - WSO2 API Manager - { Unexpected character '{' (code 123) in prolog; expected '<'}

Issue

When Invoking a API the following error was Observed.

 ERROR
{org.apache.synapse.transport.passthru.util.RelayUtils} - Error while building Passthrough stream {org.apache.synapse.transport.passthru.util.RelayUtils}
org.apache.axiom.om.OMException: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '{' (code 123) in prolog; expected '<'
at [row,col
{unknown-source}]: [1,1]
at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:296)
at org.apache.axiom.om.impl.llom.OMDocumentImpl.getOMDocumentElement(OMDocumentImpl.java:109)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.getDocumentElement(StAXOMBuilder.java:570)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.getDocumentElement(StAXOMBuilder.java:566)
at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:129)
at org.apache.synapse.transport.passthru.util.RelayUtils.builldMessage(RelayUtils.java:107)
at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:82)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:68)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:196)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:77)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.rest.Resource.process(Resource.java:297)
at org.apache.synapse.rest.API.process(API.java:297)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:83)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:51)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:220)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:492)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:170)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:228)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '{' (code 123) in prolog; expected '<'
at [row,col {unknown-source}
]: [1,1]
at com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:639)
at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2052)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1134)
at org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.next(XMLStreamReaderWrapper.java:225)
at org.apache.axiom.util.stax.dialect.DisallowDoctypeDeclStreamReaderWrapper.next(DisallowDoctypeDeclStreamReaderWrapper.java:34)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.parserNext(StAXOMBuilder.java:681)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:214)

What to Note.

This issue can only occur under following circumstances.

1. If you have enabled response caching this error can be seen.
2. If you have defined a custom mediation flow to the API. (In the IN or OUT sequence) this can occur.

The reason for this issue.

If you have enabled response caching or if you have any content aware mediators with  the API invocation flow the message will get built when it reaches the GW. So as per the stacktrace the GW fails to build the message successfully. Since the GW is unable to build the message it throws this error.

One of the reason for this is mismatching Content-type header with an mismatching content body. For e.g. Consider the following DEBUG log fragment.

[2015-05-15 16:56:04,827] DEBUG {org.apache.synapse.transport.http.wire} -  >> "Content-Type: text/html[\r][\n]" {org.apache.synapse.transport.http.wire}

[AM] [2015-05-15 16:56:04,830] DEBUG {org.apache.synapse.transport.http.wire} -  >> "{"Name":"sourish"}[\r][\n]" {org.apache.synapse.transport.http.wire}

As you can see above the Content-type header is sent as text/html, but the actual content is a Json {"Name":"sourish"}. So in this case when the GW is trying to build this message this error is thrown

So in order to avoid this issue you can follow one of the options.

1. You can change the back-end service so it sends the correct Content-Type with the response.
2. You can disable response caching and test the scenario.
3. You can add the following message builder and formatter, so the message will be built as plain text. You need to add these into axis2.xml under relevant Builder and Formatter sections. axis2.xml can be found at <AM_HOME>/repository/conf/axis2 directory.

<messageBuilder contentType="text/html" class="org.apache.axis2.format.PlainTextBuilder"/>

<messageFormatter contentType="text/html" class="org.apache.axis2.format.PlainTextFormatter"/>