Error when invoking a secured proxy service WSO2 ESB

Issue

When invoking a secure proxy service you might come accross the following exception.

ERROR - AxisEngine The security token could not be authenticated or authorized; nested exception is: 
 javax.security.auth.callback.UnsupportedCallbackException: Check failed : System error
org.apache.axis2.AxisFault: The security token could not be authenticated or authorized; nested exception is: 
 javax.security.auth.callback.UnsupportedCallbackException: Check failed : System error
 at org.apache.rampart.handler.RampartReceiver.setFaultCodeAndThrowAxisFault(RampartReceiver.java:180)
 at org.apache.rampart.handler.RampartReceiver.invoke(RampartReceiver.java:95)
 at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:340)
 at org.apache.axis2.engine.Phase.invoke(Phase.java:313)
 at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
 at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
 at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.doSOAP(MultitenantMessageReceiver.java:281)
 at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.processRequest(MultitenantMessageReceiver.java:226)
 at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.receive(MultitenantMessageReceiver.java:78)
 at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
 at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395)
 at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142)
 at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ws.security.WSSecurityException: The security token could not be authenticated or authorized; nested exception is: 
 javax.security.auth.callback.UnsupportedCallbackException: Check failed : System error
 at org.apache.ws.security.processor.UsernameTokenProcessor.handleUsernameToken(UsernameTokenProcessor.java:180)
 at org.apache.ws.security.processor.UsernameTokenProcessor.handleToken(UsernameTokenProcessor.java:61)
 at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:332)
 at org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:249)
 at org.apache.rampart.RampartEngine.process(RampartEngine.java:214)
 at org.apache.rampart.handler.RampartReceiver.invoke(RampartReceiver.java:92)
 ... 14 more
Caused by: javax.security.auth.callback.UnsupportedCallbackException: Check failed : System error
 at org.wso2.carbon.security.util.ServicePasswordCallbackHandler.handle(ServicePasswordCallbackHandler.java:112)
 at org.apache.rampart.TokenCallbackHandler.handle(TokenCallbackHandler.java:116)
 at org.apache.ws.security.processor.UsernameTokenProcessor.handleUsernameToken(UsernameTokenProcessor.java:168)
 ... 19 more
[2016-02-09 15:15:53,616] ERROR - ServerWorker Error processing POST reguest for : /services/t/test.com/SimpleProxy.SimpleProxyHttpSoap11Endpoint. Error detail: org.apache.axiom.om.impl.dom.ElementImpl cannot be cast to org.apache.axiom.soap.SOAPHeaderBlock. 
java.lang.ClassCastException: org.apache.axiom.om.impl.dom.ElementImpl cannot be cast to org.apache.axiom.soap.SOAPHeaderBlock
 at org.wso2.carbon.security.pox.POXSecurityHandler.isSOAPWithoutSecHeader(POXSecurityHandler.java:362)
 at org.wso2.carbon.security.pox.POXSecurityHandler.invoke(POXSecurityHandler.java:102)
 at org.apache.axis2.engine.Phase.invokeHandler(Phase.java:340)
 at org.apache.axis2.engine.Phase.invoke(Phase.java:313)
 at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
 at org.apache.axis2.engine.AxisEngine.sendFault(AxisEngine.java:515)
 at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.doSOAP(MultitenantMessageReceiver.java:325)
 at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.processRequest(MultitenantMessageReceiver.java:226)
 at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.receive(MultitenantMessageReceiver.java:78)
 at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
 at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395)
 at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142)
 at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

Possible Reason

One possible reason for this error is that the Authentication details (User Credentials) you are parsing are incorrect. The above exception doesn't give any indication about invalid user credentials so the users might get misleaded.

Remedy

If you see the above exception double check the user credentials you are parsing in the WSS header. Also check the allowed roles and the policy. This might be the root cause of the issue.

Error When Deploying a CAR App to WSO2 ESB

Issue

When deploying a CAR File to WSO2 ESB you might come accross the following exception. In my case I came accross the following issue when uploading a secure CAR app.

ERROR - ProxyServiceDeployer ProxyService Deployment from the file : /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : Failed.
org.apache.synapse.SynapseException: Cannot convert null to a StreamSource
 at org.apache.synapse.config.SynapseConfigUtils.handleException(SynapseConfigUtils.java:578)
 at org.apache.synapse.config.SynapseConfigUtils.getStreamSource(SynapseConfigUtils.java:79)
 at org.apache.synapse.core.axis2.ProxyService.getPolicyFromKey(ProxyService.java:822)
 at org.apache.synapse.core.axis2.ProxyService.buildAxisService(ProxyService.java:608)
 at org.apache.synapse.deployers.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:80)
 at org.wso2.carbon.proxyadmin.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:46)
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:194)
 at org.wso2.carbon.application.deployer.synapse.SynapseAppDeployer.deployArtifacts(SynapseAppDeployer.java:130)
 at org.wso2.carbon.application.deployer.internal.ApplicationManager.deployCarbonApp(ApplicationManager.java:263)
 at org.wso2.carbon.application.deployer.CappAxis2Deployer.deploy(CappAxis2Deployer.java:72)
 at org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
 at org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:807)
 at org.apache.axis2.deployment.repository.util.WSInfoList.update(WSInfoList.java:144)
 at org.apache.axis2.deployment.RepositoryListener.update(RepositoryListener.java:377)
 at org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:254)
 at org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.checkRepository(SchedulerTask.java:59)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.run(SchedulerTask.java:67)
 at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.runAxisDeployment(CarbonDeploymentSchedulerTask.java:93)
 at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:138)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
[2016-02-09 14:23:18,210] ERROR - AbstractSynapseArtifactDeployer Deployment of the Synapse Artifact from file : /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : Failed!
org.apache.synapse.deployers.SynapseArtifactDeploymentException: ProxyService Deployment from the file : /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : Failed.
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.handleSynapseArtifactDeploymentError(AbstractSynapseArtifactDeployer.java:475)
 at org.apache.synapse.deployers.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:112)
 at org.wso2.carbon.proxyadmin.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:46)
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:194)
 at org.wso2.carbon.application.deployer.synapse.SynapseAppDeployer.deployArtifacts(SynapseAppDeployer.java:130)
 at org.wso2.carbon.application.deployer.internal.ApplicationManager.deployCarbonApp(ApplicationManager.java:263)
 at org.wso2.carbon.application.deployer.CappAxis2Deployer.deploy(CappAxis2Deployer.java:72)
 at org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
 at org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:807)
 at org.apache.axis2.deployment.repository.util.WSInfoList.update(WSInfoList.java:144)
 at org.apache.axis2.deployment.RepositoryListener.update(RepositoryListener.java:377)
 at org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:254)
 at org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.checkRepository(SchedulerTask.java:59)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.run(SchedulerTask.java:67)
 at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.runAxisDeployment(CarbonDeploymentSchedulerTask.java:93)
 at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:138)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.synapse.SynapseException: Cannot convert null to a StreamSource
 at org.apache.synapse.config.SynapseConfigUtils.handleException(SynapseConfigUtils.java:578)
 at org.apache.synapse.config.SynapseConfigUtils.getStreamSource(SynapseConfigUtils.java:79)
 at org.apache.synapse.core.axis2.ProxyService.getPolicyFromKey(ProxyService.java:822)
 at org.apache.synapse.core.axis2.ProxyService.buildAxisService(ProxyService.java:608)
 at org.apache.synapse.deployers.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:80)
 ... 22 more
[2016-02-09 14:23:18,211]  INFO - AbstractSynapseArtifactDeployer The file has been backed up into : NO_BACKUP_ON_WORKER.INFO
[2016-02-09 14:23:18,212] ERROR - AbstractSynapseArtifactDeployer Deployment of synapse artifact failed. Error reading /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : ProxyService Deployment from the file : /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : Failed.
org.apache.axis2.deployment.DeploymentException: ProxyService Deployment from the file : /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : Failed.
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:201)
 at org.wso2.carbon.application.deployer.synapse.SynapseAppDeployer.deployArtifacts(SynapseAppDeployer.java:130)
 at org.wso2.carbon.application.deployer.internal.ApplicationManager.deployCarbonApp(ApplicationManager.java:263)
 at org.wso2.carbon.application.deployer.CappAxis2Deployer.deploy(CappAxis2Deployer.java:72)
 at org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
 at org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:807)
 at org.apache.axis2.deployment.repository.util.WSInfoList.update(WSInfoList.java:144)
 at org.apache.axis2.deployment.RepositoryListener.update(RepositoryListener.java:377)
 at org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:254)
 at org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.checkRepository(SchedulerTask.java:59)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.run(SchedulerTask.java:67)
 at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.runAxisDeployment(CarbonDeploymentSchedulerTask.java:93)
 at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:138)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.synapse.deployers.SynapseArtifactDeploymentException: ProxyService Deployment from the file : /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : Failed.
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.handleSynapseArtifactDeploymentError(AbstractSynapseArtifactDeployer.java:475)
 at org.apache.synapse.deployers.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:112)
 at org.wso2.carbon.proxyadmin.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:46)
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:194)
 ... 20 more
Caused by: org.apache.synapse.SynapseException: Cannot convert null to a StreamSource
 at org.apache.synapse.config.SynapseConfigUtils.handleException(SynapseConfigUtils.java:578)
 at org.apache.synapse.config.SynapseConfigUtils.getStreamSource(SynapseConfigUtils.java:79)
 at org.apache.synapse.core.axis2.ProxyService.getPolicyFromKey(ProxyService.java:822)
 at org.apache.synapse.core.axis2.ProxyService.buildAxisService(ProxyService.java:608)
 at org.apache.synapse.deployers.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:80)
 ... 22 more
[2016-02-09 14:23:18,213] ERROR - ApplicationManager Error occurred while deploying Carbon Application
org.apache.axis2.deployment.DeploymentException: ProxyService Deployment from the file : /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : Failed.
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:213)
 at org.wso2.carbon.application.deployer.synapse.SynapseAppDeployer.deployArtifacts(SynapseAppDeployer.java:130)
 at org.wso2.carbon.application.deployer.internal.ApplicationManager.deployCarbonApp(ApplicationManager.java:263)
 at org.wso2.carbon.application.deployer.CappAxis2Deployer.deploy(CappAxis2Deployer.java:72)
 at org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
 at org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:807)
 at org.apache.axis2.deployment.repository.util.WSInfoList.update(WSInfoList.java:144)
 at org.apache.axis2.deployment.RepositoryListener.update(RepositoryListener.java:377)
 at org.apache.axis2.deployment.RepositoryListener.checkServices(RepositoryListener.java:254)
 at org.apache.axis2.deployment.RepositoryListener.startListener(RepositoryListener.java:371)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.checkRepository(SchedulerTask.java:59)
 at org.apache.axis2.deployment.scheduler.SchedulerTask.run(SchedulerTask.java:67)
 at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.runAxisDeployment(CarbonDeploymentSchedulerTask.java:93)
 at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:138)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.axis2.deployment.DeploymentException: ProxyService Deployment from the file : /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : Failed.
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:201)
 ... 20 more
Caused by: org.apache.synapse.deployers.SynapseArtifactDeploymentException: ProxyService Deployment from the file : /home/yasassri/Support/MYSUP/wso2esb-4.9.0/tmp/carbonapps/-1234/1455007998187Sec_CarApp_1.0.0.car/SimpleProxy_1.0.0/SimpleProxy-1.0.0.xml : Failed.
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.handleSynapseArtifactDeploymentError(AbstractSynapseArtifactDeployer.java:475)
 at org.apache.synapse.deployers.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:112)
 at org.wso2.carbon.proxyadmin.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:46)
 at org.apache.synapse.deployers.AbstractSynapseArtifactDeployer.deploy(AbstractSynapseArtifactDeployer.java:194)
 ... 20 more
Caused by: org.apache.synapse.SynapseException: Cannot convert null to a StreamSource
 at org.apache.synapse.config.SynapseConfigUtils.handleException(SynapseConfigUtils.java:578)
 at org.apache.synapse.config.SynapseConfigUtils.getStreamSource(SynapseConfigUtils.java:79)
 at org.apache.synapse.core.axis2.ProxyService.getPolicyFromKey(ProxyService.java:822)
 at org.apache.synapse.core.axis2.ProxyService.buildAxisService(ProxyService.java:608)
 at org.apache.synapse.deployers.ProxyServiceDeployer.deploySynapseArtifact(ProxyServiceDeployer.java:80)
 ... 22 more
[2016-02-09 14:23:18,215]  INFO - ApplicationManager Reverting successfully deployed artifcats in this CApp : Sec_CarApp_1.0.0

Possible Reason

Possible reason for this issue is that by default when you create a registry resource in Dev studio the default server role is set as governance registry. So when you try to upload a such CAR to ESB you might see the above issue.



Remedy

In order to oversome this issue you can change the server profile of the project to ESBProject. To do this follow the instructions below.

1. Create a composite application project (Whic is needed to create a CAR app) and select export as a composite application project option. This will bring up CAR app creation wizard.

2. Now click next and you will see the project selection page. In this expand the Registry resource project created for ESB and set the Server role. Refer the following

After doing the above hopefully it will solve the issue. please drop a comment if you have any queries.