I want to start by saying that I was able to solve this error, which I’ll describe below, thanks to David Hill’s post about the same issue. You can find his post on his blog here. His exact solution differed slightly than what was required in my environment so I’ve decided to write a short post of my own.
I ran into this error while doing a fresh install of the vSphere Replication Appliance 6.1. in an environment with an external PSC.
After installing the vSphere Replication Appliance 126.96.36.19990 I logged into the management interface of the appliance (https://FQDN:5480) and filled in the necessary fields on the VR > Configuration screen. While attempting to Save and Restart Service I was faced with this message: Unable to obtain SSL certificate: Bad server response; is a LookupService listening on the given address?
Following the blog post I referenced above I quickly realized the LookupService Address was incorrect. I first tried using the vCenter server address, which was pre-populated in the box, but added the full path for the lookupservice: https://vCenterFQDN:7444/lookupservice/sdk. I was still receiving an error.
I then remembered that in my lab I am not using an embedded Platform Services Controller (PSC), but an external one. I then changed the address to reflect my PSC with success! The correct entry needed to be: https://PSC_FQDN/lookupservice/sdk
As you can see below once I made that change I received the certificate confirmation and was able to save and restart the service.
The strange thing was that looking through VMware documentation I was unable to find this information. If anyone knows where it is listed please feel free to share it.
Thanks for reading!