I believe the poor latency is because you are requesting N1QL to perform the sort and limit on all the results returned by the search index as opposed to deferring the sort and limit capability to the search index in which case only the qualifying results will be returned by the search index. Try this query …
SELECT META().id, search_score()
FROM `somesource` AS t
WHERE SEARCH(t,
{"query": {"match": "John Smith", "field": "externalDisplayName"}, "sort": ["-_score"], "size": 100},
{"index": "my-index-name"})