The first thing to note is that “and” is a stop word, removed by both the “en” and “standard” analyzers. So I’m assuming you’ve already adjusted the mapping to use a different analyzer.
Why is custom sort so much slower than _score or _id? The reason is that for every hit matching the search (ie, all 75k) we have to load (often from disk) additional informational we don’t have (in this case the timestamp). Even though you only want the top 100, we have to do these loads on all 75k matches to find the top 100 timestamp values. In contrast, for both _score and _id, we already have this information readily available, and no additional loads are required.
The only tip to make it faster is to try and reduce the number of documents matching the search (ie, the 75k not the 100). In your example, if you already knew that you would get more than 100 hits from “today”, you could add an additional search clause to filter the matches by timestamp as well. This would reduce the number hits that we have to load the timestamp from.
Lucene has two capabilities FieldCache and IndexDocValues that are used to make this faster. I’m not exactly sure how ES uses these without some explicit configuration (automatically building them for all fields would seem to waste RAM and not be what you want).
A year ago we decided to stay with elastic search because fast sort by date was too important for us. Now I was testing fts again. First with moss storage, with the same result as a year ago. After a hint from @sreeks I tested the same with scorch, and… The results are amazing. More than 10 times faster than moss for this sort queries, and comparable with ES for my test set of now 200k results. So for everyone who needs custom sort, scorch is definitely worth to try.
@mschoch, do you have some Infos how much the overhead of custom sort now is, compared to native _id or _score? There could still be the idea of a workaround by using a timestamp as _id that allows the correct sort order if that’s faster and more cpu friendly.