You can only remove an array element by index - e.g. REMOVE("array[2]")
If you want to be able to remove elements by value you need to use a dictionary - for example you could instead model your set by a dictionary where the elements are keys (and just use empty values).
That would allow you to remove by key.