It could be. I interpreted it during the show to mean that Doreah was more flexible with what she wears, being that she isn’t Dothraki. Irri and the others would naturally be more comfortable in their regular clothes, whereas Doreah was probably used to very different clothes in Lys.
But yeah, Xaro could’ve given her the dress. It makes sense.