Watch a fantasy movie, what music do you hear? The background music is probably some kind of Wagnerian/Straussian late-romantic orchestral score, something epic and monumental.
What about the characters in the movie, though? What music do they play, listen to? Invariably, some kind of fiddle tune, something vaguely or actually British Isles-ish. Jigs, reels. Fifes and pennywhistles are probably involved. Bagpipes, if the director is feeling especially daring.
To be sure, this makes a certain amount of sense. The biggest fantasy film and TV shows have been either located in England (Harry Potter), or in some kind of pastiche grounded in medieval British history (Lord of the Rings, Game of Thrones). The biggest fantasy game is Dungeons and Dragons, no contest, and its setting, too, is medieval pastiche largely drawing upon the British Isles. Listen to podcasts and streams of D&D games and you'll find them riddled with quasi-British-ish accents.
If you're satisfied with that, there's no reason to go asking further. I've often thought, though, there is zero reason why a "fantasy world" full of literally invented cultures should be full of this admittedly charming but narrow band of musical styles. In fact, given all the folk music I've listened to and how diverse it is even just in Europe, there is every reason to think that a truly fantastical culture would develop a music completely different from anything we're accustomed to.
The Tricky Bit
Escaping the gravitational pull of this stylistic paradigm (or any paradigm, really) is difficult. Hollywood sets audience expectations through a massive distribution of media and those very expectations then feed right back into Hollywood, rewarding the fulfillment of those expecations with hundreds of millions of dollars and punishing thwarted expectations with a flood of "it's not really fantasy" bomb reviews.
This dynamic and how it affects music choice is demonstrated clearly by the production of James Cameron's film, Avatar. Producer Jon Landau brought in ethnomusicologist Dr. Wanda Bryant to work with composer James Horner. The goal was to create a new musical culture for the Na'vi, and to that end, Mr. Horner apparently asked Dr. Bryant to bring him sounds "no one had ever heard before." Tricky work, that, but it was done.
In the end, a great deal of their work was dismissed out of hand by James Cameron. Dr. Bryant goes into more detail in her essay recalling the experience and elaborates more fully on how the commercial needs of making an obscenely expensive blockbuster film impose strict constraints on the creative drives of those involved. While she is gracious, Dr. Bryant's frustration with the process is palpable. Or maybe I'm projecting - you can read her work and decide for yourself here.
Luckily, most artists need not be beholden to such stringent standards. One work which explored the "What if..." of a fantastically imagined culture's music is Judith Weir's Airs from Another Planet.
Judith Weir took a different tack than James Horner and Dr. Wanda Bryant. Rather than looking for something no one has ever heard, she took inspiration from personal heritage (for her, Scottish) and asked, "How might these genres be reimagined?" You can have a listen here: Judith Weir: Airs from Another Planet (1986) - YouTube
In the first movement, "Strathspey and Reel," Weir uses two different strategies to effect this "locally foreign" intent. The strathspey evokes the bagpipe version of the dance, in which florid ornaments decorate the main tones of the tune. While she keeps that genre element more or less intact, the harmony is wide ranging and feels familiar without being so. She also uses the ensemble's greater range and diverse timbres to explore unusual colors - the ensemble plays all together, if not always quite in unison, to create the illusion of a single, rather unusual instrument.
For the reel, Weir uses the group a bit more like an ensemble, although there is still a strong feeling of the group functioning as a single unit, an instrument made of instruments. The piano alternates between providing a scintillating accompanimental figuration and adding color to the melody played by the winds. Unlike the strathspey, the sense of the reel as a melodic genre is more deconstructed. Its characteristic driving rhythm is present, but broken up. It might feel like a reel if you squint your ears a bit. Again, there's a sense of familiarity to the harmonies and melodic contours which echo a memory of something you might have heard once, even if you can't quite place the source.
For composers looking to create a "familiarly foreign" feel, to evoke the Once Upon a Time place, there's plenty of room in between the two extremes presented above. Regardless, the main approaches are probably going to be the same - using something relatively familiar as a baseline, identifying the genre's characteristic harmonic, rhythmic, and melodic features and then asking "What if this, but not?"
The degree to which you can create a sense of distance from the source will depend on how many of the factors you tinker with at once. The difference between a jig in 5/8 as opposed to its typical 6/8 meter will probably only be picked up by careful listeners, though I suspect most people would simply sense something "off" about it. Keeping the usual melodic contours of a jig but using a different harmonic source can result in a striking contrast while your average listener will still, if pressed, probably say "It's like a jig, I guess?" Doing both at the same time might only maintain the barest whiff of its original source.
Interestingly, in one melody I composed which used a slightly different harmonic scheme than a usual Scottish air, one commenter said it sounded "like Scotland sometimes, but also Middle Eastern somehow?" This came from a person with no musical training whatsoever, showing just how difficult it is to match the "doesn't sound like anything anybody has heard before" prompt James Horner put forward for Avatar.
Instrumentation can be an especially strong marker of place. Play some bagpipes, and I'd put money down you think first of Scotland, even though the bagpipes are a commonly used instrument throughout Europe. It would similarly be very difficult to evoke a place other than Japan if a composer used the sho, its distinctive clarity being difficult to mistake for anything else. For Avatar, James Horner even ended up blending the sounds of several drums together in order to create a percussive sound that didn't share obvious characteristics with existing instruments. Nevertheless, finding new approaches to using an instrument can probably create a lot of oppotunities for creating a fantastical sound-place that isn't too far afield for broader audiences.