My story takes place in an archipelagus in world X, inhabited by people X.

In the shores of the capital city of that archipelagus, there is a bay of deep, dark waters. It's supposed to be dark and deep enough to feel scary and lovecraftian.

From that bay, people Y simply float up from the darkness, naked, and unconscious in a coma-like state, and are then literally fished with fishing nets and revived by a permanent patrol of glorified elite lifeguards, and are integrated into society X.

Upon being revived, people Y have little or no memory of who they are or where they came from but, those who do, all have the same memory: They came from world Y, and that's all the memory they have, most of the time.

Basically, the darkwater bay is a portal between world X and Y that no one remembers crossing. (There actually is much more to it but it's not exactly relevant to this question)

Most of the story revolves around the relationships between people X and Y, their power dynamics, prejudice, and cultural differences. I will most likely come back here for more questions but my question for today is: How can people Y be visually identified? (Both X and Y are regular humans)

I tought, at first, of some sort of scar from the salt water or sunburns from floating in the sea, or maybe marks from the fishing net (I intend on working on an insult from people X to people Y that compares them to fishes or something), but I'm not sure that's even possible.

How can the experience of floating in the sea (forgot to mention it's a tropical and sunny place) and then being fished by fishing nets physically mark people permanently?

(edit: Another idea I came up while writing this: Since I'm going for a lovecraftian feel, maybe jellyfish scars might be good, because they have tentacles and can somehow inject a toxin that might work as a plot device (such as, the toxin makes them not drown or something), and eventually, some special characters might have specially large scars, that somehow makes everyone go "OH MY GOD THAT SCAR IS SO BIG IT MUST NOT BE A REGULAR JELLYFISH BUT SOME MONSTER FROM THE DEPTHS!", which could work as an "imminent disaster" plot device. I google "jellyfish scars" and they definitely look like something out of lovecraft. Any opinions on jellyfish?)

You want something Lovecraftian?

Above is a lamprey. Below is a healed lamprey wound on a human. When your poor Y-folk are recovered at sea, the fishermen burn off the lampreys that seem to love human flesh. Once they heal, they are permanently covered by circular, toothy-looking scars.

Beware the monsters and their tentacles

Given that they would be long dead if they'd spent more than a few minutes in the deep dark along with my own experience spending hours sailing on salt water. No, the sea itself is not going to scar them directly in the short term. However long term exposure to salt water will cause physical damage.

Depending what the geography is like, both hard corals and barnacle covered rocks will do considerable damage to exposed skin. The sea is full of predators who might take a bite out of a passing body, but if you're looking for lovecraftian, the giant squid sucker scars that many sperm whales carry show that there's definitely something alive down there with tentacles capable of doing significant damage as you pass through. It's entirely reasonable for your people to be coming up with injuries.

You're asking for scars though. Scars are a side effect of healing, you only see scars on old injuries. The people coming out of the water should be recently injured, not carrying old scars.

Any of the regular prejudice justifications can work in this scenario. Perhaps the people of World Y have darker skin, slanted eyes or red hair. They might be taller or shorter or even have slightly different body chemistry which expresses itself as an odor.

It is likely that cause for any scars encountered during the brief/survivable rise to the surface, would also be encountered regularly by the fishermen/lifeguards who save them. That might make for an interesting statement about the true nature of prejudice since the maligning mark on people Y would be a badge of honor, assumed to be earned in noble service when worn by a person of X.

The mark might also be sourced by something on World Y or from the void between worlds which travelers must pass through when going through the portal. Since you are going for a Lovecraftian feel, leave the specific nature of these void dwellers obscure but imply (via surviving memories) that many more people attempt the portal passage than the few who survive the journey.

Mixing all these options together, imagine that fundamentally the people of world Y are asthetically different (maybe pale green skin) which means even the lucky scar free are still victimized. Then allow the void dwellers to venture out from the portal for brief periods of time where the X lifeguards encounter them and earn their badges of honor. And give the Y people memories of loved ones who they never would have left behind but who didn't make it through the portal.

Now all that is left is to choose what you want the scar to look like and then create a void dweller creature that can inflict it. Perhaps a tentacle sucker scar like @Separatrix suggested, but triangular rather than the earth-norm oval. That way you can keep with your alphabetic abbreviations. The scars could look like the capital letter A which should provide the hateful members of the X's with plenty of slurs for their vitriol.

The Y people are exiles. They have committed some criminal act in their home world where execution is banned and imprisonment is very expensive. They are not necessarily considered bad in X - it depends on the crime. Maybe they have refused to fight in the army.

To prevent them ever returning through the portal, they are tattooed on their forehead with the legend "Exile" in their own language. The X language is pictographic and the word for exile is a picture of a beetle. The Ys are therefore nicknamed 'beetles' by the Xs.

Depending on the story, these Y people could have the general look of drowned corpses at sea: bleached skin, slightly bleached hair, a general, incurable, swelling of the abdomen, circled eyes, cold to the touch, a watery gaze, an oily mucuous layer at the mouth. You could add random drooling as a result of the water still in their body, and the inability to perceive flavors in the same way X people do due to having been with salty water in their mouth for far too long. Also, the salt left in their ear channel after drying might give them a natural distortion to hearing if not a partial disability.

While these are not literal scars, they are marks that are distinctive enough to be used for discrimination.

Just make them pale.
Xs sound like they're outdoor a lot, on top of that I pictured them in kinda the carribean or so, meaning that they would have a rather brownish, at least well tanned skin.

Y however come from the depths, maybe Place Y is also less sunny (or even in a cave, who knows) so they can't even become tan.
That would leave them in an unhealthy pale (maybe they can become "normal" [fair-skinned European by complexity] after a while, but will never reach a skin tone that is healthy in Xs environment).
This would make up for a ton of cheap and quick insults.

If you really wanna insult with fish: fish have white flesh, so you can start there