Spanish vs Swedish vs Arabic

I love learning languages. I love listening to new languages, hearing the different sounds, that melodies, the intonation. And I'm fascinated by people's abilities to master 4 or 5 different languages, and switch between them just like that.

But. There's always a but. WHY on earth do people at my job, and I mean co-workers, speak Spanish to each other when they know I don't understand? It's like in kindergarden when people talked behind each others backs although it wasn't behind each others' backs because the art of whispering wasn't fully mastered. I mean come on! And I'm being serious here. Today I worked with two ladies. One in her fifties and one in her early thirties. They're both really good Swedish speakers although it's not their native language. And still, when the three of us were in a room together, sometimes they'd just skip to Spanish - mid sentence. Like I didn't hear. Like I wouldn't understand that the rest of the sentence probably wasn't meant for my ears. I think it's incredibly rude. I don't care if they speak Spanish to each other in the staff room or wherever, but when we're in charge of 10 people together, and I know that speaking Swedish isn't at all a problem, it pisses me off. Am I unreasonable or should it annoy you too?