I've been on the fringe of a few couples that participate in "the lifestyle". Just from my view outside looking in, they all have some problems that arise from the activities in which they participate. Let me give a few examples:
With one couple, the guy would always be talking about how hot the women were that he would be with at the parties that they went to. I could see that this made his wife uncomfortable. He set up meetings for her with other guys, but I got the feeling that inside, she was hurt by the fact that he thought all those other women were where his desires went.
Also knew of a guy that claimed to be a swinger, along with his wife, but she didn't want to do it anymore. They separated, but came back together, and he claimed that she knows he still swings. Funny thing though, he has a few girlfriends that he will only see away from home, and they can't call him at home, and they know he's trying (supposedly) to get things back on track with his wife.
Finally, know a woman that has been involved with a few "lifestyle guys" but only on a one-on-one basis. She's had a FMF threesome or two, and liked it, but was always the third in those instances. She really wants to settle back into a more traditional relationship, and maybe do some exploration with her guy once they get to that point. Her other guys kept encouraging her to do that, but they also keep trying to set up dates with her again so they don't lose their access to her pussy.
Bottom line for me is that in just about all of these situations, people are never really what they seem, and there is a constant flow of lies and backstabbing going on. It seems more about control and manipulation than sex, and although some like to be controlled and manipulated, that person's partner is often the one that is being kept in the dark. Now if you can keep it casual, and move on to different partners before the possessiveness starts, maybe it could work, but there is always that question of whether you are doing it for fun, or to find something that is missing from your primary relationship.