Are oil based lubricants safe to use with LATEX FREE condoms?

I have a lot of allergies and have been thinking of using something natural like coconut or olive oil as lubricant for postpartum. I am also allergic to latex (which you can't use with oil based lubricants anyway). Will oil break a latex free condom?