IARC evaluated passive smoking (also known as involuntary smoking or environmental tobacco smoke) in a Monograph (no 83) in 2004.

A bit of history: they had previously classified direct smoking as carcinogenic in Monograph 38 in 1986.  In Supplement 7 in 1987 they said “…the aggregate evidence … leads to the conclusion that passive smoking does carry some risk for lung cancer” but that does not seem to have been a formal classification.

They said the epidemiological evidence was sufficient to show carcinogenicity in humans (the strongest evidence is a 20-30% increase in lung cancer in non-smoking spouses of smokers).  They therefore said that passive smoking is a class 1 or established carcinogen. (see an explanation of how the IARC rules work.)

For magnetic fields, they said the epidemiological evidence was "limited" and for electric fields "insufficient".  So according to IARC, the epidemiology for passive smoking is stronger than for EMFs. (see details of the IARC classification of EMFs)

Some people differ and say that the epidemiology for magnetic fields and childhood leukaemia is about as strong as for passive smoking and cancer.  On that view, we accept passive smoking as an established carcinogen because it has a biological plausibilty - we know that active smoking causes cancer so we're more inclined to believe that passive smoking does as well.  Whereas, we don't have that same biological plausibility for magnetic fields so we don't accept it as an established carcinogen.  If you take this view, the comparison is a nice example of how we make up our minds from the totality of the evidence - other strands of evidence influence the conclusions we draw from the epidemiology alone.