Platforms such as Facebook and Google support targeted advertising, promising good value for advertisers. However, multiple studies have shown that such platforms can create skewed outcomes: their ad-delivery algorithms can produce outcomes that are skewed by gender or race, sometimes due to hidden factors not explicitly requested by the advertiser. In this work, we focus on developing a methodology for measuring potential skew in the delivery of job advertisements. A challenge in studying skew of job ads is distinguishing between skew due to a difference in qualification among the underlying audience and skew due to other factors such as the ad platform’s optimization for engagement or changes in the on-line audience. Our work provides a novel methodology that controls for job qualification and audience with paired, concurrent ads and careful statistical tests. We apply our algorithm to two prominent platforms for job ads: Facebook and LinkedIn, confirming skew by gender in Facebook’s results and failing to find skew in LinkedIn’s results. We further examine LinkedIn and show that they do not optimize on the professional background of users unless explicitly requested by the ad purchaser—a choice that reduces the possibility of discriminatory outcomes. Finally, we suggest improvements ad platforms could introduce to make external auditing more efficient and accurate.