They call on the WHO to revisit its guidance on Zika, which is linked to serious birth defects.
The International Olympic Committee (IOC) said in May it sees no reason to delay or move the games due to Zika.
The outbreak of the mosquito-borne disease began in Brazil a year ago, but now more than 60 countries and territories have continuing transmission.
While Zika's symptoms are mild, in the letter the experts say it causes babies to be born with abnormally small heads and may also cause a rare and sometimes fatal neurological syndrome in adults.
The letter is signed by 150 international scientists, doctors and medical ethicists from such institutions as Oxford University and Harvard and Yale universities in the United States.
They cite the failure of a mosquito-eradication programme in Brazil, and the country's "weakened" health system as reasons to postpone or move the Olympics in "the name of public health".
"An unnecessary risk is posed when 500,000 foreign tourists from all countries attend the Games, potentially acquire that strain, and return home to places where it can become endemic," the letter says.
The biggest risk, it adds, is if athletes contracted the virus and returned home to poor countries that had not yet suffered a Zika outbreak.
They also express concern the WHO has a conflict of interest through its partnership with the IOC.
The Rio Olympics take place between 5-21 August.
The WHO, which has declared the Zika virus a global public health emergency, is yet to comment on the letter.
Several public health experts had previously warned that having hundreds of thousands of people arriving in Rio will speed up Zika's spread and lead to the births of brain-damaged babies.
But on Thursday, the head of the US Centers for Disease Control and Prevention, whose advice is quoted approvingly in the letter, said the threat did not warrant halting the games.