The move comes at the request of Panama, which has jurisdiction under international law because the crime allegedly occurred in international waters, a justice Ministry spokesman said on condition of anonymity.
Japanese television showed the suspects, Almercito Racela, 36, and Nolieto Pabiona, 41 being escorted off the ship by Japan coast guard officials. They were handcuffed and wearing orange-colored life vests.
The two, who cannot be tried in Japan, were to be transported to the Tokyo High Prosecutors Office for temporary detention before their handover to Panama, said coast guard spokeswoman Mina Azuma.
While Japan awaited Panamas detention request, the suspects were locked in separate rooms on board the ship and guarded by crew after the tanker anchored just off Himeji port on April 10. The two suspects are accused of killing Japanese second mate Izumi Shimba, 52, on April 7.
Japanese coast guard investigators conducted a preliminary probe at Panamas request and sent a report of their findings to the Panamanian Embassy in Tokyo.
The ship, the Tajima, was carrying 253,000 metric tons of crude oil from Saudi Arabia to the western Japanese port of Himeji and was in intentional waters about 55 kilometers east of Taiwan at the time of the alleged crime.
Himeji is 480 kilometers west of Tokyo.