A White House official described the South Korean president as hopeful that there would be "some talks and socializing" between Pence and the North Koreans over the course of the weekend.
Pence is leading the USA delegation to the opening ceremonies, where he has worked to pour cold water on warming ties between the two Koreas.
Pence did not shake hands with Kim, he added, but Japanese Prime Minister Shinzo Abe did, exchanging a few words with him.
But Pence's time sidestepping the North Koreans at the Olympics was punctuated by awkward, stoic photos and no-shows.
The pair will have lunch with Moon on Saturday, Moon's office said. They also made plans to attend a concert together on Sunday, to see a North Korean orchestra perform in Seoul.
White House officials and other critics believe North Korea is trying to use the worldwide stage to gain sympathy and ease punishing economic sanctions and diplomatic pressure that are beginning to bite.
Providing fuel to the ferry would have been a hard decision for Seoul because of concerns that North Korea is trying to use the Olympics to poke holes in worldwide sanctions against the country over its nuclear and missile programs.
In South Korea and beyond, commentators are taken with Kim's surprisingly stylish and relaxed demeanor.
North Korean senior officials Kim Ki Nam and Kim Yang Gon also visited the Blue House in 2009 and met with Roh's conservative successor Lee Myung-bak after leading a delegation to the South to pay respects at the funeral of former liberal South Korean President Kim Dae-jung, who held the first inter-Korean summit with Kim Jong Il in 2000.
At a Friday VIP reception for delegation leaders, Pence arrived late and stayed for just five minutes - and did not interact with the delegation from the North.
A large delegation from North Korea, dressed in identical garb and cheering in careful coordination, watched from an upper deck of the stadium.
Moon's office said he told Abe that the talks between the Koreas would not disrupt global efforts to deal with North Korea's nuclear program. International Olympic Committee President Thomas Bach then handed the podium to Moon, who declared the Olympics officially open.
Moon said improved relations between the Koreas may be a step toward resolving the nuclear standoff.
"What can not be overlooked is that Moon did not directly demand North Korea to abandon its nuclear development".
Moon apparently cited the conditions necessary for an inter-Korean summit. "He sent the country's nominal head of state and also his sister Kim Yo Jong to Seoul and they would need to have something to show for it when they return home", Koh Yu-hwan, a North Korean expert at Seoul's Dongguk University and a security adviser to Moon, said.
To help prevent this outcome, and with Kim refusing to discuss denuclearisation with his "brethren" in the South (at whom he claims his missiles are not aimed), Moon now must figure out how to build up the intra-Korea dialogue to enable talks between North Korea and the US.
The vice president's visit was also marked by bad blood between Pence and some US athletes. "But the Olympic moment that President Moon is trying to generate here is not a time to nurse those grievances", Jannuzi said. For example, the sanctions imposed in September 2017 on textiles, coal, and other exports are said to have reduced North Korean exports by 90 per cent. The night before, Kim and other North Korean delegates attended the opening ceremony of the Olympics, watching a "unified" Korean team march under a banner showing an undivided Korean Peninsula. Sitting on her left were Do Jong-whan, minister of culture, sports and tourism, and Unification Minister Cho Myoung-gyon. He had been dispatched from Washington for the Olympics in part, he said, to make sure that people didn't lose sight of how the US government perceives the North - as a misbehaving and risky neighbor in the community of nations.