The Cow - 2:125
"We have made the Ka´bah a focal point for the people and a place of safety. Use the seat of Abraham as a place of prayer. "We commissioned Abraham and Ishmael to keep My House clean for the visitors and those who devote themselves to worship and those who bow down and prostrate themselves."
The Ka'bah is the seat of Abraham:
The Family Of Imran/3:96
The first house established for the people was the blessed house in Bakkah, a source of guidance for all peoples.
The Family Of Imran/3:97
There are clear signs there: It is the place of Abraham´s stand (against idolatry). Whoever enters it must be in safety. Those of the people who are able to travel are obliged to pilgrimage to that House (the Ka´bah) for the sake of Allah. Whoever disbelieves, let him know that Allah is not in need of anyone.