From 1968-71, other sports teams protested playing with BYU, because BYU didn’t have any Black athletes, and there were scandals about Black applicants to the school getting systemically rejected. (You can read all about it in this article Utah Historical Quarterly, Volume 81, Number 3, 2013, pages 204-229, https://issuu.com/utah10/docs/uhq_volume81_2013_number3)
From 1971-75, the IRS went after Bob Jones University for segregation. (https://www.oyez.org/cases/1982/81-3) No doubt there was fear that the same might happen to BYU.
This speech by President Kimball was given in 1975.
The policy banning Black people from the temple and the Priesthood ended in 1978.
The Spencer W. Kimball talk that Elder Holland referenced repeatedly is:
“The Second Century of Brigham Young University,” 1975,
Here are some quotes from Kimball’s talk:
“BYU, in its second century, must become the last remaining bastion of of resistance to the invading ideologies that seek control of curriculum as well as classroom. We do not resist such ideas because we fear them, but because they are false.”
“When the pressures mount for us to follow the false ways of the world, we hope in the years yet future that those who are part of this university and the Church Educational System will not attempt to counsel the board of trustees to follow false ways.”
“No one is more anxious than the Brethren who stand at the head of this Church to receive such guidance as the Lord would give them for the benefit of mankind and for the people of the Church. Thus, it is important to remember what we have in the revelations of the Lord: “And thou shalt not command him who is at thy head, and at the head of the church.”
He also quoted the New Testament:
“The world hath hated them, because they are not of the world, even as I am not of the world.I pray not that thou shouldest take them out of the world, but that thou shouldest keep them from the evil. They are not of the world, even as I am not of the world.”
“We hope that our friends, and even our critics, will understand why we must resist anything that would rob BYU of its basic uniqueness in its second century.”