Church, i.e. “my people” is distinguished from “synagogue” and “places of worship.”
31 But if he repent not he shall not be numbered among my people, that he may not destroy my people, for behold I know my sheep, and they are numbered.
32 Nevertheless, ye shall not cast him out of your synagogues, or your places of worship, for unto such shall ye continue to minister; for ye know not but what they will return and repent, and come unto me with full purpose of heart, and I shall heal them; and ye shall be the means of bringing salvation unto them. (3 Nephi 18:31-32)
Church was different from the synagogue (place where people studied the law) and places of worship.