Is Inclusion Always the Best Choice?

As many know my youngest child Omega was born with Down Syndrome. We chose not to have him participate as we did not feel that to have countless people in and out of our home would be beneficial to our family. This may or may not have been the right choice at the time, either way we stuck by our decision. Omega is almost 4 years old now and has started a special education program at my children's elementary school this past week that has been fully integrated with the atypical preschool classes (yes more than one class). I am a strong advocate of inclusion but not at the cost of my son's education and care.
I have never been one to exclude my son from things we do as a family. We go camping, Omega is with us. We go to the drive in, Omega is with us. He is no less equal than my atypical children, and to be honest his having Down Syndrome most times is a non issue. To many this may seem like an issue, that he should get more attention, we should coddle him through every life experience no matter how big or small. My husband and I have included him since birth but now I have given over half of my control to a school whose system is falling.
There are parents out there who would love for their special needs child to be in such an inclusive program as Omega is in but in this, I would like a little exclusivity. This is his first introduction to being away from me or my husband, in any group setting away from his siblings and their friends (especially one this large), Am I a bad parent to unenroll him all together? My only alternative would be to pull him from one school and put him in another. He goes to the same school as my atypical children, which any parent knows to have all your children in the same school is a blessing in itself. I am almost leaning towards the benefit outweighing the risk of putting him somewhere else.
As parents we have to make tough decisions on a daily basis. Once you live in a certain school district you don't want to have to think twice about the school your kids are attending, but now more than ever the individual school needs just as much consideration as the school district. All schools are NOT created equal. I could go on for hours about why one school in a district is different than another but we don't have time for that. We will just write it off as disenfranchisement of our kids. Depending on the school your children go to, someone's child will be left behind.