The relationship between the Asylum and people of color was complicated by the time period and the implicit expectation of who should be admitted to the Asylum. This can be seen in the fact that there was only one female patient of color for a very long time and that the staff of color were only given certain jobs. It can also be seen through the lack of records on patient and staff of color. This could be because the jobs that they were doing were not considered interesting enough at the time, since it appears to be mostly domestic work. The lack of information which would be interesting now could be because the Managers or the Superintendent or Physicians thought that nothing of note occurred.
Both women and men of color were employed as "domestics," household workers who did cooking and cleaning, in the Asylum. In November 1823 "Mary a coloured", arrived at the Friends Asylum to take over the duties of another domestic. This likely included work in the kitchen, where Anne Verree was particularly helpful (Second Month, 15th, 1823). It appears that the staff of color did not work as attendants or in other roles associated with the medical aspects of the Asylum, but instead did housework and other jobs necessary to keep the institution running smoothly. These jobs were likely the least skilled of those available at the Asylum.