Accessibility

Transcript for October 2012 Brown Bag

>> Over the last maybe couple years. This is information that I actually presented earlier this year at what's called VECAP, Vocational Evaluation and Career Assessment Professionals conference, that took place over spring break. So I've had a little time to forget this, but I think I've mostly got it back. And that's why some of the slides -- you all know who I am. You know what my credentials are and stuff like that. but I was just too lazy to take it off the slides for this presentation. So I'm kind of reusing the same slides. And the topic is -- Chuck's message that went around, it kind of relates to disability. It relates to employment, it relates to higher ed, which is one of the things I think is really nice. But those three things being kind of focal areas for what we do within our work and that's a way to kind of bring together some of those areas of focus that we spend a lot of our time zeroing in on here. Let's see, what else can I tell you as kind of background? As the slide said, and really kind of the title said, I'm really kind of focusing in on, you know, participants in the state VR program. Almost everyone here is familiar with state VR. There's a couple who aren't.

 

back to top

 

So forgive me if I start tell you something you already know. I'll kind of try to bring other folks up to speed a little bit as we go along. Let's see here. So the data is really based on participants in what's called the state federal VR program. VR meaning vocational rehabilitation agencies that exist throughout the country whose kind of primary mission is to work with folks with disabilities and kind of get them employed, or help them become employed. Whether that's getting them to work for the first time, so growing up with a disability, or somebody who has incurred a disability and needs to go back to work. Or maybe even has incurred disability and simply wants to keep their job. That's sort of the main focus of the state VR agencies. They are housed under the US Department of Education. And under that what's called OSERS, Office of Special Education and Rehabilitative Services. And under that is the Rehab Services Administration.

 

back to top

 

And they sort of monitor and supervise all of the state VR agencies. And the state VR agencies are housed -- there's at least one in every state and territory, like DC, Porto Rico, Guam. Yeah, John has been to most of those places, so you can ask him questions about all those places. And a lot of states, some states have just one agency, like California's Department of Rehabilitation. A lot of states have -- well some states have more than one. And when they do, they have two agencies typically. So an agency that provides services for folks who are blind, and then an agency that kind of serves everybody else. And so the data kind of comes from these agencies. And it's nationwide obviously, so it's quite a scope. And you'll see as we kind of get into the presentation, it takes a lot of folks upon which this data is based. Folks who come into the state VR agencies for services can get a whole variety of services designed to kind of help them adjust to the disability, deal with it and they get a variety of employment-focused services as well. And later on in the presentation we'll kind of show you some of the services that folks can access through the agency. And these agencies are primarily evaluated on what? What do they do?

[ Inaudible ]

Right, the number of people who are put into employment, right? So there's a variety of outcomes we could look at. What the agencies normally look at is how many people get to work. So one of the things that we're really kind of looking at is what happens to folks who get served by the agency?

 

back to top

 

And later on we'll see I really kind of zeroed in on folks who get college as part of those services through the agency. Kind of more broadly than just the college-age group, there's been some research done on what are the things that people get or characteristics that folks might have that would lead them to the kinds of outcomes or that would contribute to the kinds of outcomes the agency is evaluated on in getting back to work? And kind of sifting through that literature there's a few different kind of services or characteristics that seem to kind of be reliably associated with getting back to work. And those are -- I apologize for the screen being a little washed-out here. I'll read you. Job placement services, okay? Typically services focused towards the end of their case to kind of help them look for employment. Education level at application. Generally kind of the higher the level of education someone has on the way in, that contributes to better outcomes, or that's associated with better outcomes. And total case expenditures, meaning how much money the person gets spent on them while they were getting their services. One thing that I thought was really interesting and kind of piqued my interest in kind of pursuing these questions was I would assume that folks that get college, maybe it's just because of where they work and the environment. That would really kind of up their chances of becoming employed at the end of their rehab phase. And actually you can find some data that supports that, but you can actually find some data that kind of runs counter to that as well. And when you delve into it a little bit more, it probably makes a little bit more sense.

 

back to top

 

And we'll get to that point. But when I say mixed findings regarding college training, there were some studies that found that okay, some folks come into the vocational rehab agency and they get college training. A community college or a four-year institution, and they may be actually more likely to be what they call closed unsuccessfully, meaning they weren't employed at the end of their case, than successfully, and that seems kind of strange to me anyway. So okay, those studies were a few years old. I would take a look at, you know, how that's playing out now. So that's one question that will get answered kind of partially as we go along. We can think about you know in terms of the kind of investment that folks make, both maybe from the counselor who's working with the person with a disability, and the person with the disability themselves. My sense is it can be useful to look at this sort of thing.

 

back to top

 

Because just think about you know, specifically looking at the expense involved. It's not a cheap service. It's not something that's cheap to provide. And you'll see some data regarding that as we go along. It also takes a lot of time and effort, okay? It takes a lot of time to go through school, and it takes effort on both the councilor's part and the client's part. They're the one in the classroom every day studying. There's a lot of effort there. And in a lot of cases the folks who are getting services through the VR agency share in those costs. The agency doesn't pay for everything usually. And so the client is thinking it's an excellent investment as well as the person in the VR agency, the VR counsellor. Any questions so far? So kind of the research questions I focused in on with that will hopefully get answered at least partly as we go along. For people who receive college training, you know people who come into the VR agency and get college training as part of their rehab plan.

 

back to top

 

[ Inaudible ]

It's fine. It's fine. Totally dark mic. We have a webcam on. There are folks that are participating via the web and they may not be able to see if I am standing totally in the dark.

>> Can they tell you?

>> Chet might be able to see.

>> Well, I can kind of see a silhouette of you.

[ Inaudible ]

>> You wouldn't be able to see me necessarily, but they could identify, yes, with the dog. Okay. So kind of the key questions here. What are the average case expenditures? And we'll kind of look at that. And what I mean is how much money spent on a case. And we can kind of look at it with the folks that got college training and kind of compare that with as a point of reference folks who didn't get college training as part of their voc rehab plan. How long did the cases last? How long is somebody in a rehab plan? You know, how much of their lives did they spend kind of going through that process? And again, we can look at that from the standpoint of folks who get college training and folks who don't get that as part of their plan. And what proportion are competitively employed? Okay, who's got a job at the end of their rehab plan? And all the data that we have is on plans that are closed, cases that are closed. So that's kind of a key piece. And you should sure hope that if people go through the time and effort and expense to go through college that they have a pretty decent shot at getting employed when they're done. So we'll have a chance to find that out and kind of get a response to that question or to the previous literature that says, you know, over half of the folks who get college training aren't employed when their case is closed.

 

back to top

 

And again we can compare that to folks who don't get college training as part of their plan. And then finally as kind of a more involved analysis, people can get a variety of services as part of their plan when they get college, or if they're clients who don't get college. And what it's kind of looking at is, are there elements of those services, or services people get or don't get that seem to be kind of reliably associated with being employed at the end of the plan? And so what kind of services seem to be kind of paired up with people who are successfully employed at the end of their time. So these are kind of the four questions that I'm hoping we can kind of sort through in the time that we have. And the first three, we can kind of answer with some descriptive statistics that aren't too terrible difficult or complicated I hope. The last one gets a little more hairy and I'll try to explain it in a way that's fairly easy to understand.

 

back to top

 

Okay, so just a little bit of background on where does the data come from that I'm using. It's called RSA 911 data. It was called this actually before 9/11, so no connection to those two things. Some folks are probably familiar with this a little bit, I would imagine. All the state agencies at the end of every fiscal year, that's RSA, Rehab Services Administration, the organization that kind of has oversight over all the state and federal rehab programs, all the state agencies report back to RSA at the end of every federal fiscal year on all the cases that were closed during that year. And a case can be open for a very short amount of time. Like some of these cases can be opened and closed in a matter of days or weeks. And cases can be open for incredibly long amounts of time. You know, we can find cases that have been open for decades. Yeah, yeah. So what we're looking at is all cases closed here. All cases closed during federal fiscal year 2009. And that will include people whose cases were closed very recently and people whose cases were opened like in the 90's probably, or potentially in the 80's a few of them, not most of them. And we'll get kind of an idea for the average time a case is open as we go on. And as you can see, there's 588,970 cases for fiscal year 2009.

 

back to top

 

That's a lot. You know, that's a huge amount of data to work with. So in some ways it's really kind of interesting and attractive to look at because there's so many folks in this data set. That's a lot. I mean, you don't run into data sets that big very often. By the same token, there are some drawbacks that we'll talk through as we go along as well. Data is gathered by somebody else, not by me. I can't say exactly what questions I would like to have answered. I can't phrase them exactly the way I would like them asked and have them phrased. So you kind of have to work with the data the way that it is. So sometimes you can't answer exactly the question you want to answer, but you can kind of get close. Then at the end we can talk about what we would really like to have. So tons of cases here. I'm going to try to not goof up the webinar here online too much. But just to kind of show you how the data -- the department of ed makes this data available to us. But it comes to us looking like this. It's a text file. They can't send it to us by email or anything like that because it's too big.

 

back to top

 

They have to mail it to us on a CD. And each of these lines -- well first of all each row represents a case, okay? So there's 588,970 of these in this particular file. And I don't want to do this because I'll goof up the webinar. But if I were to start scrolling down on it and start talking to you I could probably hold my finger on the scroll down thing, talk for the rest of the brown bag and we wouldn't even be close to being done. There's just a ton of data in there. But we have to take it, and Chuck, you've seen this data a little bit too. And figure out a way to kind of make sense out of it. And RSA does nicely provide a 60- or 70-page manual on exactly what each of these characters means. So there's a fair amount of work that goes into getting it ready to be analyzed. But once you do it then there's some autonomy and next year you don't have to start from scratch again. So I typically request these from RSA every year and it gets a little bit easier every year. The first time around it's a doozy. So each of these characters, each of these lines tells us something about the person. And this tells all kinds of stuff like, you know, what state they were served in, what day they applied, what day their case was closed, what level of education they had on the way in, what level of education they had on the way out.

 

back to top

 

Were they making any money on the way in? And on the way out? What benefits they were accessing on the way in and the way out. What services they got, all kinds of stuff in here, even though it looks like a bunch of kind of nonsense. And before RSA makes this available it kind of researches the field. Some of the things they do are they strip off some identifying information. So the stuff goes back to RSA from the state agencies with social security numbers or names or birthdates or things like that. In this field they're all zeroes. That used to have information in it but they took it out so that we couldn't identify that person. So if I had an evil purpose, I'd try to figure out who a particular person was, wanted to kind of get into their privacy, this will make it a lot harder for me. So they're trying to protect the confidentiality of folks. And typically the stuff they're removing isn't critical to the kind of questions we would want to answer anyway. Okay, so let's see if we can get that down. All right, I didn't mess it up, yet. Okay, so pretty robust data set, lot of folks in it.

 

back to top

 

Kind of the key things that I want to look at, or the key things that we start to see in the analyses that would be sort of important features of this or kind of the major outcome I was looking at was competitive employment. So the VRR agencies are evaluated on the number of folks who go back to work and the proportion of those who go back to work. And they track it in a few different ways. If you're in the VR system then you probably know what is successfully rehabilitated and that's one indication of kind of a successful rehab or successful employment outcome. However, the 26 can include people who go back to work, but it can also include people who get other kinds of training and don't necessarily wind up in work, right? Somebody is coming in and they want to become more independent at home. State rehab agencies can provide some training to do that. And if the person leaves more independent than when they came in, that can be a successful closure. Because I was focusing on college training, my senses or my gut was we don't need college training quite so much for those types of outcomes. So there's another outcome that's tracked in the data set that's called competitive employment. And that actually means working at a job and an integrated environment, meaning you can't just take a whole bunch of folks with disabilities and put them all in a room by themselves and say, "Okay, you're all employed." They have to be working in an integrated environment.

 

back to top

 

People with disabilities working alongside people without disabilities. And it's got to be at minimum wage or above in order to kind of count as competitive employment, okay? And typically for a person with college training that's probably more what we're shooting for anyway, right? Okay, so that's sort of what I use for my outcome variables. You can argue with me about that. You could look at wages or any other, a few different outcomes in the data set. This is the one that I sort of zeroed in on. One thing you might want to look at is the total cost of purchased services. So the amount of money spent on each case is reported back to RSA. That's kind of nice. It would be really nice if they said how much money they spent on all the different services. We don't get that. We get one lump sum back. So I wanted to look at that because that was one that showed up as sort of one of the predictors that has kind of showed up reliably in the literature as a predictor of outcomes. I want to look at case duration, how long the cases last. And that requires a little bit of work because they don't really tell you how long the cases are. You have to do a little bit of math. When did the person apply for services? When did their case close? And you can kind of do the math that way. So I did a little bit of subtraction so we have case duration in days.

 

back to top

 

One of the things that gets asked of folks and reported back to RSA is kind of what level they came in at education-wise on the way in and then again on the way out. And it's tracked on kind of an ordinal, no school, elementary education, you know, middle school, high school, that kind of a thing, diploma, GED. Some college, completed a bachelor's degree, completed a graduate degree, that kind of a thing. And again, we can identify people who would get college and kind of gain degrees by using where were they on the way in and where were they on the way out. And then we'll see a listing here. There's 21 different services that are tracked and they're kind of tracked in more or less of a yes/no. Did the person get it or did they not get it, kind of a format. There's a little bit more to it, like who provided it, who paid for it. But for our analysis here it almost works on a did they get it or did they not get it kind of as a variable. And those are going to be important things in kind of looking at the outcomes. These are kind of the key variables that really factored into the analysis. You doing okay? All right. You're all pretty quiet. Little lunchtime nap? Yes?

 

back to top

 

[ Inaudible ]

There is a variable. There's a few variables in the data set that capture disability. So they capture like the type of disability and the cause of the disability. And they may even capture, I think they may capture secondary and tertiary disabilities. So those are in there. I didn't necessarily look at those in this kind of analysis, but you could certainly extend this by factoring something like that into it. Or if you have additional questions that you think could be answered by this data set and disability type is important to you, then that is in there. You can easily pull out everybody with a visual disability in California if you wanted to, for example. Go into HMI. Okay, so kind of going through what we found in terms of these research questions. First of all I just kind of want to show you how this idea of college training plays out. And so if you were to look at who gets college as part of your plan, if you remember there were 588,970 folks. Question?

 

back to top

 

>> It's from the chat room. When we get questions, can you repeat the question? They can't hear it.

>> Okay. If we get a question, okay got it. So the previous question was whether disability was represented in the data set. And the short answer was yes. We're coming back to this slide, who gets college and who doesn't. And there's 588,970 folks in the data set for this particular year. And that's pretty reliable year to year. There's between 500,000 and 600,000 almost every year. Out of that, who got college training and who didn't? 542,000 roughly did not. Roughly 46,000, 47,000 did. So about 7.9 percent of the folks have college training reflected in their plan. And then furthermore because of some of the analysis we'll get to later, I wanted to sort of identify groups that not only got college training, but got it at a level which they sort of advanced their education to completion of a degree, to an AA or to a bachelor's degree or to a graduate degree. And for somebody with an AA, we kind of figured that out by saying, "Okay let's take all the folks who got college training and came in with less than an AA and left with an AA." So during their rehab plan they could have advanced their college education to that level. The we do the same thing with bachelor's degrees and the same thing with graduate degrees. And so about 5,200 folks advanced to an AA. 6,800 advanced to an undergraduate degree. 1,200-1,300 advanced to a -- did I say undergraduate? Yeah. 1,200-1,300 advance to a graduate degree. Yeah?

 

back to top

 

[ Inaudible ]

No, these are the same groups. This would be who came in with less than an associates.

>> Can you repeat it?

>> Oh yeah. The question was along the lines of, and correct me if I'm wrong here, are the people who got the AA if they then continued on to a bachelor's degree, were they reflected in both groups?

>> Right.

>> Yeah. And the short answer is no. And typically the reason why is we said, "Okay, tell us all the people who came in with less than an AA and left with AA as their highest level. And then all the people who came in with less than a bachelor's degree and left with bachelor's as their highest level." And same thing for graduate degree. Yes?

[ Inaudible ]

>> For the AA degree, does it include advanced certifications? I did nothing together. I kind of remember how the definitions work in the data set. I can look it up. I can look it up and I can probably answer that question at the end. But I don't think of certification to be the same as an AA degree. Strictly AA. Now somebody who's not in these groups would be like if somebody came in with an AA and they earned another AA, I'm not going to capture those. That's a little bit of a weakness. Likewise if they came in with a bachelor's degree and they got a second bachelor's okay? I'd miss those. And here they don't make a distinction between a master's degree and a doctoral degree. So if somebody came in with a master's degree and left with a master's degree, they're not going to be picked up in this category. Likewise if they came in with a master's degree and came in with a doctoral degree, I'm not going to pick them up as somebody who advanced their education. And that's kind of one of those limitations in the data and the way it's tracked. So if I had my preferences, would I like to do this change? Absolutely. You know, but the likelihood that's going to happen, not likely.

 

back to top

 

[ Inaudible ]

Does the definition of college training include vocational training? I believe it does. But as I'm telling you this question, I don't think it's going to get a certificate that they're going to be reflected as having finished an AA. I'm pretty sure you have to finish the degree in order for it to show up in this kind of a category.

[ Inaudible ]

Yeah, yeah. Correct me if anybody knows I'm wrong, but I believe that if somebody takes a class at a community college that's not necessarily degree-focused, that they would still show up as having college training provided, even if it didn't lead to a degree.

>> That's correct.

>> Yeah. So if somebody goes to heating and air conditioning school.

[ Inaudible ]

Well there is a category for post-secondary education, no degree. And I would suspect that folks who are sent through that kind of training would probably show up in that category, more so than the degree categories. Okay, the category for AA is actually associates degree or vocational technical certificate. Okay, so those folks would be reflected in there. So a lot of what we're focus on from this point forward is comparing the not provided and the provided group to one another and then also doing a little bit of comparison amongst people who have advanced their degrees and wound up at these different levels. Okay, so now I can ask you a question. Given what we have talked about, given what you already know about rehab, if we were to look at the cost of people's plan, okay, and we were going to look at maybe comparing the folks who didn't get college training and the folks who did get college training. Do you think that the folks who got college training are going to have, first of all, different average costs of their cases? Do we think it's going to be lower, the same, a little bit higher, a lot higher? A lot higher? Okay.

 

back to top

 

[ Inaudible ]

Okay, so general consensus is a lot higher, and then you were saying that you feel like for some folks -- is that right? That for some type of folks it may be even more, depending on what kind of supports they need in order to get through education?

>> Right.

>> Okay.

>> I think college will cost more.

>> Good guess, everybody. But we can kind of take a look at exactly how much here. So what I have here is the mean cost of services for college training. Now this isn't just the cost of college. This is the cost of everything. Remember I said you can't separate things out by specific services, but we can look at it as an overall for the cost. The overall cost then on particular cases. So what we have here is a real apples to apples comparison between these first two columns where you see no college in solid gray and college training in the diagonal stripes. And a considerable difference in terms of cost. Some of that might be explained by college. There's likely a bunch of different factors that can explain that difference in cost as well. And so just to kind of put some numbers to that, the average cost of purchase services for somebody who doesn't get any college training at all, about $1,800 in terms of case service costs. Services purchased for somebody who gets college services provided, about $8,600.

 

back to top

 

So about 4.7 times higher costs. Now I actually had to go back and look this up down here. Degree completion is a subset of college training. So these are the folks who actually completed their degrees, completed the AA, completed a bachelor's or completed a graduate, advanced their level of education. They're actually in this group in terms of comparison. So they're already reflected here. But I wanted to kind of pull them out and separate them out and say, "What are those costs?" Completing a whole degree might take more time, right? Okay, and it's probably a greater investment of resources. So and in fact, for the folks who do advance their education and go complete like a certificate or AA or bachelor's degree or graduate degree, the costs again are considerably higher. And just kind of for your information, that cost is almost $15,000. So if you compare them to the no college training provided, about 8 times, a factor of 8 higher.

 

back to top

 

Did you guess right? These aren't going to be terrible difficult questions. If we were to look at how long a case would last, comparing no college to college? The college case is shorter, same, a little bit longer, a lot longer? A lot longer? Okay. A fair amount longer. So this is that in days, this little chart here. You can see for the no college group that 487 days or 1.3 years, kind of the average duration of a case. If folks get college then that goes from 1.3 years to 4.4 years. So a lot longer, right? And again, with this group, the degree completers are the subset. They're already reflected in this group here, but if you pull them out and separate them just to look, then for those folks who went on to actually complete a degree, about 5.8 years. So difference of 1.3 years here, 5.8 years here. It's a considerable investment of time, it's a considerable investment of resources, both on the agency side, the counsellor side and on the client side. People aren't necessarily in college incidentally the whole time, right? They might have a case open for quite a while before they go to college. They might keep their case open for quite a while after they get done with college, okay? So college likely alone doesn't explain the entire difference, but it's probably a contributor.

 

back to top

 

[ Inaudible ]

You're about one slide ahead. So I was going to ask you, like my previous questions about employment outcomes. And keep in mind this time we're using the competitive employment column. That's what I use. So if we compare say no college to college training, do you think the folks who wind up in competitive employment is going to be the same? Less? A little bit higher? A lot higher? Moderately higher? A little bit higher? It's actually a bit higher. So for folks who don't get any college at all, by the time their case was closed 27.3 percent were competitively employed. For those who got college, 51.3 percent were employed. And again, the folks under degree completion are in this column here, but if you separate them out. Okay, I want to see if there's any difference for folks who follow it all the way to the end. It is higher. About 82.1 percent. So completing their degree really does bring people up quite a bit. Well, it's a contributor to it. It's probably not the sole cause. Yeah?

 

back to top

 

[ Inaudible ]

People who are not in plan? Right. This would be everybody who, it would probably include people who were not in plan. It would include anybody who didn't get those services.

[ Inaudible ]

Who were not in plan yet. And frankly it would also include people who had plenty of college before the agency for services. So we likely have people with bachelor's degrees. We definitely have people with AA's, graduate degrees, advanced degrees in the group. They just didn't get it as part of their rehab plan.

>> I might be jumping ahead, but is it difficult? It seems like when you look at the data, one of the things you could maybe say is that people who have college, if they came into the plan development stage with kind of a higher level of potential with jobs.

>> Taking down achievement of job?

>> Is it difficult to say that college is the key difference?

 

back to top

 

>> Yeah, I would say it's impossible to say it's the key difference. Who decides? Thinking about rehab as kind of a client/councilor partnership, how is the decision made to go to college and to provide support to an individual who wants to go to college? What factors in?

>> Motivation.

>> Motivation, someone said. Anything else?

>> The client.

>> The client.

>> Acceptance.

>> Acceptance. What are kind of the characteristics of somebody who would go to college and the counsellor would be like, "I'm good with this"?

>> Self-directed.

 

back to top

 

>> Okay, self-directed.

>> Perhaps prior success.

>> Prior success, right.

[ Inaudible ]

Potential, right, is something they might have. Okay, persistence. Probably the interpersonal skills to negotiate the college experience, right? So those are all sort of embedded in this, right? So it's not like -- I suspect that the no college group, if you were to measure some of those characteristics which would be difficult to measure. Persistence. Something like prior experience would be easier. We would find differences between these groups. They're not completely identical groups with the only difference being college. Those other issues that we just discussed are likely all embedded in that. So you can't necessarily say college alone is causing this, is the sole cause. It's likely that all these things are kind of wrapped up in a rather complex picture.

 

back to top

 

[ Inaudible ]

There are statistics showing --

[ Inaudible ]

Okay, so are there statistics that show people that started on the college educational path but didn't complete their education? Yeah, and in fact they're kind of embedded in this middle category here. Anybody who got any college training is in here. And then I pulled out the people that actually completed the degree.

>> That's what I wanted to know.

>> Yeah, but there are people in here, plenty of people in here actually. These represent percentages, right, and not hard numbers. This group is likely a very small group and a small part of this whole group here.

>> Okay.

>> Yeah, so they are kind of embedded in here. There are people who went to community colleges and took one class and they would be in here. And then there are people who got one unit shy of finishing it and they would be in here as well.

>> Okay. So they're not broken up individual?

>> I didn't break them up specifically, no. Yeah?

>> This is from Vanessa in response to the question I had posed. She says, "Hi!"

 

back to top

 

[ Inaudible ]

>> Sure, that would be a decent motivator, somebody forgoes earnings for a period of time in order to complete an education.

[ Inaudible ]

Quality services with college. And that's a thing that we'll come back to, this idea of quality of services. Okay, so one thing I want to kind of highlight here before we move on, remember earlier on there were some criticisms that were levelled at college training, saying, "Look, less than half the people who get it actually wind up being employed." With the competitive employment definition I think I'm using a little bit more stringent definition of employment than was actually used in previous studies. And we get pretty close to half and half. So it's just a hair over that, 51.3 percent. So in this case the finding kind of runs counter to that previous finding. But the other piece of that argument that was never really articulated was like, "Okay, how do people who don't get any training do?"

 

back to top

 

And that was sort of left out of it and you're seeing those folks over here. So there is a gap there. But it's not entirely explained by college. It's explained by motivation level and whether they can forego earnings and intelligence and their personal skills and that type of stuff as well as the college experience. So I would at least, given my own personal biases, at least have that 50 percent mark. How about if we looked at the type of degree that somebody completed, whether it was that AA level or a bachelor's level or that graduate level degree, and we looked at competitive employment? So only for the degree completers. Do we think there will be differences in competitive employment between the different degrees completed? Will those differences be not at all? Will they be big differences? Will they be slight differences?

>> I think the more education you get, the better chance you have.

>> So you think the higher the education the better the chance of competitive employment. Okay. Well you're right. I was a little surprised by this though. If you look at it, so the first message is they're all fairly high, right? But the second message is though that there's not a huge amount of difference between completing a degree and a different degree. If you completed that AA level, about 77.7 percent were competitively employed when their cases closed. It goes up to 84.4 percent with the completion of a bachelor's degree. And 87.9 percent with completion of a master's degree. Yes?

 

back to top

 

[ Inaudible ]

Okay, so that would be kind of employment in an integrated setting, meaning a person with disabilities working alongside everybody else, not kind of sequestered with a whole bunch of other folks with disability. The employment is at wages at least at minimum wage or better. I think that's kind of the two key criteria. I don't remember what else is.

[ Inaudible ]

No, no, no. It's not true, but these people could all be working at Jack in the Box or something like that and they would show up as employed, right? But there are other measures in the RSA 911 data where you can actually look at not necessarily long-term earnings, but how much they were making at the time their case was closed. And that would give you -- that's another one of the indicators that we have agencies frequently evaluated on. I didn't choose to focus on it here, but you could certainly look at that. You could look at earnings and that could be a productive line of inquiry if we want to relate degree completion to earnings. You know, if you find that regardless of the degree they complete they're all making the same amount, that might tell us one thing. You know, if we see marked differences in the amount they earn by degree, that might show us something else.

 

back to top

 

[ Inaudible ]

Yeah, on the competitive employment side, right. So nobody here would close as like a homemaker or something like that. That wouldn't meet that criteria of competitive employment. Okay, so not too far to go. We said earlier that people can get a variety of services through this VR agency and I know these are small here. There's 21 difference services that people can get.

>> I'm sorry, before you go further.

>> Yes.

[ Inaudible ]

I didn't actually look at earnings. It would be something I could do in a matter of an hour or two with just some additional analysis. In this presentation I didn't look at earnings. But it could be in terms of an area to go into, to investigate. It could be interesting. Speaking about it, one of the things I'd be interested about is for the folks who complete degrees and don't necessarily go to work, I'm kind of a little bit curious about the role that benefits play in that, you know. Like is it a breakdown somehow for the folks that are comparing -- I just want to know how that plays in. And that could be another area that could I think be productively looked at.

 

back to top

 

[ Inaudible ]

Within the data set you can look at whether they were receiving the benefits or not and how much they were receiving and that sort of stuff. You can separate them out or you can do comparisons. So we can compare competitive employment rates of folks who completed degrees and weren't receiving benefits to those who were, that kind of thing. That could be interesting to look at the differences. Okay, so 21 different paid services. These are also services that folks can get in their categories or classes of cases, types or services. And these were all sort of factored into this last little bit of the analysis we're going to look at. Do any of these things tell us anything about the likelihood that somebody's going to wind up competitively employed? And so as you're kind of looking through these, you might think, "Okay, which ones?" You can ask some questions. Which one of these things hit me as probably an indicator that somebody is going to wind up competitively employed? Assessment, diagnosis and treatment of impairments, counselling and guidance, occupational and vocational training, on the job training. Can everybody read these, or is it helpful to have me? It is? Okay. On the job training, remedial literacy training, job training, disability related augmentative skills training, miscellaneous training, job certification, job assistance, on the job support, transportation, maintenance, rehabilitation technology, reader, interpreter, personal attendant, technical assistance, information referral and other services. As we're kind of thinking about that, let's look at a few different ways that these case services could potentially show up as predictors of whether somebody is going to show up in competitive employment or not.

 

back to top

 

[ Inaudible ]

Literacy training potentially making a difference, rehab technology potentially making a difference.

>> Job placement.

>> Job placement. Okay, so it's a little bit of a guessing game here. We have a rehab client. We don't know the outcome, but somebody does know the outcome, whether they wind up employed competitively or not. How much help do you think knowing whether a person received these services or not for each one? Kind of a yes/no basis. How much help is that going to be to us in terms of predicting whether they wind up competitively employed? A lot of help? Okay. Now another thing to think about is think about all the little factors outside these services that play into whether someone winds up competitively employed or not. We already talked about some of them with respect to suitability for college.

 

back to top

 

[ Inaudible ]

So to kind of paraphrase, there are folks who could get very little, but because of other characteristics they do very well regardless. Okay. Well, so sort of the question is, how much do these things help us in our ability to kind of predict the variability and whether somebody becomes competitively employed or not competitively employed? The way that they're defined now, this helps us to explain. If you think about variance, what's called the variance in competitive employment, how much of the outcome can be explained by knowing these characteristics? Not a huge piece of the pie. About 7.8 percent. Now start thinking about a few things. One, what we just talked about. None of those service variables really capture people's motivation, or their interpersonal skills or their intelligence or their abilities, right? Or the level of support that they have from other folks, right? So those are the ones that are probably the most important piece. It's another piece though. Remember how these service variables are captured. Did you get job placement assistance? Yes or no. Did you get rehab technology? Yes or no. Do you think that there might be a way to capture that information that would actually broaden this little piece of the pie to help explain more? Yes? Like what would we like to know?

>> What job placement assistance?

 

back to top

 

[ Inaudible ]

Okay, so how intensive were the services?

>> Yeah. How individualized?

>> How individualized? Yeah.

>> How long?

>> How long? Like was it one 15-minute session? Was it four months of twice a day? Yeah, and we might do that with any of these things. We have technology. Was it a relatively insignificant piece of equipment on a one-time basis? Was it a full assessment? Were there updates based on changing the person's plan, right? So maybe we can think of this in terms of the precision or the preciseness of the variables. Here they are relatively gross measures. Yes or no. You got it or you didn't get it. If we knew more, and again this would be like, would I love to ask them this? Yes. Would they even listen to me? Probably not, right? And do the counsellors have time to provide that information? It's easier for them to check the yes/no box than it is to describe in great detail precisely. So it's a bit of a pipe dream. But my gut sense is if we had more precise definitions of those variables we could actually, just knowing the services, we could make that piece of the pie bigger, okay? Would it ever cover everything? No. There's just too many other possible outcomes, right? Too many other possible factors. We have to factor in a lot of those other things. Sure.

 

back to top

 

[ Inaudible ]

Well it has value above and beyond just the way I'm looking at it, right? So Chet's question is, can you argue that this data doesn't have a lot of value? I think it's useful for a lot of other things. I think this is somewhat valuable, and actually we'll get to a couple more slides, but one in particular I think does demonstrate some of the value. So this gets a little technical. I'm not going to spend a lot of time on it. But what we're trying to do with knowing these variables, and we only have a few minutes left, but I'm pretty sure I can get through it. If we didn't know any of those service variables at all, and we're trying to build a model, right, which is essentially an equation that predicts whether somebody is going to wind up competitively employed or not competitively employed. And we say, "Okay, do it, but we're not going to tell you any of the service variables that people go." So you basically have nothing to go on to build this model, other than you know that 51.3 percent of the folks wound up being competitively employed. If you wanted to maximize the likelihood of guessing correctly, you would just simply guess that everybody got competitively employed and you'd know that you'd be correct 51.3 percent of the time, right? So I beat chance, right? Okay, so that's what the equation is trying to do here.

 

back to top

 

I said, "Okay, I'm not giving you any information, but I'm trying to get as many right as possible." So actually you can see here's what the model of the equation predicts. By the way, there's no quiz on this, but this is called binary logistic regression. We're trying to use that model to predict accurately who winds up competitively employed and who doesn't. So what the model predicts when I have nothing to go on. It predicts that everybody should be competitively employed. It predicts that the folks that weren't actually turned out to be not competitively employed and those that were competitively employed would be competitively employed. And so it doesn't predict that anybody is not going to be competitively employed. And it's right 51.3 percent of the time. The reason I show you this one is to kind of show you the next one, which is, the ones that ended up being kind of critical, the service variables that I asked about, which ones are the most critical to predicting whether somebody ends up successfully employed. If we throw those in, how much better can we improve over 51.3 percent? And you already saw the pause. So what's your guess? Can we improve our prediction rate, not at all? Does it get worse? Does it get a little bit better? Does it get a lot better?

 

back to top

 

>> A little bit better.

>> A little bit better, right? We got that 7.8 percent or whatever that we're holding onto. Okay, so our prediction goes from 51.3 percent to 60.9 percent. So not great, but it is an improvement on the original when you didn't know anything, right? And you can see now that there's a little bit more trying to discern who's going to wind up competitively employed and who's not going to be competitively employed. So if you looked at the not competitively employed as predicted, and the not competitively employed observed, that's accurate. Those are right. And also for competitively employed over here, and over her predicted competitively employed and observed competitively employed, those are right guesses. And then the other two, they're wrong. That's where the model misses. Okay, so we need more information. We need better definitions of those service variables that we got, or we need to know information about the person's intelligence level and their interpersonal skills and their academic inclination and persistence and all those types of things. And if we had good measures for that kind of stuff, we could probably do an even better job of kind of predicting who's going to wind up competitively employed and who's not. So this is maybe not the most common-sensical way to interpret this. I'm going to show you one more slide here that looks like a lot of numbers and stuff like that, but hopefully you can get this to a point where it's relatively easy to understand. So that model, what it works on is I'm trying to figure out which are the variables that when taken in combination do the best job of predicting whether someone is going to be competitively employed or not competitively employed. And I'm referring here to the ones that are associated with winding up competitively employed, positive indicators. When a person gets the services, they've got a greater chance of being competitively employed.

 

back to top

 

[ Inaudible ]

The ones that are statistically significant. So all these are highly significant. If that doesn't ring a bell, don't worry about it. And the nice thing about binary logistical distribution is what are called odds ratios. And these are the kind of thing that you get back to interpret. And the way to look at this is, if a person got a particular service, the odds ratio is sort of like the amount that the likelihood that they'll wind up being competitively employed increases. So the way I would phrase this is, people who got job placement assistance are 2.05 times as likely to be competitively employed when their case is closed as people who didn't get it. Okay, so your odds are more than doubled, okay? So is the service alone explaining the cause? Probably not. Who gets job placement assistance?

 

back to top

 

>> Lots of people.

>> Lots of people get it. Do they get it right at the beginning of their case?

>> No.

>> No. They're got to stick with their case for a while, right? Okay, they have to get to a point where they and their counsellor determine they're kind of ready for that. Yeah, okay, so in some respects it's an indicator of -- the service itself probably contributes to that outcome. The characteristics of the person probably contribute as well. And we can kind of go down this list. But what else seems to be associated with increased likelihood of competitive employment? If somebody gets maintenance, and that's usually money, right, that's provided for transportation and that kind of stuff, living expenses. If they get that, people who get that are 1.6 times as likely to be competitively employed when their case is closed as people who don't get it. Okay?

[ Inaudible ]

I believe occupational and vocational training would be noncollege kind of training, right? Like adult ed type.

[ Inaudible ]

 

back to top

 

There's a separate category for literacy. I can actually look it up. So the question was about occupational/vocational training. Occupational, vocational or job skill training provided by community college or business, vocational, trade or technical school to prepare students for meaningful employment in a recognized occupation, not leading to an academic degree or certification. So great guess. Job placement, you can see all these other services. If you get that, you're almost 1.6 times as likely to be competitively employed. On the job support is 1.4 times, almost 1.5 times as likely to be competitively employed. The closer you get to 1, 1 would be like you get it or you don't get it, it doesn't make any difference. So assessment is statistically significant, and it improves your odds a little bit, or it's associated with improved odds a little bit. But it doesn't carry nearly the weight as the ones up near the top. The 2.05, 1.62, 1.59. Okay, one last slide and one minute. So kind of the broader implications. We've already kind of covered a lot of this. So we're just bringing some of these things back. One kind of documenting association between college training and competitive employment. Getting some of those questions answered regarding, how do plans that involve college training compare to plans that don't involve it? And obviously cases are longer. Greater investment of time and resources by the clients and the counsellors. They're more expensive. But the employment outcomes are higher. Identification of potential for pursuing college training. And that relates back to what I said with the odds ratios. Those services don't necessarily cause competitive employment rates to increase, but they are associated.

 

back to top

 

There is some kind of a link. So for some individuals we might look back at some of those services and say we're putting some of these in place for somebody to help them out if college is in their plan. We talked a little bit about minimum service variables. I bet we could do a lot better job. We could predict a lot more accurately if we had more robust, more precise definitions of those variables that we looked at. And then kind of the last thing is, of the variables that showed up in the literature before I kind of launched into this whole thing that seem to be really tightly connected with winding up in competitive employment and successful outcomes. Again, job placement kind of rises to the top for this group that received college and university training, just as it has for more general or different groups that have been explored in the past. Take almost any group you want to look at, that variable is going to be one of the major predictors. Questions? Thoughts? Complaints? I can take it.

>> There's a question from the webcast. Can you track people by type of disability? And notice in the data there's another variable in there that talks about sort of the nature. They might call it something like incidence or etiology.

 

back to top

 

[ Inaudible ]

Maybe someone has graduated law school, but then acquires disability through an accident and then goes back to school again. That person's profile.

[ Inaudible ]

There are usually about 600,000 cases a year because it allows you to keep that large a data set. It allows you then to look at some groups of people with a little more precision. The sad news is like he said, on that array of services it's yes/no on every one of them. And as you guys pointed out, there's a whole lot more than yes/no. This person got some job search training or something like that. I think that's always going to be a limitation of the system.

>> And just for the people who are online, Dr. Jacobs was saying that there are variables in the data set that allow you to look at not only the type of impairment, but the cause of the impairment. And that seems to have potential for some exploration with respect to whether the disability occurred early I life or later in life. Did I capture that?

 

back to top

 

>> Yep.

>> Okay. Yes?

[ Inaudible ]

No. So Jan's asking, at least in this analysis, did we document whether people were employed and what they were trained for? And there are occupational codes including the outcomes, which is probably a strength there. I think the disadvantages -- I don't think there's anything that tracks the type of degree that they earned above and beyond whether it was an AA or bachelor's degree or graduate degree. So we don't know what their degrees are in. Jeff?

>> This is from Vanessa. How does one extrapolate meaningful data from the RSA data set? Is there a program that can interpret it?

>> Okay, so how do we kind of go from that text file that looks like a big mess to something that we can actually work with? We use SPSS actually. So we developed some syntax that tells SPSS exactly what each of those characters in that data set mean. And then we define all those variables. And then once we get it there, then we've got it in a working format that we can use.

 

back to top

 

>> I think the issue is as research do you have to do all that yourself? Do you have to define the variables?

>> Well, yes, but it's more of a clerical function than any sort of a high-order intelligence function. Because RSA gives you a 70-page document saying, "Here's what all the variables mean." And so it's basically taking this information within SPSS or you can use Stats OR or Excel or whatever it is that you use. And you can take this information and kind of it becomes this almost mind-numbing. Then you can basically give instructions to SPSS in SPSS's language, or to SAP, SAP's language or Excel or whatever. And then it becomes usable again. So RSA is trying to provide the data set in a form that takes up not so much space and then is usable across a variety of platforms, which is why we have to go through that step of saying, "I'm going to use SPSS so I've got to put in the formulas for SPSS."

 

back to top

 

[ Inaudible ]

Is the information provided by gender? Yes. Gender and ethnicity. Gender is reflected in the case file. Ethnicity is reflected in the case file. Right, and you've got those. And we've got that information if you are then interested in focusing in on say folks of a particular ethnicity of a particular gender. You can actually then tell SPSS or Excel or whatever, you can do a set of commands that will even pick out, "I want to focus on folks who are Hispanic or Latino females." And even age. You could even say within a certain age range.

>> Okay. Thank you.

 

back to top

 

>> Sure.

>> This is from Dylan. I bet there would be various differences between services provided by the counsellor and services contracted out.

>> There would be differences in services provided by the counsellor and services that are contracted out. And you would have some ability to kind of look at some of that. Because the case services of I treated them were yes or no cases, but there are two variables in there. One is, let's see, who provided it and who paid for it. So you could look at the provision piece. And when services were provided in-house, what were the outcomes of those compared to the services that were contracted out. Yes?

[ Inaudible ]

If it turns out the way we think it will. If it turns out the way we hope it will.

[ Inaudible ]

For Vanessa and Dylan, Chet was just saying that while the outcomes according to degree are useful, you could certainly add another layer of depth and kind of help counsellors out a little bit more by looking at also wages earned and doing the same kind of comparisons by those wages. And that, you know, there's a lot of just setting up the data and setting up the groups and that kind of stuff that takes time. But that's basically why you have the data file set up now. That way now so doing the wage piece could be done within these parameters pretty quickly. Yeah, could do it. Okay, no problem. We'll be working on that. Okay, well thanks. That was fun. I just needed to job my memory. We ran over a little. Thanks for sticking around. I really appreciate that. Thanks, Vanessa and Dylan.