Accessibility

Transcript for March 2016

>> Thanks for being here. We'll spend the next hour or so kind of talking over a little presentation that Chuck and I have. We've been working on. Looking at predicting whether people advance or don't advance degree wise and certificate wise when they get services through voc rehab, in particular people with chronic brain injury, and this is been something that has been accepted for a presentation at the NCRE Conference next month. So were kind of doing this is a little bit of a dry run, and then also an opportunity to kind of solicit feedback and kind of talk things over and get some insights that might help us kind of improve our presentation next month, and in addition, I think we anticipate going eventually to publishing off of this, and so again, feedback here would be really useful in terms of helping us to kind of develop something for a manuscript for publication.

 

back to top

 

>> And we have just recently conducted the analysis. So one of the thing were going to want today from you guys and also we've got some people online is just some help in terms of explaining some of the findings. One of the things we're going to get into with this data is that this is the limitation of the data source we use, the RSA 911 data, is you have to speculate quite a bit on different reasons why certain things are happening. So we'll have a number of slides to get some of your ideas, and as Mark said, I think, before our NCRE presentation, to have more like firm ideas of what may be going on, and then eventually, we'll submit this for publication, and this particular study is building from a previous analysis Mark and I did when we looked at case service predictors for people with TBI in the state and federal system that had advanced educationally and had successful employment outcomes. So we're looking at a variance of that by focusing now on for the people that have succeeded academically through college in the state VR system, what seems to be related to that? What kind of role the state VR have in terms of helping people in that area?

 

back to top

 

>> Yeah, so, just this may be a little bit of a repeat of what we alreadt just talked about, one of our main goals here is to kind of identify the VR services, voc rehab services that are associated with people who have been provided with college. What are the VR services that are associated with folks that actually complete a degree or a certificate? And then we don't really get into this here, but in the analysis, you'll see we had attempted to control with certain demographic characteristics, as well. So we've been willing to take those sort of out as possible explanatory factors.

 

back to top

 

>> You know, and this whole area about just looking at what people with TBI need to be successful for college, there've been a few studies that have looked at this, and not surprisingly they have localized variables such as, you know, the post kinds of impairments that can happen after brain injury for things like cognitive problems. Things like memory issues. Different things, you know, directly related to the brain injuries. Some studies have found, you know, those things have been related to an injury. There has been some previous research that is looked at for people with TBI who left high school. You know, what predicts which people are most likely to go to college?

 

back to top

 

There was a study done in 2011 in Oregon and Washington. They surveyed the number of people with traumatic brain injuries, 66 persons with TBI, about 1.3 years after leaving high school, and they basically found that a few had higher socioeconomic status, you're more likely to enter college. Females were more likely to attempt postsecondary education, and then those who are injured earlier in life tended not to go to college as much as those who had been more recently injured. So in some ways, that'll connect with some of the things we found, but again, it's also going to relate to the limitations of the data that was sent. We can't really look at the injury severity factors to the same extent as if we're doing a perspective type of data collection, where we define all the items in terms of before we do the data collection.

 

back to top

 

Again, we're really relying on this data that we use that we'll talk about further. All right, so when we think about this population of people with TBI, you know, one of the things that Mark and I are really trying to establish as this is an underserved population. In the US, we know that there's around 1.7 million people that have TBI, and it's a disability that impacts people in a variety of different ways, and so when we think about all the things that can happen when the brain is injured. You get problems with memory, executive functioning, all the psychosocial issues that can happen, personality changes, not surprisingly, employment is a real challenge, and some studies have found unemployment can be as high as 80% for this population.

 

back to top

 

So, you know, what results is you have people who are staying at home with family, not productively engaged in work, not having a means for economic self-sufficiency, you know, all those things that we have from work, and we also know that people who do go back to work with TBI really have to, often have to look at a new kind of path which relates to this issue about college as a way to prepare people for a different kind of career path, because many people with TBI do not go back to what they used to do preinjury. And then when we, you know, we don't have employment for this population, it puts more stress, not only on the person but also their family, who have to, you know, meet that person's financial needs, care needs, and all the things, you know, again, with not having employment. All right, so in terms of looking at how can we predict employment? And specifically related to college. It's limited to the amount of data that talks about the fact if people with traumatic brain injury do go to college, it appears to have some level of prediction on are they going to be able to go into employment?

 

back to top

 

So in terms of if you're a state VR counselor and you're looking at do I support the college plans of my client with traumatic brain injury? You know, there is some evidence to say that that could be a factor to getting into a successful employment outcome. One study we look we looked at, we studied here from [inaudible] in 2003, looks at data from the 911 data set, which we'll talk about further, from '92 to 2000, and they found that 5.2 times more likelihood that if people with traumatic brain injury received college training as part of their employment plan, that they were 5.2 times more likely to obtain competitive employment. So in terms of, you know, making a selling point for making that investment, again, some of this data that does exist seems to say that that might be a good investment. So the data that Mark and I used for this analysis refers to the RSA 911 data, and it's offered by the Office of Special Education and Rehabilitative Services, which is part of the Rehabilitation Services Administration.

 

back to top

 

So everyone in the room is well acquainted with RSA 911. For those of you guys watching online or if you're going to watch this video at a future date, you know, you may not be familiar with this, and I think in the field of rehabilitation counseling, it's an often-used data source for a number of studies. One is that it's free to the public. So any US citizen can request to have this data. Now when they send it to us, Mark has a slide which we didn't include in this slide, which includes how it's sent to you in terms of raw data. It basically is just millions of data points you have to put together into first and Excel file, and then to FPSS file to be up to do analysis on it. So it's not, again, in any user-friendly format, But once you do have the data there's a lot of things you can look at, and basically, across the United States, for all 50 states and all US territories, they have to maintain this data from all the different kinds of client cases that are closed in the fiscal year. So it's a tremendous amount of data.

 

back to top

 

Now the value to this is that it's free. It's available for analysis like were going to talk about today. The problem is it's not perspective data, meaning that you don't define what's going to be in the data. You have to work with what's available, and later on, we talk about the limitations of the data, that's a real problem, because there are certain questions that you can't answer because you can't dig into the data and go further beyond what's available. So you often will have to speculate, and later on today, when we talk about what do our results mean? How do we make sense of them? We largely do have to speculate. We have to think, well, maybe it's this or maybe it's that, but we really don't know, but again, I think it's a valuable resource to have just for research purposes, just for the amount of data that's available and the fact that it covers the entire US population.

 

back to top

 

>> Me? >> That's you. >> Okay. So it's just kind of foreshadowing a couple things that we're going to look -- that will kind of be important later on to the analysis, to some of the kind key variables that we're looking at, and one of them is an outcome variable that is basically whether or not people who were given college, provided with college while they're voc rehab cases, did they actually complete the degree or the postsecondary certificate that they were pursuing. That's not in the data set itself. You sort of have to tease it out by taking a look at what their education level was on the way in, whether or not they go college, and then what their education level was on the way out. So using those variables, we were able to kind of pick out for the folks who got college, which ones advanced to the level of postsecondary ed that they were at, and which one kind of stayed where they were.

 

back to top

 

Then we look at some demographic characteristics primarily in the interest of controlling for them. Like taking them out as explanatory factors, and those are race, which is kind of encoded in a way that we'll explain later on, which is maybe a little bit contrary to what you might expect. Ethnicity. Hispanic ethnicity are not being of Hispanic ethnicity. Gender and then age, and that age is the age when they apply for rehab services. And then other things that we looked at were the rehab services that are tracked, that are provided and are tracked in RSA 911 data set, and besides college, which is one service. There's 21 others, and I won't necessarily list them here, but we'll see them later on, the 21 services that are listed. Yeah?

 

back to top

 

>> [inaudible]. So are you able to track whether or not they actually did complete college or not on the RSA 911 and whether or not they got [inaudible]. >> It's not there but we are using other variables about the folks, you can figure it out. >> Cool. >> Yeah, so you can, as you probably know, the level of education somebody has on the way in is tracked. The level of education when their cases closed is tracked, and then we looked at those two variables as ways of knowing whether people have advanced their level of education or not. If their level of education was higher when that case was closed than when it was opened, then obviously, they'd moved up when that case was closed than when it was open, then obviously, they've moved up, and then, but you can move from no education at all to elementary education, and that would be advancing, right? But we further limited it to those who had gotten a postsecondary certificate or degree, yeah. And, you know, one of the things about this, we can probably talk about it now rather than later, is we used data from 2009 to 2013. So a five-year block of data.

 

back to top

 

At the time, we ran this analysis, 2014 was also available, but the services are tracked in a different way, and the services tracked are little bit different. So we couldn't bring them together. So that's why we used that five-year block, rather than the -- so five years from now, assuming there aren't a bunch of changes, then we'll be able to kind of use the same kind of approach with kind of the new or revised data. So let's talk a little bit about the population. Who it was the kind of ended up studying, and you know, when you smash together five years of data, each year has roughly around 600,000 cases in it. So we didn't do a full analysis on 2,959,994, but that's how many people are in the data set or how many cases are in the data set.

 

back to top

 

There's probably a few repeat visitors when you put all that information together. From there, you know, the population we were interested were folks that had a primary cause of impairment of traumatic brain injury. So when you select that group out, then you have about 46,800 folks who, you know, went through VR at that point in time, and, you know, had a primary cause of impairment of traumatic brain injury. There may have been some folks in the data set that had a secondary cause or unidentified cause. Those folks aren't in this group. Furthermore, we were really interested in the folks who were provided with college, and so if you select that group out, it's less than 10% of all the folks who were served over that five-year period. You know, Chuck earlier on mentioned, okay, this is a little bit of underserved population. If you think about it, when Chuck said 1.7 million brain injuries, that's per year, right?

 

back to top

 

So I think the thought about, okay, how many people in the US are running around living with a brain injury right now? And disability with a brain injury, it's more like 5 million, 5.3 million, something like that, and so over this five-year period, you know, 4000 people with brain injuries going through VR are provided with college is a pretty low number, if you think about the potential folks who would be eligible, and then of that 4128, how many folks advance? Actually increased their postsecondary level, 1225. So that's kind of a small subset, and that's really the populations we'll be interested in looking at. Those who got college who had traumatic brain injuries who advanced and who didn't advance. They'll be able to compare and contrast later on, and then, furthermore, you won't see this anywhere else in the analysis, but we were just kind of curious. How many people advanced to an AA or a certificate? That's how it's tracked in the data set. If it was up to me, I'd like to separate those two things out. I think they are separated out in the newer data, but for here, this is one of those situations where we just had to live with what we had.

 

back to top

 

So 469 advanced to AA or a certificate, 629 to a bachelor's degree and 127 to a master's degree or higher, and again, we start looking at master's degree or higher out of, you know, sort of 5.3 million people in the states and 127 of them, and then Chuck, this really stuck with me. He brought this up. He was pointing this out to me. He's like, "We know some of these people. They're graduates of our program." You know? And so we probably are the rehab counseling program. The state probably accounts for like 2 or 3% of the people out of 127 people over that five years. We started naming. We aren't going to do it here, but we started to kind of think about who we know, and there are folks who are clearly in that group. >> That's disturbing. >> Disturbingly small, yeah? I would agree. >> So I think it speaks to the potential that there. They really have a much broader impact to reach out to a much larger group of folks. >> I have a question. Is it important to know if they're actually using the degree or is it just the [inaudible] people who are actually getting through it [inaudible].

 

back to top

 

>> Well, this is another limitation of the data. So we know in this particular study, we didn't look at how college advancement related to employment. We did that on a previous study. So now theoretically, you could have somebody who advanced educationally but they're working at McDonald's, and they were considered to be successfully employed because of it. Now hopefully they would be using their college training related to their employment, but we don't have a way to track that. So in terms of some of the demographics, gender, it follows the general pattern of what you have in TBI in the United States. It probably, in the general population or the general population of people with TBI, there's probably a higher number of people, of males than females, but generally the data in this area was consistent with what we see, but when you get into this next slide and you look at population by race and ethnicity, we start to really see some potential disparities in terms of who is coming into the state agency? Who is being served in terms of college plans? So this graphic will show that we've got 3574 people that are classified as white, no other race. 554 nonwhite or mixed-race.

 

back to top

 

So in terms of percentages, that comes out to the white group is 86.6. The nonwhite mixed group is 13.4%. In the general population, this was as of July 1, 2014, in terms of the overall US demographics, it is around 77.4% white. Black/African-American along 13.2%. Asian along 5.4%. Hispanic, ethnicity of any race 17.4%. So the numbers here are matching up to numbers we have on the 911 data, are matching up to the general demographics, and this is something that's been talked about with a number of studies, not only in TBI, but other studies based on the 911 data, is that these racial disparities have been seen across a number of disability groups, where, you know, we see the white group served disproportional to what's in the general population. Yeah? >> Okay, the figures that you just have in those groups, were they like the general population? >> General population. >> Or in the general 911 data population? >> Those figures are Census Bureau's. >> Yeah, just general population. >> So then, my follow-up question is when you see that ratio, which is just your ABI, right? >> TBIs right.

 

back to top

 

>> Then how would that compare to the remaining population within the 911 data set or do those proportions remain the same, or do you see a difference? >> You know, I didn't look at that specific question, but I looked at kind of a semi-related question. So I said, okay, let's look at all the people who have TBI. This was the sort of incidental. Like imagine Dr. Jacobs asked me a question like this. How might I sort of respond? I said, okay, well, let's look at the group that had TBI that didn't get college, and how does that break down race and ethnicity wise? So those who don't get college, Chuck, those who did get college, 86.6% white, of those who didn't get college, 80.2% white. So, you know, it does seem like that folks with the same disability, you know, if you come in in TBI is your primary cause of impairment, and are hoping to get college, your odds are slightly improved if you're white and if you're nonwhite right now.

 

back to top

 

>> We had a question from online. To repeat the question, so the basic question was, how do the disparities that we see in our data specific to TBI compare to other populations in the 911 data set, and I think that's something, you know, when we get to the point of writing the article, I think we probably will look at that. That's a good point. [inaudible] >> Mark, the numbers that you just gave me for all TBI, the 40,000 or whatever, right? [inaudible] TBI for a five-year period? >> Well, it's the 46,000 minus the ones who got college. So I was comparing the race ethnicity breakdown. Yeah, right, right. During the same time period, yeah, from the -- structured from the same data set.

 

back to top

 

>> All right, so the next slide basically kind of extends the discussion specific to Hispanic ethnicity. So this slide will show basically we've got 91.2%, 3765 of those provided college with TBI were non-Hispanic, 8.8% Hispanic. On the previous slide, we talked about, in the general population, based on US Census Bureau data, that the number of people of Hispanic ethnicity or any race is 17.4%. So again, those numbers, what's being served in the 911 data is not matching to the general population. >> So that's just more demographics. We looked at age on applications. So how old were the folks with a primary cause of disability of TBI when they actually applied for rehab services, and we debated whether or not to throw this one out, and Chuck really like the way that it looked. So we left it in, and you can kind of see, you know, going ages 13 to 69, and I'm sort of encouraged that there's a VR counselor out there that provided college to somebody who was 69 years old when they applied for services.

 

back to top

 

It's kind of cool, but there's a predominant spike, you know, around 18 years of age. A lot of folks, coming in right at that point. Out that 4100 total, about 550 or so coming in right about that age, and what we thought was important about this was that it kind of speaks to the idea that, okay, the college experience that folks are getting that comes along with being a part of VR is likely for many of them their first college experience. It's not like some of these folks may be returning to college later on, but a lot of them, this is going to be there first time. >> Yeah. >> In terms of just the general population of people with traumatic brain injury in the US, the highest population right now are the elderly population, but the second highest are adults around the 16 to 24 age group. So this data is pretty consistent with, you know, what are these major population areas for TBI? Okay?

 

back to top

 

>> So the meat findings? >> Yes. >> All right. >> Or lack of meat. [chuckling] We can talk about explained variance. >> There are areas where there is lack of meat, but there's other areas where there something to talk about. So we looked at a couple of other sort of demographics that are characteristics of these cases, and one was, and this was whether that person finish the degree or didn't finish the degree. Kind of how much was the average cost of purchased services for the case? And these aren't entirely college expenses. They could be for other things as well. You don't get things broken down by -- during this time period, you didn't get things broken down by what specific service. Moving forward, that data is actually being tracked by service. So that will be sort of hopeful the future, and may actually help us deal with some of the limitations that we encountered with this particular input, but average, like a mean cost of purchased service, $10,465.

 

back to top

 

So, you know, it's a fairly -- you know, these plans are not cheap plans necessarily. Median, you know, which is, so a mean of the mathematical average of all the dollars spent divided by the total number of individuals. The median is like what's the mid point? You know, how many cases had more money spent, and how many cases had less money spent than kind of the middle case, and so the middle case is around $5,700. So that's a little bit lower, and then we provided the minimum and maximum for purposes of illustrating, you know, perhaps why the mean is so much higher than the median. And so there's at least one case in there with us at $189,000 on an individual, and that's going to drag the mean up, right? If you average that case in with everybody else. How long are the cases? On average, about five years, and we probably expect that. Somebody's, you know, getting college. But go ahead.

 

back to top

 

>> Did you do median on the [inaudible] of cases. >> We didn't [inaudible] up a median, but we could. [ Inaudible Speech ] Oh, oh, so the question was did we get -- did we do a computer meeting on the average duration of cases. >> [inaudible] can be a few cases, especially if [inaudible]. You get the average case like [inaudible]. I was just curious about how representative that was. >> We didn't do that. So the rest of that comment was could we have, you know, relatively long cases that would stretch the mean? Just like the mean was artificially inflated with the cost of purchased services. We didn't do it, but we could. I mean it would be -- we could do it in a couple of seconds. So that would be a smart thing to do. >> You may get to this. This goes back to my previous question. As a reference, I know from some of the stuff I've done with the 911 data, anytime there's a plan that involves postsecondary education, you're going to see a longer duration of services and a higher cost per case. So those figures would take on a lot more meaning if you compared that to the rest of the 911 data set, and then maybe even some selected disability groups within that, I think that would add a lot of power to it. >> Yeah. So the comment there was that based on prior work, Dr. Jacobs has found that any case that involves higher ed is to be, you know, longer in duration and cost more and that we ought to consider doing some comparisons to the rest of the 911 data set and/or various other disability groups. You know, you earlier on asked about, you know, competitive employment, you know, and that wasn't really a focus in terms of the outcomes of this particular inquiry, but we did take sort of a quick look at it, and what I found, at least, what I thought was kind of interesting to me.

 

back to top

 

I'm not sure what explains it, but we looked at, okay, who was competitively employed in the cases were closed, and looked at individuals who got college in advanced their degree, individuals who got college but didn't advance their degree, and then individuals who didn't get college, and all of those were people with traumatic brain injury, and, you know, those who did advance, 75.5% were competitively employed when their case was closed. So that's kind of higher than just the average case. What I thought was particularly interesting was those who got postsecondary education and didn't advance the degree, 34.7% of those folks had their cases closed in competitive employment, and that figure was actually lower than those who didn't get college at all. Those who didn't get college at all as part of their plan but who had traumatic brain injury, 45.2% were competitively employed. I had sort of like before I ran that data, I sort of anticipated that those who got college but didn't complete with still have competitive employment rates higher than those who didn't get college, but that didn't turn out to be the case.

 

back to top

 

>> With the idea, all right, that they were like deemed acceptable to go to college they probably had more preemployment skills, more employment skills in general. >> Sure, or the interpersonal skills to convince their communication skills and convince her counselor that college is a good investment for them or, you know, any variety of things that would tell, sort of, counselor that this person has the potential to be successful. >> But you don't have that in the 911. So that was speculation. >> Right. >> One thing you do have, and I think you guys have even published on this before. One of the key outcome variables is how much job development, career counseling, and so forth as a part of the services they receive, did you cut on that variable to see what difference it made or might you do that in follow-up? >> Well, so the question was do we examine some of the services that folks received and use that as a way of maybe helping to understand why we saw this discrepancy in employment outcomes, and I'm not sure how to answer that in a real succinct way other than to say that the services and the analysis that we did were predictors. So we looked at the correlation between those services and the outcome of completing degrees. We didn't look at the correlation between those services as predictors and employment outcomes in this particular study. We did do it in previous studies.

 

back to top

 

>> Yeah. >> Yeah. >> Yeah. >> You guys [inaudible]. Are you able to look at like whether there's greater rates of competitive employment by region or whether there's greater rates of college completion by region? >> Yeah, we talked about that actually yesterday. We didn't look at it for this analysis, but this is one value with the 911 data that you could do that. You could break it down state-by-state. You know, federal region by federal region. So that may be part of the future type analysis we do, and I think there could be some differences because, you know, later we're going to talk about like workability as like one of these kinds of VOR or state VR funded programs that could make a different educationally for people that's not available in other parts of the country.

 

back to top

 

So there might be some differences related to -- just to add to that, and Nick's original question was could you look at differences regionally? With this data set, the tiniest level you can track down to a state. So you can look at state-to-state differences, and that's data that's available starting in 2014, ZIP code is in there. Yeah, I know, and county. So I'm excited about that, because I think the potential to do those kinds inquiries is pretty interesting, and then furthermore, the ability to say if you wanted to rule out sort of geographic differences with an explanation for employment outcomes or things like that, you could do that too, but to me that's pretty cool, but just, some of the populations that we're looking at are so small, that we have to kind of stack a couple years of data together, and so I'm like waiting for like when's the next one coming out?

 

back to top

 

>> That's for '14. >> 2014 it would be on. [ Inaudible Speech ] Right, right, yeah. Eva? >> Yeah, I know you guys [inaudible] demographics for gender. Is this [inaudible]? >> It was about 60/40 men, and so Eva was asking what was the gender breakdown of this population. About 60% men, about 40% women roughly, ballparking it, yeah. >> [inaudible] about this correlation with the study you did on Social Security [inaudible] postsecondary education and the link to their education and obtaining employment [inaudible] finding those people who started [inaudible] outcomes from those who didn't ever even go to college. So that's [inaudible]. >> Okay, so Chuck -- >> That is one you did at [inaudible]. >> Jeff is being very generous. He's reminding me of a study I was involved in where we -- the findings around competitive employment were very similar. Those who started but didn't complete weren't as successful as those who'd never completed at all. He actually did most of the work on that, and was kind enough to include me in that work. But it is good to see those similarities. Okay, so we haven't talked about the primary. The sort of two major research questions that we had pursued, and you can probably see everything we talked about up till this point leading up to these two, and then were kind of going and get right into the findings on these two. So we were trying to look at, as you sort of hinted before, what combination of VR services, voc rehabilitation services best predicts completion of certificates for degrees for the VR participants who have TBI

 

back to top

 

After sort of taking out the influence of some demographic variables that, you know, are sort of beyond the individual's ability to control or the counselor's ability to control, and while it might have been nice to investigate every demographic characteristic under the sun and rule them out, what we were able to kind of do was to kind of rule out things like race, which we basically tracked, as you saw earlier, as either white, no other race, and then everybody else was in the other group if they were nonwhite or of mixed race. Ethnicity, and that kind of boiled down to Hispanic, Latino ethnicity or not. Gender, male-female, and age at application. So you'll see we kind of pull those out. I'll explain more later, but we try to control for those or rule those out as explanatory factors.

 

back to top

 

>> And, you know, our focus in looking at demographics, when Mark and I looked at the connection to our previous study on college attainment or college advancement related to positive employment outcomes, we originally didn't look at the demographics. The logic, I think, was that we wanted to isolate things that the counselor had control of. You know, the counselor doesn't control, obviously, things like race, ethnicity, gender. But when we sent it to the Journal, that was one of the conditions to have it accepted is they wanted us to control for those variables, and as we thought about it more, I think, it's a reality that we have to account for the influence of these factors, even if the counselor has no control over these factors. And so with this present analysis, that was intentional to make that part of our design. The other thing that we had done in our previous study, we only looked at one year data, and one of the areas of feedback we got was to look at multiple years. So that's part of why we chose to use the five years of the 911 data instead of focusing just on one year.

 

back to top

 

>> Yeah, the case numbers are pretty small for any one year. >> Yeah. >> In the previous study where feedback that we needed to kind of consider that, we weren't looking at everybody who got college. We were only looking at those who had advanced. So like a particular year you might only have a couple hundred people. So that was where it was a real benefit associated with bringing together five years' worth of data. [ Inaudible Speech ] Well, there's a couple reasons. One, the population was so overwhelmingly white that the other little populations would have been, by race, are very, very small, and then the second consideration is the way that RSA encodes race and ethnicity is pretty challenging to kind of -- because there's five different race categories and one ethnicity category, and it's yes/no to each one. So it's pretty hard to come up with an elegant way of categorizing race or ethnicity, and it's actually a good way of tracking race and ethnicity, just when you boil it down and want to make comparisons, it gets a little bit more challenging. So we combined everybody who is essentially not all white into one group, just for the sake of being able to pull off the analysis.

 

back to top

 

>> Yeah, yeah. Yeah. >> I'm looking at the wrong computer. That's what happens when you put two computers in front of me. Okay, so the kind of analysis that we did was a logistic progression, and if you're not familiar with that, I wouldn't sweat it too much. Try to explain it in a way that kind of makes sense. We're looking at this outcome variable of do people who got college and had a traumatic brain injury, to the advance their education level or do they not, right? So the outcome is kind of a yes or no. Yes, they advanced education level. No, they didn't. And then what we're looking for is after -- we're looking for the correlation or the relationship between each of the service variables and that outcome, okay? And we want to find out which of the services has a strong relationship with that outcome, and which of the relationships have kind of a weak relationship with that outcome, and furthermore, we want to look at that outcome, after we take out the effects of age, gender, race, and ethnicity, and so what you wind up with then is what's called a model or a collection of variables that predict that outcome, and then you also have variables that have a really weak relationship, and the kind of analysis we ran throws those variables out, takes those variables out. So the first thing we did is we said let's give age, race, ethnicity, and gender the opportunity to explain as much of the variation in whether somebody completed a degree or not first. We let them account for as much of the variation as they could, and then we said, "Okay, here's the 21 service variables. Let's come up with a parsimonious model that accounts for as much variation as possible in whatever is left after we take out the effect age, race, gender, and ethnicity had on college degree completion, so there's 21 service variables, and what you're looking on in the left-hand side of the variables that ended up being in the final model that did somewhat of a job predicting college outcomes, and then the ones that are excluded are the ones that had weak associations. John, do you stand behind it? Yeah, okay, all right. So real quickly, I'm just going read through those, you think, Chuck? Okay, so I'll read you the ones that were in the model.

 

back to top

 

>> No, yeah, just read -- >> What's in the model. So these turned out to play a part in actually predicting whether somebody completed a degree or certificate or not. As you'll see, some played more important parts. Some play less important parts, so we'll get to that point, as well, later on when we can talk about the relative contribution of each of the services. Vocational rehabilitation counseling and guidance was a service that was in the model. On-the-job training, disability augmented skills training, job search assistance, job placement assistance, on-the-job supports, transportation services, maintenance, readers' services, and information and referral services. One thing we want to keep in mind and you'll see later, these variables had a part in predicting. Sometimes they predicted a favorable outcome. Sometimes they predicted a unfavorable outcome. So not all of these variables are associated positive college outcomes, and we'll get to look at that later too. Anything you want to add?

 

back to top

 

>> No, that's good. >> Okay. >> Yeah, I think one thing, we'll do like a little prediction or kind of a thing to come. You would think now with all these variables, we have all these demographics. We have these 21 service variables, that when we look at what's, you know, what we refer to is explained variance. Like when we look at trying to see what accounts for people with TBI that are most likely to be able to succeed in their college training, you would think, well, this probably like explains most of it, but we're going to note later on that it's a very small percentage of the variance, which we were both surprised and disappointed by, and we'll talk about that in the context of, again, limitations to this data. >> So that's actually what folks are looking at now. So that's actually applied too soon. [inaudible] So how much of this collection of service variables and demographic characteristics, how much of the variation in college outcomes can they explain. Well, as it turns out, about 7.2%, which is a little bit lower than I think Chuck and I were expecting, but then I started to reflect on it a little bit. We provide -- sort of talk about that. I'm just kind of curious if anybody has any reactions to kind of that.

 

back to top

 

[ Inaudible Speech ] >> Yeah. >> Yeah. >> Yeah, that seems really low. I can't imagine. >> And for those of you guys online, if you chat your response in, I'll read it off to the group here. So if you guys have ideas, as well, let us know. >> Yeah, so that -- when Mark and I had met to talk about this, we're saying, "Well, what else?" Let's say if we could tell the 911 data any kind of data we wanted, what else might better explain, you know, college success? So that was one thing we wanted to try out to the group, because it's going to relate to what we talk about at the NCRE conference. It's going to relate to what we put in the publication.

 

back to top

 

>> Well, one of the things that we discovered on our study, Mark and I, was that family influence, family presence, family mores really had a big impact on somebody's success. And that's not the kind of thing we weigh. >> Dr. Onley was just saying that family influence in a variety of ways can have an effect, and basically, she found that, she had Dr. Compton found that in a previous study, and that's not accounted for in the variables. >> Yeah. >> I don't want to go back to my same thing, but like would region have something to do with that? >> It could. >> But it's not accounted for in the data though, right? >> Right. Nick was saying that region is another possible explanatory factor that -- yeah. It's not accounted for in the data, yeah. >> Yeah, one observation. You were up front, very careful about citing the limitations of the 911 data. So this is an example of that, but Mark just mentioned about, you know, the sort of immediate supports and influences in a person's life. There's also the intra-individual. You don't have any information about that. You don't have the counselor's take on this, you know. The whole counselor/client relationship. You know, there's nothing you can do about that. So another way to look at this is look at what we don't have. What we do have to work with explain that chunk. That's not so bad. You know, it's a good way to look at it, it did answer part of that question, and you know, it sort of begs the other question, well, are we in a position to define a group, a study group and then permission with all of higher RV issues associated with that to interview counselors or interview members of this pretty unique and well-defined cohort about how did you make it? How did your client make it? You know, stuff you couldn't get to in the 911.

 

back to top

 

>> Yeah, just real quickly. Dr. Jacobs was saying, you know, you're also missing the individual factors on the personal level. The counselor/client relationship, and how that plays into it, and then looking at all the things that we actually don't have. You know, you could look at this as interesting to see that 7% is actually explained by what is actually a relatively small amount of information about the individual and the individual circumstances, and then furthermore, this might be an area where we could continue to do some work by actually going out and interviewing people and gathering data directly. Nick? >> It kind of makes of me think of like exactly what you just said. Thinking about client engagement, and like a presentation I saw on that and how that affects certain outcomes in the VR process. I don't know. If I could just look back on it again, I think you could find a lot that's actually [inaudible]. >> Yeah, Nick was referring to client engagement, as well, as being an explanatory factor.

 

back to top

 

>> The problem is, if you got to permission to investigate it, and you have the energy and time and all of that, you don't have any identifying information. So you're stuck. I mean, you can't go find the 4000 people over the last five years [inaudible]. But you'd have to start, basically, you'd be starting from a situation where you're contacting counselors instead of looking at this, and could you give us information, but that still doesn't get to why, like if you look at to explain percentages and the unexplained [inaudible] instead of going back and saying, okay, well, maybe looking at something like, I don't know how you would pull this but 911 [inaudible] the number of cases and individual has open and closed during that five-year stretch and whether or not you could determine whether the persistence of the counselor in opening, keeping, and reopening or not reopening the case played a role in whether or not somebody succeeded over the long-term. You know, were they patient enough with the person, the ups and downs, to sort of, I don't know if you could find [inaudible] in the data.

 

back to top

 

>> So Chaz was saying, you know, you could potentially look at the indications of a counselors' persistence and working with a client through multiple cases being opened and closed as another factor, and could we pull that out of the data. Within a single year, you can. Well, shoot, no-no-no. I don't think you can. You can tell whether a case is the second case or the third case or fourth case of the year or closure of the year, right? But you can't link it back to the previous one and then, furthermore, after a year, you can't do it. But, you know, another thing that, and I'm kind of jumping the gun on limitations a little bit. So I'm doing this because I can see how disappointed Chaz is in this, but one of the things about this is the variables are tracked. So if somebody gets counseling and guidance, it's sort of tracked on the yes/no basis. What we don't get in terms of any robustness is was the counseling any good? How much of it did they get? What was the quality? All that sort of stuff. So I am kind of giving away a lot of our limitations already, but I think if we had that kind of information, and you can look at any particular service. Look at like job placement. They get it one time for five minutes? Or, you know, over a year. You know, and was the person any good who was doing it? That type of thing, I think, you would see the variants increase if we had kind of more robust measures of what we're going after. So I don't want you to kind of jump to the conclusion that all the services don't really make a difference or anything like that, because we know that they do, it's just not tracked the right way.

 

back to top

 

>> No, I'm just trying to -- I'm sharing a little bit of your frustration [inaudible] the link and what's the explanation. >> So I wouldn't call this a silver lining. This is something not up to the level of silver lining, but in the newer data, which we might be able to use as, you know, we get a few years under our belt, we don't get really much more robust measures beyond, yes, the service was provided. No, it wasn't. But we get a little bit more robust measure. It wasn't, you know, how much of the service did they get or was it any good, but we do get how much money was spent on it. So there you get a little bit more precision. Right now, if somebody gets $5 worth of counseling and guidance or $5000 worth of counseling or guidance, they're both a yes for counseling and guidance. Whereas down the road, if they get $5 worth of counseling and guidance, we'll be able to distinguish that from somebody who gets $5000 worth of counseling and guidance. Yeah. >> I was just wondering like what may be the difference between the severity of the TBI, or is it priority category [inaudible], you know? [inaudible] services you're going to need [inaudible]. >> Right, I think that's a big -- Nick was saying severity of disability is another explanatory factor, and I would tend to agree. We both talked about this earlier. Unfortunately, in the 911, in the data we have now, there's like severe or not. It's tracked, like -- and everybody is severe. So it doesn't work very well as a variable. No, it's not that the really severe -- >> [inaudible] severity but I think [inaudible].

 

back to top

 

>> Right we talked about that too. If folks were severe enough to get on SSI or SSDI, there'd be some potential drawbacks, in that there are some people in this data set that are probably on their way to getting determined eligible, and that type of thing, but still, it'd be one way to select your group that's somewhat homogenous on that characteristic. >> We didn't talk about this, but what was it? Like around 25% had SSI or SSDI among the sample? >> It was between 20 and 25% on each. So somewhere between 20 and 25% were getting SSI. Somewhere between 20 and 25% were getting SSDI, and there was likely some crossover between those. There was folks who were getting both. >> I know you looked at primary disability on this [inaudible] but did you look whether or not existence of the secondary disability was a factor at all? >> No, but we could. I mean, yeah. >> The other thing is customized possibly in length of case, and then that -- since you can't pull the number of cases existing in a 12 month period, it probably won't tell you a lot meaningful. If the case gets, and I know you looked at the median, not the mean, that you assigned, if individuals -- you'd have to do it within degrees, right? Because of course it makes sense if somebody pursues a master's it takes longer. So if we just looked at those individuals who obtained each one [inaudible]. If you look at the length of time that takes as open as a predictor of success, that might demonstrate, and I'm stuck back on the counselor [inaudible]. Which I'm guessing is based on experience that was a factor is a counselor's patience and willingness to allow multiple attempts that weren't particularly successful where they found the support needed and the [inaudible].

 

back to top

 

>> Yeah. >> In order to be successful, and that really doesn't get shown in many ways, and [inaudible] potentially length of the case, cost of the case [inaudible] within the [inaudible] multiple attempts at something. So, yeah, interesting. >> Yeah, we may look at that. It's possible [inaudible] length of case is an indication or a counselor's persistence in working with the individual. I almost wonder, we speculated about this yesterday a little bit, the low number of people who started. The lower employment outcome number, the lower percentage of folks who were competitively employed, who started college but that didn't complete, I was wondering like okay maybe the person sort of bombed out of college, and the counselor's like, "Forget it. I'm closing the case." And then so they're closed unsuccessfully, and then the person may come back through again. So that may be one of the reasons why we're finding that.

 

back to top

 

>> Does the 911 still have, under disability, the type of disability? Then you also have an etiology variable? >> Right, yeah. >> And I'm wondering on etiology, especially with the number of the service-connected head injuries, if you might see something different. Because that's a somewhat homogenous age group, a somewhat homogenous etiology. So I think there's something you could do with that, along with, like Chaz says. You know, within the limitations of the 911 data, there's some other things that you can do. Where my previous comment about, well, we don't know about the intra-individual characteristics that the circle of support, that kind of thing. Nothing about that really links to the 911 data. What I was suggesting is your recommendations might be included. Something like an RFP for a funded study outside of the 911 data that could go out as a [inaudible] initiated research or something like -- they still have those [inaudible] research. Where somebody could look at this, and maybe not even go directly to the consumer, but work through the counselor. The counselor's perception or you could get with the, you know, sort of the recidivism. I had to open the case and that kind of stuff. Or "Here's my perception on this person's persistence and commitment, you know, work habits," and so forth, and then even your service variables, if you're dealing with the counselor is not yes/no on each of the [inaudible]. It's just like "I really went heavy on this." Our [inaudible] I purchased this. I did this myself. You know, some things you could do that way. Okay, so that's sort of a pipe dream, but things you could recommend.

 

back to top

 

>> Yeah, Mark and I talked about, we're going to make a series of recommendations to enhance the 911 data set, as well, but yeah, I mean, we'll throw it all out there. I mean, I think that's a great idea. Yeah. >> [inaudible] happen, yeah, right. >> [inaudible] go back to the slide with the services on them one more time [inaudible]. [inaudible] included [inaudible]. I was just looking at the ones that are included that are slightly unusual from a typical case, and the only one that really pops out is disability related [inaudible] skills. That's the only one that I would say kind of generalized case of services.

 

back to top

 

>> So that's an unusual disability-related, augmented disability training was an unusual case service. >> Comparatively speaking. >> Right, right. So it's offered kind of to a relatively low number of individuals, and then particularly, maybe from, it might be offered even less commonly to people who are, say in college. >> I don't know about that. It's just not a common service. >> Commonly employed service, yeah. >> So it strikes me as -- I was trying to think, well, maybe the most unusual ones there might be somewhat more illustrated or something. >> I think it may wind up in the prediction model because like the people who tend to get that tend to all sort of cluster together in terms of their college outcome. It serves as a good predictor. It's got a strong correlation either between a favorable outcome or an unfavorable outcome. So that's kind of why it's in the model. It sort of pulls its weight in terms of being able to kind of predict how things are going to turn out. So I know we're coming up -- no-no-no we're coming up on an hour, and there's like one more slide that's definitely what I think of is neat.

 

back to top

 

So I want to make sure that we kind of get to that, and that is the result of the second question, which is related to the first. Okay, now that we have this prediction model that protects a lofty 7% of the total variation in outcomes of college, what demographic characteristics, what service characteristics are kind of doing the most work in terms of -- or have the strongest correlation or the strongest relationship with advancing or not advancing postsecondary ed? And so this is a table. It's kind of not the most fun thing to look at, but we'll kind of try to talk through this. So you see the predictors on the left, and they include demographic characteristics and service variables, and we've put little asterisks on the ones that turned out to be statistically significant, meaning the ones that you could take a little bit more info out of, and so, you know, significant.

 

back to top

 

Anything below 0.05 is going to be identified as significant. So a couple of other things -- this is one of the things that I really like about this type of analysis. You get an odds ratio for each one of those, and one of the ways to think about this is if the odds ratio is one, it means the person didn't get the service. They were just as likely to advance their degree as if they did get it. If it's greater than that, it means they had a better chance of completing the degree if they got the service. If it's less than one, it means they had a lower chance of completing the degree if they got the service. Some of these I have to explain. So gender was coded as 0 for female and 1 for male. So the way to sort of think about that, males were only 0.7 times as likely to complete a college degree as females. So females were actually more likely to complete the degree than males. Ethnicity, Hispanic was a 1 and non-Hispanic was 0.

 

back to top

 

So if you are of Hispanic ethnicity, you're 0.75 times as likely or less likely to complete a degree than if you are not of Hispanic ethnicity. Race was encoded as white was 1 and non-white 0. So if you were white only, you were 1.2 times more likely to complete the degree than someone who was not white or not all white I guess. I don't know. That's a terrible way to kind of say that, but you know. White of no other race. And age, it was -- the only kind of variable in here was like a skill variable. It's like a number line variable, and the way to look at this one is for each year increase the odds ratio was 0.97. So for each year older you are at the time you applied, you were slightly less likely to complete the degree. So the clock is not your friend in this case. The other ones -- and those were -- the other ones were all if you got the service. So like on-the-job training, if you got the service, that service, you were only 0.6 times as likely to complete the degree. So that was an indicator, giving that service was kind of an indicator of an unfavorable college outcome.

 

back to top

 

>> Yeah, some of these, as we go through some of these, you know, we're not able to directly measure injury severity, but we were talking about things on on-the-job supports, on-the-job training might be like a type of proxy indicator of injury severity. So if you're receiving those, you probably have a greater severity of disability, and then if you have a greater severity of disability, you have more barriers to college success. So we're probably going to talk about that as something we [inaudible] on in terms of the findings. You know, whereas we look at things like job placement as a positive impact on the jobs. Job search and job placement, which would indicate maybe that if you're receiving those services, you're job ready, and perhaps is another proxy indicator of having less injury severity, which is then related to you're more likely to be successful in college. So it's sort of like this kind of, you know, you're taking multiple steps in terms of making inferences, but I think it's reasonable, you know, to make some of those guess. >> I think you've earned the right to the state those and the responsibility to state those inferences, as yeah, you're guessing, but it's a data-based guess. But those are, you're looking at those services from the group of individuals that completed college, and then whose cases ultimately ended up being closed or just completed college.

 

back to top

 

>> No, who got college. So these were people with traumatic brain injuries that got college and either were or weren't successful, and these are the services that tend to kind of predict -- >> You wouldn't get job placement or job search or on-the-job support unless you advanced towards employment? >> They would typically come towards the end of the case, right? Yeah. >> So use them. You want to be careful about that. Because they're already [inaudible] >> That's [inaudible]. >> You wouldn't predict something other than the, say, the after effects of somebody completing [inaudible]. Now I know why I had such trouble in school. I didn't have a reader. >> No, yeah, no kidding. Yeah.

 

back to top

 

>> That's, I think, now when you think about it, that's really interesting because it could very well get to what they're greatest need is in college and meeting them directly through having somebody available through that system [inaudible] one of the key issues. >> Notice Yvonne said that [inaudible] that they don't use. >> Yeah. >> I've actually [inaudible] that accommodation is not there but here it's [inaudible]. >> Well, it's not significant but the odds ratio is interesting. Maintenance services is significant, and I thought that was kind of interesting. It's a service that I think counselors are oftentimes not necessarily all that eager to provide, but it does seem like in the cases in this instance, and this is a little bit of over interpreting, but if the counselor was, you know, maybe it's an indication of the counselor/client relationship and the amount of trust that's involved there or I don't know exactly what, but when the counselors provided that service, they're 1-1/2 times more likely to get the degree.

 

back to top

 

>> Well, and it also probably speaks to the fact that if their financial means are addressed, they're more likely to be successful. >> Probably persist, yeah, right. >> [inaudible] go to work, right? >> Right. >> [inaudible] understands being kind of [inaudible] about this that I am but, would you have a odds ratio of 2.3 it would not be significant? >> I'm guessing it has to do with the small number of folks who actually were provided the service. >> [inaudible] power to identify? >> Yeah, I'm thinking this might be a service that was -- I mean, it's pretty low incidence service provided, right? It's not really commonly used, and so I suspect, you know, the five people in the sample that got it, probably most of them had good outcomes, but it's just -- it was just so little, yeah.

 

back to top

 

>> Do you have any? >> Yeah, I can figure it out. And I think when we do the publication, and we were talking about this yesterday. I think we will identify how many people actually received each of the services, just so we kind of know, you know, across this relatively small number of people, you know, like just to be able to answer question like that, maybe only five people got, you know, maintenance or these other kinds of things, and it kind of further puts it into the context, and maybe we'll do that for NCRE too. >> Yeah. >> Yeah. >> Right. I don't know. Should we put in the publications? >> We can do that if you want. We're running on time. So if you have to bail, we understand. We understand. Thanks for being here.

 

back to top

 

>> Yeah, so if you guys can stick around for five more minutes and also for everyone online, we've done a lot of this already, just kind of talking about what do these findings potentially mean? You know, one is that -- and this just gets to your point about looking at regional differences, state differences, and then with the new data, the 2014, being able to look by ZIP Code, that if we look at the power of programs like Workability, because when we look at those things that seem to have like the positive effects in terms of significant findings and the odds ratios that really predicted positive outcomes, it really corresponded a lot of ways to what Workability does, and I think that's one of the things that Mark and I are really going to try to stress at NCRE, because a lot of the people, you know, attending our session, they won't have like a Workability program in their state, and just talk about the value of that, and I think for like further analysis, we could look at, you know, California because of the Workability programs. Now the issue might be that we may not have enough people to be able to do this kind of analysis, because you get into a problem of physical power which is sort of like this issue that we just talked about.

 

back to top

 

Where there may not be enough people to do that kind of comparison, but I think it probably does have a role, and so like when we talk about future program development, you know, we're going to recommend programs like this to really incorporate some of these services, it really seemed to make a difference you know, we talked about some of these limitations in the data. Again, nothing you can really do about it, but we're not going to try to directly address that in terms of the recommendations we make. And then, you know, I think, we don't have this on -- or we do have this on here, this issue about some of the racial and ethnic disparities. It is something that has to be addressed by the profession, and I think we'll talk about it at the presentation. You know, some of these things are really outside the control of the counselor in a one-to-one type of way but in terms of policy and getting more people into the state VR system, I think there's a lot that can be done to, you know, to get more people that really matches the population of people that have this disability.

 

back to top

 

[ Inaudible Speech ] >> The question there was we mentioned Workability, but we didn't mention college to career, right? >> Yep, yeah. Yeah, well, the thing, you know, the thing with college to career, because it won't include people with dramatic brain injury, and that analysis is focused on people with intellectual disabilities, and autism, but I think if there's, you have programs like that, generally, the concept of it, you know, could be, perhaps, something that could be introduced for dramatic brain injury. You know, the idea of having educational job coaches. You know, like you have in C to C, you might make a difference. You know, because, like, that one variable that we found, the readers, it's not exactly like what an ed coach would do, but it's kind of in that same general area about providing more of that direct educational support that the individual might need. Anything else, Mark?

 

back to top

 

>> Yeah, I think we could go over in terms of limitations, I think we Artie talked about most of those in terms of the data set not being able to do a whole lot with that. The only thing that I thought I would probably add is with respect to kind of looking at the services that were in and the services that were out. The kind of procedures that we used. If there were two services that sort of accounted for the same kind of -- explained in the same variance in terms of particular outcomes, that analysis will kick one of them out. So just because something wound up not in the model doesn't necessarily mean that it's not an important service with respect to college and university certificate and degree completion, but that another one of the services and there kind of covered the same territory. On a different day, it could wind up in the other could wind up out. So I would avoid kind of interpreting anything that's on the outside of that model. I wouldn't look at that and say, "Oh, that stuff's not important to college degree completion." It could be.

 

back to top

 

>> All right. Well, thank you guys for being here. It really helps. >> Thanks very much. [ Applause ] >> Thank you. For you guys online -- >> Yeah. >> So I'll contact everyone online and also here about the CRC units, and so you'll just be getting an email from me as far as how that's going to work, but thanks to everyone online and here too. >> Yeah, thanks. >> We need to figure out a way to send donuts electronically. [ Inaudible Speech ] No, they don't have it ready yet. Meredith is going to have to send that out. Yeah, but I'll write your name down as being here. So you'll get it eventually. >> You'll get credit, yeah. >> It's only an hour, and it probably wouldn't be that significant, except for everyone wants to get credit. >> Oh, yeah, like [inaudible]. [ Inaudible Speech ] Oh, no I did. I know how to do that.

 

back to top