Video with full transcript
While scientific and psychometric measuring for “fit” has existed for decades, it remains very hard to sell into big organizations. This panel led by John Sumser debates the tried and true vs. the innovative and unproven in assessing talent for the Future of Work. Panelists include, Alina von Davier, PhD, Senior Vice President, ACTNext, and Josh Jarrett, Chief Product Officer and Co-Founder at Koru, Inc.
I’d love to hear your thoughts about the discussion we had in the interview. Post your commentaries on HRExaminer.com or my YouTube Channel >>~ ATAGEND
Panel with full transcript
SPEAKERS Moderator: John Sumser, HRExaminer — Link to this video and transcript >>~ ATAGEND
Important: Our transcripts at HRExaminer are AI-powered( and quite accurate) but there are still instances where the robots get confounded and make errors. Please expect some inaccuracies as you read through the text of this conversation. Thank you for your understanding.
File Length: 00:50: 01
JS- John Sumser, Principal Analyst, HRExaminer AV- Alina von Davier, PhD, Senior Vice President, ACTNext JJ- Josh Jarrett, Chief Product Officer and Co-Founder at Koru, Inc
00: 00:09; 00- 00:00: 12; 27- JS I’m John Sumser, I operate the HR Examiner, thanks for coming and we’re going to do some introductions that’s going to include everybody in the room. We’ll began with Alina. Why don’t you introduce yourself.
00: 00:28; 02- 00:00: 39; 18 | AV
Sure, thank you John, Hello everyone, I’m Alina von Davier and I result ACTNext, which is the business invention and research and growth division at ACT. And my background is in mathematics, and constructing mathematical models for learn, for assessment, for classification, and for prediction. I think that’s good for now.
00: 00:40; 29- 00:01: 01; 03 JJ That’s great, I’m Josh Jarrett, I am the co-founder and chief product officer of a company called Koru, K-O-R-U, we’re a Seattle based predictive hiring company. So I have I have one-thousandth of the mathematical training of Alina but I try and connect the dots genuinely down to the users and how people employ data to make better decisions.
00: 01:03; 22- 00:01: 33; 10 | JS So one of the things that’s interesting about about about these two is this conference has been traditionally in edtech meeting over the years, so there’s a lot of warmth and inclusion over the ed tech side and this is part of a stretch out into the human capital marketplace.
These two represent where that Venn diagram overlaps. It’s a pretty interesting thing. So we’ve got an academic opinion of what appraisal is and the commercial opinion of what assessment is. And. That’s what we’re going to mine over the course of the conversation.
For starters would you mind because we’ve got period. Tell me your name and what company you represent or how you got here. Because, the audience is small and we may as well know who we’re talking to. So let me start with you. Would you tell me your name and who you work for. Thank you
00: 01:58; 20- 00:04: 37; 24 Audience members introduce themselves to the panel.
00: 04:38; 14 – 00:04: 59; 07 | JS So Alina, what does your company do?
00: 04:59; 08- 00:05: 14; 09 | AV So, ACT is known for its college admission test mainly but ACT does more than that. Then some of you might have heard the proclamation we made today ACT and ASU are partnering, so that was the proclamation at 1 o’clock when I left you guys.
04: 59 | JS What does partnering mean?
00: 05:00; 10- 00:05: 41; 07- AV So we plan to collaborate on research projects and new products. So we will work together as an extended group. We want to put together a center. And we will work bringing in students postdocs coming up with new research projects. And as I said before products hopefully.
It has not yet been all you know outlined. It’s going to be some work to do to figure that out. But this is just an example about how ACT as a company is changing so ACT has been around for about 59 years. It started as you know as a disruptor, pretty much, Lindquist who was a professor at the University of Iowa devised the automatic reader of, for multiple choice test. And with that he pretty much disrupted the route the assessment took place. He also proposed that the test for automation at the university should be based on what students learn so that the students can actually do something with the information they receive from the test is not merely a score so they can know what we improve on as opposed to other tests that were more aimed as aptitude. And called aptitude at that time. Meanwhile the S.A.T. itself transitioned quite the plenty. And there’s also more curriculum oriented but at that time it was not done for many many many years. There was this difference between the ASU details that was more curriculum oriented and S.A.T. that was more aptitude oriented. So over the years the company became quite successful to the phase of becoming the lead in the country of admission polish. That’s about 60 percentage or more of the market is of a city. Now we got the new CEO about two years ago and he is changing the direction of the company and is moving it into learning and into becoming an ad tech company. And that’s of course a strong claim to make. And but he definitely. Talks the talk and walks the walk pretty much one of the things he did among other things of course but since it’s about me now he hired me 18 months ago to lead this innovation group and I put together a squad in this 18 months and two at the conference of the parties as well as at last week’s seminar in New York City at they are in CMG. We had our first presentations based on the work and results that we are genuinely conducted in different groups. The approach that we take for learning is to based everything on our. On our strengths. So we what we know to do best is measurement and assessment. So for us the question is how can we distinguish ourselves from all this mass as you can see here of people who act in this learning environment. What do we bring to the table. So our learning components are within the evidence that design framework which is quite familiar for those who have been working in assessment for years. So that’s pretty much what we do. We work our we have I have a team on artificial intelligence and one an advanced cycle metrics and another one on learning solutions. And together we try to build these systems and we do foundational research in developing models for how people learn how people forget. How people deal with scaffolding and with hints then we have an artificial intelligence team that is working on packing content instructional content as well as. Assessment content and the learning answer person is driving the how and the what and the why. So so is your work still focused on. The sort of.
00: 09:29; 16- 00:09: 32; 19 Academic surrounding or the issue of structuring.
00: 09:32; 19- 00:10: 25; 22 We are definitely stretching out. I entail in the working group we also develop new products. We have several research based prototypes at the moment. We build an educational companion. The idea is to have an app that accompanies a person from younger age to the workforce and can provide support the replies and advice with respect to everything educational which also includes social emotional learning abilities. So yes the short question the short answer is we are also developing prototypes. However it’s an at the moment in the structure of the company. It’s not us. We’ll bringing those to marketplace it is the new product growth team that is working on that. We also develop capabilities that we provide to other existing products.
00: 10:25; 22- 00:11: 07; 07 For example we have a patent now on a new cognitive back gnostic model that accounts for the behaviour analytics based on the data that we collect from the interactions of the student with the system. So this became in itself a capability that can be used by other existing products in the company for example cognitive diagnostic model for the ACP test or for the work is exam or aspire and so on. So roughly the the mark of excellence of your work is so rural that we have to measure that shows you the assessment target correctly
00: 11:07; 29- 00:11: 14; 15 Or learning nation. So if we work through yeah.
00: 11:14; 16- 00:11: 34; 03 So I believe the I think your tale illustrates your Venn diagram and that we start from the business problem first and then run backwards so the core business probably work on is that hiring is basically violated which is that it’s sort of the definition of madnes we do the same thing over and over and expect different outcomes.
00: 11:34; 06- 00:11: 52; 11 Think of campus hiring for example we go to the same 12 schools we fight with our competitors over the bottom third of candidates in those classes. We screen out 80 percent of people because they don’t go to the right school with the right GPA the right major didn’t work at Google Facebook or Amazon before
00: 11:53; 07- 00:11: 58; 23 Then we are surprised when we have turnover we have performance issues and we have diversity issues.
00: 11:58; 23- 00:12: 24; 23 So we really took that as a problem both of access opportunity from presidential candidates side and business problem with real or ally and fiscal incentive to solve it. And as we look to how to solve that we talked to employers and hiring managers and said OK be said about the best hires you’ve made. What were the attributes how did you know what an experience they had work with. Where they studied.
00: 12:24; 23- 00:14: 12; 17 It kept coming back to. It wasn’t about that. It was do they have the grit in the run ethic. Do the government had the teamwork skills. Can they guess on their feet. Are they actually curious. Are they accountable. They follow through. If they have that stuff will teach them the tactical skills if they’re that kind don’t care where they went to school. We use all these things for proxies. Increasingly people are figuring out or predictive. So Google did a big research study. They determined that GPA was worthless as a hiring criteria. So utilizing that info we’re really the banks. Okay. How do we start to look at bio data which is a fancy word for. Stuff about people being you know where do you work. What classes did you take. What leadership stances you hold. What prior work did you have. That’s that gives you some predictive power. But there was a piece that was missing that’s humanistic traits we call them impact skills. And there’s seven that we focus on grit rigour impact teamwork curiosity ownership and polish. And we said that’s what hiring directors want to measure. Can they be measured. So then we get back to the research. Eighty five did research studies other people validating work in terms of sub competencies that roll up into what those hiring directors were saying. Leveraging I spent seven years at the Gates Foundation before co-founded crew. So I expended a lot of Bill on what there’s money and a lot of my hour I try to answer these questions of what are the things that that could predict success beyond GPA and where you score etc .. So we mind the best that the investigations and married it up through a 20 minute online tool. That is one of the inputs into a machine learning driven predictive hiring. So what’s excellence when you just say one thing because you mentioned the Google stuff. And
00: 14:13; 02- 00:14: 56; 21 You and I have talked about wrath on the phone. So at the end when you talk and I think it’s fascinating what Carl does. And now at the end you said you are looking at all these skills that are necessary in addition to the GPA. And I want to emphasize that in addition to and here is why the Google study regrettably is flawed psychometric study. And the problem is the following. The people that they invest in that field for which they checked the team war skills their ingenuity and so on we’re already very high on their STEM qualities. So psychologically speaking that is. It’s called a ceiling effect.
00: 14:56; 21- 00:17: 12; 03 So there was no body Asian in that particular body. But that plus reflective of the technical skill. So when you do a traditional value did this study if. You know people have not had the exposure to psychometric analysis they might tend to see no evidence that that particular body up one is relevant. But the truth is that’s the most important valuable and merely after that you’ll have the financing at once. So actually I did. Comment on that and I recommended Google to hire psychometric shots because. It’s a problem because you know people will want to see that. Oh always come from Google. So it must be correct. So I wouldn’t comment. This happened. I am I work with the working group from at the heart genuinely which is a university activity that. Is highly it’s an elite university and they’ve been using profiling trauma to screen for their graduate students. And it turns out that the Jerry Moss didn’t show up to be relevant to talk but once we stopped looking at it it was because everyone. Had an almost perfect score on the Jerry moth so if everyone scores very highly or very low or whatever the deviation is very low on a specific variable then that variable won’t appear as a valid and predictive tool. And that’s a you know that’s that’s why you need psychometric shoots to help look at conditional distribution and conditional predictive value. So we found out that everyone had a great math rating. Therefore we had to look at other variables that would be identified. And. Colorful scores. In that case what extremely relevant. But if you merely prefer by total score because that plus the variable that indicated the highest predictive value then they wouldn’t have been able to do the programming for which they were brought into school. So personally I’m very concerned and this is about the title that we have here with not all glisten is gold. That’s a good example for that. Yeah there’s Google.
00: 17:12; 06- 00:17: 24; 02 I think it’s a great point. And I think that they’re here if. We trust data it’s the truth. Despite flaws in the quality of data selection bias and survivorship bias.
00: 17:24; 10- 00:17: 36; 13 And we have to be very careful drawing absolutism spin and constructing these big leaps of inference however.
00: 17:36; 26- 00:18: 04; 04 One of the things we like to say is grit over grades and what that is reminding people is that relying on singular variables particularly ones that carry a significant amount of bias is a bad style to do hiring. All right. So if you were to select people only on there where they went to the exclusive school right. You probably that actually “wouldve been” illegal in the United States to use that as a single mother variable because that carries disparate
00: 18:06; 06- 00:18: 10; 05 Against productive classes. All right. That’s just it’s a that’s a mathematic I entail it’s just a
00: 18:10; 20- 00:19: 20; 02 Fact. So what it reminds people is how do we guess more holistically about candidates and holistically about how we hire. So we say great of a great school I love that. I love that. You know and I had breakfast with Angela Duckworth this week and she was like remind recollect Josh grit and grades are called right. Not surprisingly I say yes. But what we’re what we’re saying is reminding people to look more broadly and then we build predictive models that are context specific and that gets to the point of these broad presumptions. You know there’s really dangerous. We build generalizations from one data set to another one context to another but we build local models on that decide and I can say that grit is not always predictive and in some cases grit is negatively predictive. You can have too much grit. You can be too narrowly focused you mean not accepting of other people’s opinions. Can we not willing to be flexible and grades sometimes are predictive. And then we work with our clients. Do you want grades to be you college. U.S. News college report ranking is predictive here. Do you want that to be included in your model going forward. You want to give Harvard kids credit an advantage or you want to take that out of the model and just use these other variables.
00: 19:20; 02- 00:20: 01; 06 So I think it’s an art and a science and it’s not for the swooning of heart. I think so. So as I listened to your talk I think here one of the purposes of part of something that every startup is going to face that every practitioner is going to face here and that is that is there there’s a there is a theoretical purity that you could attain and there is a practical application and anytime you do for the practical application you students or theoretical purity and anytime you step in the world of zero purity you end up with a whole lot of people who don’t understand what
00: 20:01; 14 – 00:20: 17; 21 You’re talking about. And so so as you take your work out into the world both of you how do you deal with that. Well you see it because because there are vastly more people who don’t understand what you’re talking about to. Do
00: 20:17; 28- 00:20: 32; 01 Right. Well it’s also what the bar what do you consider that the quality success. I think that a leading product because I love everything I mean is doing and I’m a big supporter of it but I’m not being provocative because why are we here at 4:00.
00: 20:32; 10- 00:20: 35; 27 Let’s get the boulders here.
00: 20:36; 01- 00:21: 20; 04 But I would say I guess our bar is quite different right. My bar is. Can I help influence a hiring process so that it creates economic value for the organization a better economic opportunity for the candidate and is less biased than the process has been used today. And anything that moves in that step in the right direction is good. And if imperfect process get people saying you’re right we’re all higher with our gut. This is ridiculous. I hire athletes. You hire people who work or you know. Head of the marketing club. And we can get away from that start using data and start holding data back up to us and start moving the needle slightly better than it was yesterday.
00: 21:20; 04- 00:21: 25; 05 That’s my threshold to got to get that’s probably different than us.
00: 21:25; 06- 00:23: 56; 11 Well I entail communicating about what we do is definitely challenging with some groups of people and straightforward with other US as always. So I’m trying to. You know do to communicate with those that understand the route we talk directly and then work with communication experts to help me translate some of our technological verbiage into something that is digestible for the other people in terms of stunned that I’ve been actually my dream is to create abilities that will help people like you like cordwood with companies like Horowitz and many others here too without the need of getting into all these details but to be able to use this capabilities for a more accurate predictions as our CEO also said today. One of the things we work on and that again differentiates us because we are a large and not for profit company. We want to be very transparent. So what we do we will publish our methodologies. We don’t propose black boxes with John Dowd you Well just trust me because I’m Lena. What we have tried to do is to write papers that are peer reviewed and that published out there. And once we come to any of you with a commendation and that capability we can say well check it out. They are out there and you are able to kind yourself an sentiment. But we also try to be very careful at building interfaces. So we work with you decorators so that even if Al Gore is this really complex they use up doesn’t need to know that that is what they’ve established in order to second order when using that capability. So we want to make it as self-contained as possible. Our dreaming is actually to become part of the people solutions because we realize that what we do is I don’t know what else can I compare with. Take your phone for example I have assured it. It’s the same thing. You real the the algorithm for Siri to speak with you but you don’t need to know the maths behind that algorithm. So it’s exactly for us. We want to create those. I’ll go and give it to companies like us and work with any one of you will be your first testers.
00: 23:56; 15- 00:24: 04; 11 Wonderful let’s say and I would say to your point on your design I know John this is an area of a lot a lot of passion around too.
00: 24:04; 13- 00:25: 29; 02 I think we learned very early that year that a black box doesn’t help a process where humen are engaged. So if we could outsource the decision a hundred percentage of black box is fine. Let’s run it consider whose black box is better at the end of the day we pick the one that performs better. These are human processes and most things that apply machine learning AI it’s going to be the human cost the machine that beats the machine alone or the human element. And so that entails it’s a it’s a human it’s a human computer interface challenge. And so we will even even ship less predictive models because they’re easier to understand. So just saying Josh is perhaps two it’s not helpful. I need to be able to unpack the black box and say because remember we did the research six variables were most important. And Josh is is hitting four of those six as potential imperfections on the remaining two. And then you continue to follow that down all those imperfections. Here’s how to probe those in an interview various kinds of close the loop and satisfy that decision maker where they are. We like to say is that our goal is to help build these people better deciders not ever make a decision for them. And that’s a high bar when you have a distributed process moving very quickly a recruiting team is a it’s more like more like a factory floor than you know than a job shop.
00: 25:29; 05- 00:25: 32; 22 And so it’s a big big design challenge. Absolutely.
00: 25:32; 22- 00:25: 48; 20 And like you said it’s going to the data is going to float further and further to the background when you log into Netflix to get your recommendation feed. You don’t see it a oh here’s you know here’s Jumanji. It’s got a thirty two phase seventy five three to fit with you.
00: 25:48; 20- 00:25: 55; 09 It says Hey because your kid watched something else you know we recommend you much.
00: 25:55; 23- 00:26: 09; 22 So so assessment has almost universal acceptance of the education marketplace in industry. It hovers somewhere around 40 or 50 percent and the rest of the people who do
00: 26:10; 10- 00:26: 19; 20 Use assessment think that it’s food. What do you suppose explains the difference between the two worlds. It’s more than simply that.
00: 26:20; 09- 00:26: 25; 26 Well there’s a couple of things I think. One is there’s legal ramifications. We
00: 26:25; 26- 00:26: 52; 16 We have a set of employment statutes that are very clear about protecting employment rights and they’re actually stiffer on the front end in many ways than firing on the back end. And so in places like the EU and Australia other places where it’s much more difficult to get rid of employees that aren’t a good fit and lower bar to its implementation of increased protection on the front end. You insure a lot more assessment adoption because the cost of a bad hire you’re stuck with it.
00: 26:52; 22- 00:27: 33; 03 I think that’s one I suppose presidential candidates experience is another one particularly as we get much more in passive nominees and connected in. I’m reaching out to you. Everybody’s kind of constantly in this search like you know 80 percent of a candidate at 80 percent of employees are. You know. Of open and available the idea of throwing up a old school evaluation attains you feel like you’re back at high school. The bubble chart a number two pencil is pretty off putting. It’s really to create a win win experience and then I think there’s a sense of I know what I require. It’s the hippo versus the geek. I’m the expert. I’m a higher expert for my team.
00: 27:33; 10- 00:27: 43; 04 And so yeah we should reduce bias it’s all time. Hey I’m really glad we’re working this project will help reduce bias. Yeah all those harmony is their bias. You should certainly work with them right. They need to figure it out.
00: 27:43; 04- 00:27: 48; 05 No not me right. So anything you’re implicitly telling somebody they’re not good at their job.
00: 27:48; 08- 00:29: 36; 06 There’s resistance and I agree with everything you said. I only want to point out a few examples where the assessment is actually employed and you wouldn’t want it any other route. So think about. The lawsuit that just happened yesterday the pilot. Who. Brought the plane in you know under those circumstances merely saved the life of all those people that were left there. So in order for her to become a pilot she had to be tested and you wouldn’t want it any other route. You would want your pilots to be tested. And you want to feel comfortable that they passed. Similarly you want your “doctors ” to be tested. And they are a lot. Not merely that the objective is tested once they start but the objective is tested regularly. I know every five years every eight year. Again you would not want it any other route when you go out to choose your doctor you would rather want to know that those doctors have been certified. Similarly with other humen the military if you got into the military you need to pass a test. It’s actually one of the oldest tests we have in the country is to ask for the military entryway exam. And I can come up with other examples but I guess I already communicated that there are jobs that have a really high. Level of scrutiny. And that of course the legal requirements are attached to that. But not only the legal one is just a professional level who are in need of is so high and it needs so much precision that you. Do expect them to be to be tested and you want people to be certified.
00: 29:36; 13- 00:29: 45; 20 So what do you think John we observe this space. Let’s see what else did we miss. Sure there’s a couple of things that you would you have softly accounted for the 50 percent of adoption
00: 29:46; 29- 00:29: 58; 24 In places where the lawyers will get you when the plane falls out of the sky. If the if the welder isn’t certified properly that’s all. That’s all very well done.
00: 29:58; 29- 00:30: 28; 21 When you’re talking about people in non-life threatening non security situations the process of installing assessment means that the organization has to make a mindset change and it has to adopt a way of thinking about people and the language associated with that way of thinking about people. And that is an extraordinarily difficult organizational transformation to make happen. It is by leaps and bounds.
00: 30:28; 28- 00:30: 41; 06 The biggest failing phase in the expansion of evaluation as a tool because assessment isn’t just a test appraisal is a framework.
00: 30:41; 06- 00:30: 56; 03 It’s a way of is speaking to things. It’s a way of thinking about things. And if you and I agree on this and Josh doesn’t understand what we’re talking about the assessment adopted because Josh is going to perpetuate it if you say that
00: 30:56; 14 – 00:31: 26; 29 Grid is the thing in our organisation and you don’t evolve processes to cultivate and reward and incent that and genuinely stimulate the assessment system stick it doesn’t it doesn’t. So it builds off the exhorts and it can be perfectly wonderful science but perfectly wonderful science that isn’t designed to be held by an organization with its organizational procedures so it fails.
00: 31:27; 17- 00:32: 24; 15 I think it’s a good point. There are tasks for which probably we the interview that’s not quite qualified that’s an assessment. Of. It. It probably also depends on the type of job. Even for a company like ours if you hire research scientists for example I’m sure you don’t put that person to a test with multiple choice but you respect that person to give a presentation and that presentation is being rated so it is in a way a performance evaluation if you want at least at the research level. So for other jobs probably where you don’t have a lot of hazard you you can be content with recommendations and presume. But it depends on how. I still believe that it depends on how much. Responsibility there will be on that person and how much how. What type of legal repercussions will be sure.
00: 32:24; 18- 00:33: 02; 25 I suppose many of the people who were in this room during storybooks care about organizational fit did organizational fit kind of transcends those arguments. It’s the assertion that you can somehow without perpetuating discriminatory biases figure out what attains somebody stick in an organization versus what what doesn’t stick inside of an organization. So it’s less of a I think that’s the heart of what you’re doing is trying to. Make sense about whether or not I’m going to get fired two weeks into the job because I don’t belong there.
00: 33:03; 03- 00:33: 04; 14 Yeah I think that’s right.
00: 33:04; 14 – 00:33: 12; 06 I think you’re right on both those issues. One being that culture fit is better a shield for
00: 33:12; 27- 00:33: 21; 29 This person is not like me. And hurting our diversity inclusion and belonging run.
00: 33:21; 29- 00:33: 34; 08 And so I think that we try and tie these things not as abstract constructs of how people are or they like me but instead do they demonstrate the behaviors that are required to succeed in the job.
00: 33:34; 11- 00:34: 02; 09 So while you know teamwork is we’d like people who play nice with others it’s like what is like this specific action the observable behaviour that this person may have done at a previous undertaking that they will they will report being good at. How do they resolve conflict on a team. Do they describe others out into a dialogue. Are they the person that gets asked to present findings. Right. Those are tied to the job itself.
00: 34:02; 09- 00:34: 31; 26 And I think if you if you hear people talk about cultural fit things and you can’t map them to bullet phases on the work description you’re at this abstract level which is at risk of remaining up on the big letters on the wall or letting itself go down into a culture fit place. We actually get people talking about values of it. Which is a different way to try and get beyond this black box of culture and in hiring scientists. I’m definitely more
00: 34:32; 18- 00:35: 30; 13 Focused on value fit. And then. By any means fit they find a different way. I was wondering what do you think about transplant because in my job as a administrator of kinds of lead that often research is define genuinely high end such as. I was often in the responsibilities of hiring people from all different disciplines and bringing them into education. Brilliant people and with many of them after many many dialogue I still had the impression that they could then construct the transfer sort. Even in terms of start these things. Talking about the data on the models instead of talking with the application in the same methodology and talking about the application into the learning process. So many people don’t seem to be able to make that. Jump. I was wondering that
00: 35:30; 19- 00:35: 36; 06 Does coral look at transplanting or do you have other body stuff that get a transplant.
00: 35:36; 16- 00:37: 16; 21 Well I think we would. We would want to see that show up in some of the variables that are correlated with success. Sort of I think that the data would tell us is only three things that predicts someone’s success a job correctly or incorrect as “youre supposed to” know that’s a much better idea. But what is cognitive ability. Right. There’s good connection at least to a point. And for a lot of jobs to his technical skills you do have the hard skills the stats or whatever. The third is the all else bucket right. We call those impact ability soft abilities personnel whatever those the situation was. And so the question to be if somebody is not making that transfer. What is it. Was it A. Was it a horsepower issue was it the SATs or there’s there’s other areas. So. We’re looking for proxies for our assessment is really trying a proxy. It says it’s you know how close can you get at direct assessment is still a close proxy as opposed to a far proxy. And so if people been successful there in the past have all worked in education before right or having worked in education increase the probability that being a fit here we should see that come through in the bio data which is things we haven’t talked much about is this Let the machine find the determine the correlation without any research based explanatory power. And I think we try to combine both of those because to try to take advantage of the big data used but not just say that know left handed people that post on Facebook on Tuesday nights are better researchers so therefore that’s the profile. So are they right. I don’t know. I believe I think there’s that would be one way that we do either of you work with a variable that is sort of the capacity to fell one reference frame for. The
00: 37:16; 24- 00:37: 30; 04 Reference one of the things I know in this transfer process is that all organizations have a point of view and they me harness the same technical knowledge and genuinely harness the same softer skills.
00: 37:30; 04- 00:37: 47; 08 But that point of view may have customers first it may have employees first it may have the value of cost savings for us but there’s a whole scope of things inside of the inside of the new environment that differ from the old surrounding.
00: 37:47; 09- 00:38: 05; 02 That’s where you get the hiccup. You get people who can’t stimulate the switching for perhaps genetic reasons and perhaps Shorten should be the worst firing was too brutal. You don’t really know. You do have variables that address that.
00: 38:05; 09- 00:38: 06; 28 They do change.
00: 38:07; 19- 00:38: 38; 22 So yeah. So the two that are relevant here in the literature. One is openness to change. So that is that willingness to listen to different points of view being OK. You know who moved my cheese. You know that that’s not going to totally derail you. And then the others quick learner. If you have those two attributes that idea of kind of learning agility the willingness to take on a new challenge. To have your preconceived notions challenged and change them and then promptly adapt.
00: 38:38; 23- 00:38: 53; 19 So I was on a panel with a woman from Cisco a couple months ago and she said learning agility is the new smart-alecky. It’s not about horsepower it’s how quickly you can take in these new things. So those are two of these things that we look at that are connected. Yeah
00: 38:54; 15- 00:39: 19; 28 I entail I definitely is is a challenge for many companies. Everyone is talking about change and we are changing all types of things around us and we expect the employees to notification to actually “ve learned that”. That is why it change in the way they run. And then the skills that they need to build there is an expectation that people will do that speedily.
00: 39:19; 28- 00:41: 27; 26 And I believe it’s hard for some. I haven’t quite figured out why. I mean I don’t I truly don’t think maybe it’s a different construct. It’s not pure ability. Maybe this this agility of learning that. Stands out as a separate construct in my suit I was actually talking about something that’s even more difficult. I was looking for people who came from other disciplines such as physics particularly those who studied physics ought to have exposed to modelling very complex large datasets and working with dynamic models and working in within this house theory. So where do you you know you develop a model that may change its state and my not my never converge to. And that can leave state and that type of thinking and modeling was important is still important I believe for what we are trying to do with learning because the learning process sometimes is totally unsuccessful and we need to figure out what external factors to come and interact and destabilize that process. So I’ve been looking for people with B HD physics to come and work with me in application to start modelling this type of problems and many of them had difficulties building the transfer between the data that they were using for physical experiments or from astronomy into the data that would come out from application as text for example from I know from the essays or as nation in intelligent tutoring system but as a mathematician if you stand back and you only write it as random variable it shouldn’t be such a big deal. I thought well it was I was wrong. So for many people that was a problem. So it turned out that in order to hire. I and I ended up hiring really really brilliant very good people.
00: 41:27; 29- 00:42: 39; 04 But I expended as a hiring director I expended a lot of time in pre hiring screening that were pretty much working on the board with them and it was nothing that if you would read in any book about how you’re hired at this level you don’t you can’t ever find that that you would actually view on working with them on the board like a school. But that’s what we ended up doing. And it turned out that a number of our side set for hiring at that level was an incredible sum of hour so I would love to hear more from companies like yours that would help me hiring at that level. I don’t think it’s something we can make automatic at this phase. I think it’s one of those places where artificial intelligence and human intelligence you know can Why that official one can. Quite compete with what the human brain should table. But that would be something I would really love to. Watch from code all that. Up high and helping the hiring manager hiring up high end expertise put on the product please.
00: 42:39; 29- 00:42: 41; 14 Let’s do a last word.
00: 42:41; 18- 00:43: 44; 17 One of the things that concerns me about about the ways machines are starting to control standards for hiring. When I include this is a broad category of standards for hiring. Is it the living rates the various kinds of serendipity that really constructs organisations work so. So if you say here is this profile and you gotta make this profile what you do is just establish a bar. That eliminates in practice the possibility of somebody who doesn’t fit the criteria. Coming in and excelling right and that is. Maybe a primary source of innovation in organizations. It’s certainly a primary source of learning inside of inside of innovations and so how do you how do you prevent. Good assessment testing from becoming a kind of a result beyond which you can’t take anything yeah.
00: 43:44; 21- 00:44: 22; 26 I think there’s really good research on positive deviants they they are deviant from the style they about the operating system in the organization but for good. It’s like the Robin Hood you know. In the company. So it’s something it’s something we think about we get questions a lot on that and I don’t have a perfect answer. But I’ll say a few things that we’ve done there is one is there’s more than one way to get a high school. Right. There is not a singular you fit this you get a good score set. As I said May 6 or perhaps 8 and 10 twelve variables and very few people are actually hitting the bet on all 12.
00: 44:22; 29- 00:45: 23; 24 So that you know there’s multiple ways being eighty two is we’ve actually built something into our candidate profiles this is why is this person unique. So it actually calls out things that are rare among the dataset. This person is a triple major and then hears of this three majors. This person than ninety ninth percentile in communication polish and then nobody was ever taken this test. So we try and flag that as a way to catch the attention. But wait a second I’m going to set them on this team where the whole thing is they’ve got to stand up and present the findings and conclusions if they can’t be believed that authentic that no one’s going to take them. So I think that’s you know I believe the other thing is that we also say seem the assessment needs to be guardrail in terms of how much of the decision it’s taking on. Sometimes people have said you should never use an assessment for more than a third of the data in your decision making. So it it’s true you give people a tool and yet be careful how they’re going to they’re going to exerted it. I utterly concur with everything that
00: 45:25; 29- 00:45: 30; 08 I determine on this one. I think so.
00: 45:30; 09- 00:45: 35; 11 Let me you a you which is which is where we’re in this. He wants us to fight normally.
00: 45:36; 09- 00:45: 40; 19 And we are learning every day. What we knew yesterday was wrong
00: 45:41; 13- 00:45: 54; 29 Copying presumptions that don’t make it so. So what do you think the report is. That there’s going to be a variable that pops out of the data other the big data flowing that upsets your apple
00: 45:55; 29- 00:45: 58; 13 Pie. It’s very likely.
00: 45:58; 19- 00:46: 15; 26 I think it’s quite likely especially now with the help of technology if we are indeed able to observe people in action. I think we have learned much more about them. I entail sure personal privacy and legal aspect might intervene and control a bit what they could do.
00: 46:15; 26- 00:46: 23; 29 But definitely I think that it’s a good thing that we are able to detect a new new feature.
00: 46:23; 29- 00:47: 23; 29 I mean even this one that we’ve been talking about. I still hope that we would be able to find what would be a good proxy to transfer what is a good proxy for creative thinking and I didn’t like what you said about the positive deviants because this is one of the scoring for creativity. So you want off the stand the test for creative thinking if you will ask people to come up with a number of solutions for using say one boring object that frankly is very common and then you would look to see what people came up with and the ones that have this positive deviance. So that has that hasn’t indicated up many times since they’re out there by itself. That is all fun. Find that when I was. Get a higher weight for scoring creative creative thinking. So perhaps transfer is part of creative thinking. So while. I’m sure that we will have four responses to some of these questions in the next year years with that technology.
00: 47:24; 17- 00:47: 45; 12 So I suppose the other thing is that there’s a lot of inputs that a lot of outcomes. What that entails is it creates multiple sources of truth and that’s really confusing and destabilising. So nobody read the article weeks ago. A couple months ago now where someone said their DNA to twenty three me. They said it’s like all the five major ones and they got back like you’re twenty five percent Native American and the other was like you’re twenty five yet.
00: 47:45; 17- 00:47: 50; 13 So the results were totally different. So as long as you only ask one of those you feel confident.
00: 47:50; 13- 00:48: 07; 25 But as soon as you have you send your DNA to two places you feel like you have no idea. And so it’s the same various kinds of thing you look at the day is it great. What are the predictors of retention. Here’s the that’s the surgery. What are the pretenses of making quota on a sales team. Oh wait those are different. And then the next cohort next year. So what what predicts quota next year. Oh what’s.
00: 48:07; 25- 00:48: 40; 08 Well that’s different too. So I think one of the things is we’ve been playing with this how do you how do you smooth the data. How do you create a rolled median. So I think we like stock price. It changes a lot and that sort of destabilising we like what’s the trend. And so rather than how do you add the information contained in and look at it and frequent enough standpoints so that you watch the evolution so that you don’t have these giant shocks where you say yeah we added you know 100 more records and things have started to merge a thousand more records and so you know you don’t
00: 48:43; 05- 00:48: 48; 05 Close your eyes for three years and then things merge and then you realize you screwed yourself last year.
00: 48:48; 22- 00:48: 51; 17 So thanks. Thanks. I’m going to take my
00: 48:52; 16- 00:49: 14; 24 Bellow with information to try to summarize the thing that builds me happiest about this conversation. is we we’ve talked about assessment in the workplace for an hour and nobody claimed to eliminate bias, and that makes me happier than you could ever know because that is that is a singularly nonsensical thing that’s been percolating around the environment, you cannot do that.
00: 49:15; 14 – 00:49: 29; 22 | JJ Which you can never do. You can reduce it.
JS You can mitigate it you can reduce it you can do all sorts of things but human being without bias are not interesrting.
They’re tables. JJ
JS They’re tables, that’s exactly right. They’re furniture. So, So I want to thank you for taking the time to do this if you would join me in( applause ). Thank you. Thank you. It was wonderful. Thank you. That was
JS Thanks for coming in. If you are in the business if you would gives people your card that would be phenomenal. We will get you into the report.
Watch more of John’s talks on his YouTube Channel >>~ ATAGEND
Read more: feedproxy.google.com