Assessments Are Dead. Skills Intelligence Is Taking Their Place

[00:00:00] Kaffee in seiner besten Form mit der neuen Qubo One Kapselmaschine von Chibo. In jeder Qubo Kapsel steckt Spitzenkaffee aus besonderen Anbaugebieten. Für Espresso, Kaffeekremer oder Kaffeekrande auf Knopfdruck. Die neue Qubo One überzeugt mit Premium Design, kompakter Größe und kleinem Einstiegspreis ab 29 Euro. Dank innovativer Press Brew Technologie wird jede Tasse besonders aromatisch mitsamtiger Crema. Entdecke jetzt die Qubo Kapselmaschinen in deiner Chibo Fiale und auf chibo.de.

[00:00:44] Das ist William Tidcup und Ryan Leary und du bist, und du bist, und du bist, und du bist, die You Should Know Podcast. Heute haben wir Kian von Workera und wir werden über Assessments und Biasen. Und so wir werden viele verschiedene Dinge an der wir gehen mit diesem. So Kian, würde du uns ein favor, den audience ein favor und introduce dich? Ja, first, thanks für mich, guys. I'm Kian, ich bin der CEO und founder von einem Unternehmen called Workera.

[00:01:12] I'm also an adjunct lecturer at Stanford University, where I teach neural networks, deep learning in the Computer Science Department. And my passion is really at the crossroad of AI and what's happening right now in the workforce. The need for education, the need for reskilling, upskilling, you know, all of that I think is very interesting. And we need a lot of help and technology to make it happen.

[00:01:39] Ryan, is adjunct professor at Stanford a flex? A little bit, a little bit. But I need to hear of it. Is that SUNY New York or SUNY Oswego? If I was, if I was, if I even knew how to get to Stanford, I'd flex on it. I've been on campus. I can say that. I've done a tour. It's gorgeous. Absolutely gorgeous. Kian, William and I, we talk all the time about this stuff.

[00:02:07] And it's AI in the workplace and AI and what it's doing in work and all of that. Why don't we start by just what are you seeing out there? What's the trends? What are you seeing? What's kind of giving you that feeling right now? Are you thinking about heartburn or excitement? I mean, both, right? I mean, he's, he, I mean, you got experience on your own. You've got the students at Stanford. Like, well, what are you seeing out there? Yeah, that's cool.

[00:02:33] Well, I mean, a few things I would say, you know, one is the direct impact of AI. It's like, you know, the trends around are some jobs going away? Are new jobs being created? You know, the World Economic Forum has some of the best data in the world around that. They're saying by 2030, we expect still more jobs created than jobs gone by a margin of 78 million. But, you know, who knows what these jobs are and who knows exactly what jobs are going away?

[00:03:03] People understand that customer support is getting highly affected, impacted. But there's a lot more areas where we're not sure. And it's likely not automation as much as it is augmentation and part of the job tasks changing. On the other hand, what some people forget is the indirect impact of AI. The indirect impact of AI is that AI moves so fast. Things move at a crazy pace.

[00:03:30] And if you wake up every day and you look at the news, there's a new model. There's a new capability that may make change the equation for companies and for people in their careers. There is a metric that summarizes that called the half-life of skills. Half-life of skill is like how long is a skill useful in your career? You know, many, many years ago, you could learn one skill and have it for 10, 20 years and you'll be good. That age is gone.

[00:03:58] And I think today it's around 2.5 years in digital areas. It's very low, which means that the indirect impact of AI is we need to learn all the time. All of us, if we want to stay relevant. And so, you know, I think people oftentimes forget about that. Yeah. When people used to – when they'd introduce themselves and they'd say, I'm a lifetime learner, that was like a differentiator. Now that's like if you're not a lifetime learner, you're unemployable. Yeah. It's table-to-face work.

[00:04:28] Yeah. Too funny. All right. We want to talk about assessments. So tell us about the world of assessments as you're seeing it right now as well. Yeah. I mean, assessments, I would say historically are known as stressful situations. Like you're being tested to get in a university, to get in a school, to get in a job. And I think it's very unfortunate that it's known like that because actually assessments are so powerful for you as an individual or for a company.

[00:04:57] Because learning starts by learning about yourself and learning what are your gaps, what are your strengths, and how can you act on it. And so the whole reframing that me and my team are working on is to sort of reinvent assessments in a way that empowers people in organization to run better their workforce, to run better their careers, to make decisions that are fair and meritocratic. And so I think it's super fascinating.

[00:05:24] But really working against a history of assessments that have felt stressful and opaque to a certain extent. The IO psychologists, the folks that come out of SCIO, they've used assessments as a screen-out tool for so long. And Ryan and I have talked about this in the past of like, why don't the candidates get three things? An understanding of what's being assessed. Just like here's a broad overview.

[00:05:53] It's a behavioral assessment. We're trying to figure out blah, blah, blah. Two, here's how long it's going to take. You know, it's going to take – and, you know, we lie to them and say, it's a five-minute survey. It's a five-minute bit. It's never five minutes. So giving them a kind of a real understanding of what it's going to take, what it's measuring, what it's going to take. And then giving them a copy of what it means to them and what they can learn from it. Like there's – most of the market doesn't do that.

[00:06:23] Doesn't do those three things in particular, but for sure doesn't give them a copy of their assessment. Yeah. I mean, you're perfectly right. I would even say start with why. Why am I being assessed? What's in it for me? What are the rewards? If I'm hitting a certain score, what are the rewards? Because, again, people think of assessment as in hiring, but for us, we call it skill intelligence. We don't call it assessment. We call it skill intelligence because it applies in learning.

[00:06:50] It applies in everywhere where you're trying to, you know, get rewarded to display a certain behavior, skill, or trait. And so start with why. And then, again, you said what is being measured. That's called competency modeling. Modeling. It's the competencies that you're going to measure me in. And then going one level down, how it's being measured. Today, part of the revolution is that we measure with multimodal AI. So we have an interface that can speak to you.

[00:07:18] If it's measuring certain type of skills, it may ask you to draw something on a whiteboard. It may ask you to code. It may ask you to problem solve. It may ask you simple multiple choice question. The range of cognitive levels that we can measure is way more than what we could do even five years ago or 10 years ago. And that's revolutionary as well. How do we build? And I think you've answered this, I think, a little bit. But maybe go a little deeper.

[00:07:48] How do we build trust in assessments? And I ask that because I was with Kinex a long time ago. And I spent some time on the assessment side. And this was like eons ago where we had the IO psychologists actually getting on the phone to do the assessment with the person. So not an internal use of the tool itself. But nobody ever trusted. They never trusted them. The recruiters.

[00:08:18] You're talking about the recruiters didn't trust it. No, no, no. The candidate. Like that candidate. Yeah. Or even the employees going through assessments internally. But we weren't really measuring. We really weren't measuring skills internally at the time. Right. It was just screening people out. Exactly what William was saying. And I think that's built a lot of bad trust. Ill will through the years. How do we get past that? How do we build that trust? Yeah. That's a key question. And so two things. One, if I had to give you the TLDR, it's transparent design,

[00:08:49] continuous auditing, and then finally human in the loop to fix the AI system. So those are the three buckets, if you will. But now if I get more concrete, what are we working against? Like what's our bar? Our bar might be traditional assessments that are multiple choice based. And, you know, turns out multiple choice based assessments are very valid. And they're very good in many cases. People don't know it all the time, but they're very valid. They're just so dry and they're not reflective of the real world tasks.

[00:09:18] So people would say, you have a great trusted assessment, but does it measure actually the, you know, the construct that it was supposed to measure? So that's another question. But we can work against traditional assessment, or you can work against the human because the human is imperfect. You know, I tend to say there is a bug and humans have it. Like bias is a bug and humans have it because people tend to, you know, rate highly people that look like them, that sound like them,

[00:09:49] that believe in things that they believe in. So you're not working against... SHRM has, SHRM has seven defined biases, hiring biases that they, that they lump in. And I'm sure there's more, but SHRM, they've, they've literally got seven in their kind of knowledge, book of knowledge. And so they, it'll like me bias, the latency bias, all that stuff. So again, it's, there's going to be bias.

[00:10:16] And this is actually really something that Ryan and I wanted to talk to you about. There's human bias and there has been human bias in hiring since the beginning of hiring or the beginning of humans, depending on your point of view. So are we ever going to, even with AI, are we ever going to get rid of bias or does the bias change? I, I, I don't know. You know, honestly,

[00:10:45] that's a, that's a difficult question. I think what we know though, is I, I'm fairly confident. I could say very confident that AI is less biased than humans and is the, the partial solution to the problem. The thing is if, if someone is racist, they're not going to wake up a day and be racist. You know what I mean? So it's a long time, a freaking long time for people to change. AI doesn't take time.

[00:11:14] If you actually know what's the problem and, you go and you fix it, it will change overnight by definition. And so it's easier to fix AI than to fix humans. For that reason, we expect that, you know, as we're starting to fix AI more and more, it's going to pass the threshold where no single human, no group of human will be less biased than a, than, than an AI. On top of that, think about these large companies that have, you know, hundreds, thousands,

[00:11:42] sometimes tens of thousands of managers that are rating their employees, that are hiring people. Even if they're trained as much as you can on interviewing and on talent management, they're probably all running a different rubric, even if they are trying to stick to the same rubric. So, you know, even from a consistency standpoint, better to have one system than a thousand different systems, right? Yeah. That's the standardized interviewing so that everybody hits the same or

[00:12:08] similar experience so that you can judge them in the same or similar way. Right. Well, the, the, the method, the method I think needs to be standardized, but the actual assessment can be reflective of the role and the person that you're considering. So the topic of the assessment, your competencies might change all the time, you know? Right. Yeah. Oh, go ahead, Ryan. So the question I have then is when I would say when the machine, so when the AI,

[00:12:38] when the machine tells someone that they're not ready for the role or the promotion, who actually owns that decision? Is it the AI? Is it the creator of the assessment? Is it the employer themselves? Who, who owns that? Well, I tell you that I'm not an illegal expert, but you have a cast. You have a, are there? I need to waver before the question. Fair enough. Fair enough. I think you have different responsibilities for different layers of the application.

[00:13:07] So you have the foundational model providers that are responsible for certain safe safety measures on their models. You have the application provider like us, that's responsible for other things. And then you have the employer that's actually using the tool who's, who's responsible for what are you using it for? Are you being clear about the use case that you're using it for? And so, you know, that would require different levels of stakes to be vetted and validated. But, you know,

[00:13:33] if I think that employees or candidates should have a way to contest the result and you should have, that's where the human in the loop comes in and would tell you, you know, we we've reviewed your, your, your, your contesting of the result and we don't agree with you, unfortunately, or we agree. And we, in fact, you allowed us to fix the AI. You allowed us to make our system better for the next candidate that's going to come or for the next employee that's going to use that assessment for learning purposes. Right.

[00:14:03] I want to read something to you. Ryan and I do a new show called the barf. And this was a story that, that came up and I want to get your take on it. So seven of 11 AI superstars recently hired by Meta were poached from open AI. And this is a quote from the actual report at the heart of Meta's talent acquisition strategy is a direct personal and relentless involvement of the CEO, Mark Zuckerberg in a departure from standard corporate recruiting protocols.

[00:14:34] Zuckerberg has positioned himself as the chief recruiting officer for the company's AI ambitions. His methods are highly unorthodox for the leader of a fortune 500 company reflecting a sense of urgency and a belief that securing the right individuals is a task too critical to delegate. So that's a quote unquote, the source is Clover, K L O V E R dot AI. If you, if you, if you want to read more about it,

[00:15:04] uh, but can, when you hear that, what do you think? One, Zuck is smart. Whether you, he knows, he knows what he's doing. Um, so a few things. One is people get stuck on the amount that these people are paid, but the truth is it's a small fraction of what they're going to use for compute.

[00:15:32] Like when you're given a budget of X million to train an AI model, your salary is just a small piece of it. So to, to, to, to someone like Zuckerberg, it's easy for him to give more money to the researcher, uh, then give them like twice more budget, you know? So that's the one thing on the actual, uh, amount they're getting paid. The second aspect is, um, it's a, it's a talent war in AI and those skills are in shortage.

[00:16:00] You can count maybe on more than your fingers, but you, you can count the people who are able to train, um, a large scale algorithms and deploy them at the scale of humanity, pretty much, you know, hundreds of millions of people. And so, uh, they know who they are and they're going after them to bring that talent in, um, which, uh, you know, and, and he believes it's going to pay off over time and he has a vision for it. So totally understand now to bring us back to what does it mean for society?

[00:16:29] And can we expect that to continue? Um, I do think that it is healthy to get in a skills-based merit, meritocracy where your skills determine how much you get paid. And, you know, the same way we have to get used to certain skills, getting a huge premium at times, we will have to get used to other skills, getting not a premium and actually salaries going down on other skills. I think it may be shocking to a lot of people today, but in the future,

[00:16:56] it might feel more like the NBA or some sport where, you know, you do a great season and you're highly valued on the market and you do less well later on and you're less valued. And I think it will be healthy because it will create incentives for people to learn and to have that learning mindset that is lifelong. If you see what I mean. I was out for pay equity. We were just talking on a previously on pay equity and how to, how do we close the gap? And here we are.

[00:17:26] We're like, we go a hundred million more than you. because you're good. pay equity might become less about the final number and more about the skills. Skill equity. Yeah. Yeah. Because I think you could, you could still have like, so I think if the default is skills based pay, you're paid based on your skills. First, it requires a very good measurement layer, which is what we work on. Really good measurement layer that is trusted. Secondly, you can always have programs,

[00:17:53] which some may call like positive discrimination or others that, you know, call programs like universal base, basic income, which might give people the time to learn, you know, maybe your skill or descending in value. We will pay you to give you the time to learn. And society is comfortable with that. We'll give you the time to learn the next skill. Essentially. All right. Two things. Oh,

[00:18:20] two things that come to mind is how do we assess for potentiality? Potentiality and comma. How do we assess for character? It's a tough one. Sorry. I left all my easy, easy questions on the last show. You know what I'm saying? Both of them are difficult questions. I get it. And that's why I asked because someone might know when we're assessing them,

[00:18:49] they might not have that skill. However, skills could be taught. So training and development, learning and development, LMS, all of that experiential type stuff that comes behind it. If I can get a gal or a guy that has the potential to learn those skills, how do I, how do I know that? How do I, how can I pick that out of the list of all the people that have applied and, you know, that type stuff and then character. Let me start with potential. And,

[00:19:19] and it's, it's my answer is not going to be perfect, but I'll tell you what our customers do today. And, and, you know, we're excited about it, even if it's not the end solution. So today we have a metric that we measure called learning velocity. So you measure someone in January and you measure them in March, you compute the difference in skills, and that would tell you what happens. How fast they learned. How much they improved. Yeah. You don't, you don't care about what happened in the middle.

[00:19:48] Someone may have taken a podcast. Someone may have been coached. Someone may have learned a course. It doesn't matter. It's outcome based. It's the velocity. It's the velocity. Exactly. It's outcome based. So that's part of what might be a part of potential. Maybe someone who has a high learning velocity, you will have a higher confidence that they may have high potential for something. On top of that is the skill adjacency. So, you know, I might look at your skill today and say, the thing that the project that I want to give you,

[00:20:17] do you have the potential to achieve success in that? I will look at your skills today. That might be different from those skills. And I will look at the adjacency. Is do I know the connection between those skills to be able to say, I have a pretty high confidence that William, you're going to, you're going to be good at that project because I know you have a high learning velocity. And I know that your skills are fairly adjacent to that project. You see what I mean? Does any of that come down to personality?

[00:20:45] Like dealing with ambiguity, dealing with agility, dealing with change management, dealing with some of these things that we're talking about. Does any of that come down to, some people's personalities are just not fit for that. Well, I, I would split it in like, you can measure personality as well, but there's assessments that are for that specifically. We don't do much of that, but there's assessments to measure. Are you extroverted? Are you introverted? You know,

[00:21:15] et cetera, which, you know, is more of a self-assessment than an assessment. But so that might be measured. And then ultimately there, you know, there are jobs where personality traits play in the, in the performance on the job. Like, you know, companies have different opinions on it, but some companies say, you know, if there are certain traits of your personality right now in Silicon Valley, the high agency approach, like it's very hard to measure high agency. And that's why even in an,

[00:21:43] in an AI driven hiring process, you would probably split the skills or competencies that AI is less biased than humans at measuring. And you would actually let the human measure some of those where AI, we just don't know how to do it, you know? So, you know, probably the human is the proxy for the hard stuff that is more, you know, sometimes a gut check. Sometimes you trust the person. Sometimes you feel like the person would, you know, align with the values that you want them to have on the job. So, you know, I'm not an expert at that part,

[00:22:13] but I think humans will continue to focus on that for a long time for now. Do you see a world or a time where, and we've talked about this in the past, where candidates or employees, I should say employees own their assessment data and they take it with them almost like a verified, like a skills passport. So they leave company A, they go to company B, but they take that with them. Absolutely. And 100%. It's like a suitcase. Yeah. I think the idea of a,

[00:22:43] of a meritocratic version of LinkedIn that is not based on what you write about yourself, what it actually is. Verified skills is, is tremendous. And it's going to help a lot of people stand out that, that are, don't have a good way to stand out today. I think Ryan, I think if we, if we ever get to a point where blockchain becomes more normal than not, and having verified skills, their accredited verified skills will help a lot of people. It'll,

[00:23:11] because then they've got that little satchel passport, you called it. They've got that thing that can go to any employee, any employer, and it's verified. I think that's the key is the difference between what we have in LinkedIn is, is self-reported and, and gamed on all levels. And so it's having something that's very third party verified. And, and I think that's, that helps people. And I would tell you, yes. And I would tell you that,

[00:23:41] you know, in the early days of LinkedIn, it was, it was a little hard for people seeing companies, let their employees put their job on LinkedIn. And over time, over time they, you know, it became normal. Like, yes, you let your employees have a LinkedIn profile. You don't prevent them from having it. I think the same thing will happen with the skills passport, you know, you, but, but maybe with not every skill, when you join a company, there are certain skills that are durable skills that you show, um, that you will take with you.

[00:24:11] And sometimes within our customer base, they measure highly custom skills that can't be shared out there. Like you get tested on certain manufacturing processes that are unique to your way of work. That I don't think you'll be able to share everything, but I think you'll be able to share some of the skills that are portable, that are durable, that make sense across companies, et cetera. Non-proprietary, uh, things as, as well. Um,

[00:24:40] so I guess the question I want to ask is if someone's buying your technology right now, what questions should they be asking you? Like, cause they've, they've grown up with traditional assessments. This is different than traditional kind of assessments that they've dealt with for the last 40 years. So what should they be asking? What's the questions that they should be asking? What should they know differently than what they've done in the past? Yeah. Uh, great question. I would say, um,

[00:25:10] how do we deploy an assessment in a way that's empowering to our workforce? That flips the mindset of people, like creates change management. Um, I'll give you one example before every assessment, people self rate themselves. We ask them like, where do you think you are? And then at the end we tell them, you know, here is where you, you're actually are. And then that, that just, that actually creates change in, in their willingness to learn and their openness to change. So, you know,

[00:25:38] one question might be that another one will be, you know, assessments have been, I guess if you look at a company, they might have 20 different assessment systems with different methodologies that are running. Some are video based, some are text based, some are, um, multiple choice. How does one method scale across all skills, any skill, any language, any moments, hiring, upskilling, um, uh, project resourcing, et cetera. So, you know, deployment questions because it's new to them. Um,

[00:26:09] and then validity, like how, how do I differ differentiate a quiz that I just generated from chat GPT, um, with an actual assessment where you're telling me that you think this person has that skill and this person has not that skill. Um, you know, that validity question is very important and I'm never going to say we have full validity. You can trust a hundred percent of what we say, but I would tell you there are methods, there are features, there are technologies that we put in place to make sure we can defend our validity,

[00:26:38] essentially. And I think that's a very important question as well. Do you find that assessments motivate employees? So when using them internally like this, they actually, they go through the assessment, they're being assessed at a point in time. Does that motivate them to get better or see potential in themselves? Um, you have a set of employees that are power users that are inherently going to take assessment again and again, because they like to challenge themselves and they like to know where they stand.

[00:27:08] They're very comfortable with knowing that, uh, you have other employees where the key will be what's in it for me. You will have to tell them what are my incentives? Like, you know, if you get that score in that new Python area, you will get a brown bag launch with your VP. Maybe one example I have on the other end of the spectrum companies that say, we'll pay you cash once you reach that score. Uh, you know, so the incentives that you create, I call them the carrots, um, are going to be compelling to these employees.

[00:27:37] And the more carrots you have, the better. Uh, and then of course the assessment design, the experience, the UI, what they get out of it is going to also play a part in their, their excitement for, uh, being assessed. Um, the other thing I'd say is, uh, shifting the mindset of employees from reaching a defined target versus completing a course is extremely important in the years to come. The old way of, uh, delivering content. If you say last decade is a manager says, I like this course.

[00:28:07] Everybody should take this course. It takes 40 hours. And you're like, damn it. I don't think I need that course. It's not relevant for me. It's too easy. It's too hard. And then 10 people do it. And you collect stories from these 10 people. I think that the change here has to be, we will ask you to reach a score and there's a reason for it. It's tied to certain rewards and there's a business reason for it because it's tied to certain projects or OKRs. And then I let you take the assessment and maybe you'll reach it within the same day.

[00:28:36] Maybe you'll have a learning plan of three hours. Maybe you have a learning plan of 20 minutes and you'll get there and you're going to get your reward. I think that's a much more healthy, targeted and faster way to upskill. Yeah. Yeah. I love the carrots because you've got the punitive, you've got the sticks as well. But as long as people are, if you, as long as you give them a clear path to incentives and you've communicated that most people are going to say that, say, I can make more money. I can get to another level where I might be promotable to another position.

[00:29:07] I'm in. I think, I think, I think the thing is, is making it more like you did at the very beginning, making it more about skills and less about, you know, cause skills again, you tie that to learning and it's like, here's where you are. Here's, here's how you get to the next place. So the next time you take the assessment, make sure you do these things, whatever they may be. I absolutely love that. Got a question. I want to,

[00:29:36] I want to look at what, how to roll these out. So we've, what's the best way to say it? So we, we've seen, we've seen things fail, right? For adoptions, like adoption reasons, and it just doesn't get rolled out well. So, I mean, obviously there's a huge value for a company to, to bring assessments in and to run and do all this stuff. How do you bring this into an organization that just doesn't have this right now?

[00:30:06] How do they, how do they leverage this? I got, I want to answer for, for Keanu does. Oh, I think, I know. Well, William, I don't beat me out here. I don't know. He might not be. You probably will hate everything that I say. So let's just put it on real quick. No, no, here's the deal. Yeah. I think we need to, the word assessment has so much, as it relates to hiring, it has so much toxic baggage. That's a,

[00:30:34] that's associated with it. I think we need to kill that word. And I think we talk about skills and then we come up with another word that helps them understand what we're really doing. Evaluation testing. There's other synonyms that we can use, but the word assessment, in my humble opinion, is I think you just, if you say assessments, recruiters are getting tone deaf, hiring managers are tone deaf, executives are tone deaf, and candidates are tone deaf.

[00:31:04] So I think the sooner we shift away from saying the word assessment, like how can, how, how, how quickly can we kill the word assessment and make it about skills? And then whatever that next word is to explain what we're doing. Well, it often, because it often ties to the pre, employment, the hiring process. Right. Yeah. Which is, which is a screen out, but I understand understanding where people are with their skills, their skills, inventory, whatever.

[00:31:33] I think when we use the word assessment, you've already got a bunch of faces in corporate America. They're just going to be like, yeah, I've, I've been down this road. But yeah, that's my opinion. So take that, throw it all away. You have your own opinion. No, it's, it's, it's great insight. And to start there, so assessment is the thing that, you know, you're using is the product. It's not the value. So even what, what,

[00:32:03] what our product is, we call it the Sage, the AI mentor, because assessment is one of the things that a mentor has in its toolkit. And it turns out that, that if a mentor can't assess you, it can't really mentor you. Well, you know, so, so assessment is, is one of the critical piece measurements, the capability to measure in a trusted and precise way is one of the key capabilities of the best mentors in the world. You know, and so that's what we're talking about here. Now, when it comes to deployment,

[00:32:32] there are strategies that I recommend. In fact, the majority of customers are going to deploy with us in two phases. The first one is focused on AI. Everybody's undergoing an AI transformation. AI is a skill that most people do not have. It is easier to get verified and mentored on a skill that you're not supposed to have today, but we want you to have next year.

[00:32:59] Then if I assess you on something you're supposed to be doing 30 years ago, and you're an expert at, you know, so companies today run a set of badging programs with our AI mentor on, understanding AI, applying AI, and building AI. Of course, not everybody will get to the building AI. Everybody would at least go to the understanding AI level and will certify them in AI, in Gen AI, and in Responsible AI. And then as you get at the top of the pyramid,

[00:33:28] you'll start certifying people on how they build an agent, how they deal with language models, how they apply or use APIs. You know, these things are more for a narrow group. So that's the first step. We have a pyramidal framework. You can benchmark and upskill fast everyone on AI stuff, and people love it. The second question is any other skill. What about other skills? Our product managers need to be good at product on top of AI. Our marketers, they need to be good at demand generation on top of AI usage for content generation.

[00:33:58] So then those custom use cases is about how do we teach the enterprise to create mentorship assessment experiences that are fun and empowering, give insights to people, set the bar high for everyone. And that's the custom capabilities that we have that are sort of the second part of our deployments, typically. I got two questions. Brian, do you got something? Go ahead. I'll wait. Patiently. You'll wait patiently. Okay. So first of all,

[00:34:28] we're having this conversation a year from now. What are we talking about? We're talking about how much the workforce has progressed in AI readiness. We're talking about the average learning velocity of our country. We're talking about... Our country versus other countries? Yeah. I mean, regions of the world, like how fast are people learning? I think this is a critical competitive differentiator for companies, for any organization, they're learning velocity.

[00:34:58] So by next year, we'll have way more benchmarks than we have today. And I think we'll talk about how assessments have been reinvented for the better. I'm going with the word measurement. So I've made the decision during this last five minutes, it's skills measurement. Here you go. You called it. And then, because again, it is, it's measurement. I mean, there's no end to that measurement. The other question is,

[00:35:28] do you believe you could get the, when you sell to a client, do you think you could get the executive team and the board to do an assessment? Yeah. It starts at the top. I've said that many, many times and I still stand by it. So it turns out actually, the worst way of doing AI change management is when leaders pretend that they know AI. They pretend that they know AI and then everybody pretends. And then you end up, and that's not my words, it's one of our customers,

[00:35:58] you end up with a bunch of dangerous amateurs. Yeah. And, and so to me, it starts at the top. It starts by leadership, you know, showing what, what score they're at today, how much progress they're making and making people feel safe about it. Because it's not about, you know, pointing fingers. It's about knowing your baseline, knowing your gaps, and then running toward the targets. Pretty much. I love it. Yeah. What's the, what's the risk of over-assessing?

[00:36:27] Do we kill creativity? Do we, do we, do we stop the employee from actually being creative and thinking? If we over-assess and put too much in their head? You, you can actually assess creativity and, and you can assess this and quantify it. But, but I would say the, the, the question is more like, how do you define people's jobs?

[00:36:51] Like people's jobs are made of tasks and certain job actually require very rigid set of tasks where you need to be master at it. And you just need to execute other jobs. They need a significant portion of their time and tasks that are more fuzzy. They're less interpretable. You can still measure the underlying construct that you're looking for. Like, you know, I have people in our team that are way more creative than I am. So, so be it.

[00:37:19] Like they're in roles where their creativity can like shine, but, but that can also be measured. So I would not equate the role and the tasks with essentially the construct that you need to be measuring in the first place. Right. I think you were getting to fatigue. Fatigue. I understood it. Like measurement fatigue. Yeah. The risk of over-assisting fatigue. It's a wonderful,

[00:37:39] it's a wonderful question of how frequently can we do measurement without making the employee like hate measurement. No, absolutely. So we don't want them to hate that. So there's got to be a, every six weeks, every month, every three weeks, or when they get to a certain point, I don't know what the answer is, but Ryan's right. We've seen it in our lives with the fatigue of, of a lot of what they like help employee satisfaction surveys. You know,

[00:38:08] we went from once a year to then maybe twice a year. And then all of a sudden we skipped all the way to daily. Yeah. It's like every time you were in a bathroom, you hit the smiley button. Yeah. You got it. And so what we've seen in that space, which is different from yours, but it's fatigue. Employees get fatigue. The executives get fatigue. Everyone gets fatigue. And then all of a sudden it's just, it means less or is meaningless, whatever that may be. I think that's where he was trying to drive you.

[00:38:37] Just to understand how frequent, before we start really pissing off the employees. Yeah, I, I, I agree. I mean, you, imagine you're being watched at all time and everything. It doesn't make any sense. Like there's so many things that are not even measurable. And there's also things where, you know, you don't want to be measured at all time. You want to be able to show off your progress. That's what you want to be able to.

[00:39:05] And that might mean every month you have an assessment that benchmarks your progress every year, every six months. I think that depends on the pace of the organization, the need of the business, you know, and then you have the self-directed aspect, which is, can be on a daily basis. Like some, some users, they need it on a daily basis, but you know that your manager doesn't see your score. You know, there's permission levels on the platform. So, you know, that, you know, my manager can't see my score.

[00:39:34] My manager can see the team score or the organizational score, and I'm doing whatever I want with my mentor. And my mentor is benchmarking my progress and I get that feedback and I can turn it off. And, you know, so I think that's where the future will be. It's like, there will be some formalities and there will be a lot of self-directed that is safe. And if I be hitting new levels, Ryan, I want everybody in the, in the companies to know that I'm hitting new levels, especially if I'm already ahead of them. Oh, you want to, you want to. Oh, well, oh yeah. I want to,

[00:40:03] I want to put that somewhere where everybody's like, he likes the humble brain. Yeah. Yeah. I just got to, I just got to level two 40. I don't know where y'all are. Yeah. This has been absolutely wonderful. Thank you so much for coming on the show. It's just been a great show and we appreciate you. No, it was fun. Thanks, William. Thanks, Ryan. Absolutely. And thanks for the audience until next time.