Minding the Machines. Jeremy Adamson
Читать онлайн книгу.
meaningless.
In the 1920s the practice of scientific management led to improvements in the productivity of teams by breaking the process into elements that could be optimized. Through a thorough motion study of workers at Bethlehem Steel, Frederick Taylor created and instituted a process that optimized rest patterns for workers and as a result doubled their productive output (Taylor, 1911). He advocated for all workplace processes to be evaluated in terms of their efficiency and all choice to be removed from the worker. This brutal division of labor and resulting hyperspecialization led to reduced engagement and produced suboptimal outcomes at scale when all factors were considered.
Practitioners need to avoid those actions and policies that create a form of neo-Taylorism within their organizations. Models that fully automate a process and embed simulated human decision making remove the dynamism and innovation that comes from having humans in the loop. It cements a process in place and reduces engagement and stakeholder buy-in. Analytics should support and supplement human endeavor, not supplant it with cold efficiency. It is essential that analytical projects are done within the context of the business and with the goal of maximizing the value to the organization.
Model accuracy needs to be secondary to bigger-picture considerations, including these:
Technical Implementation Is the architecture stable? Does it require intervention?
Political Implementation Does it conflict with other projects? Will implementation create redundancies?
Procedural Implementation Will this fit in with existing processes? Will it require significant changes to current workflows? What are the risks associated with implementation? Will it introduce the potential for human error? Does it have dependencies on processes that are being sunset?
Interoperability Are there downstream processes depending on the results? What are the impacts of a disruption to these processes? Can it be shifted to another system? Does it create dependencies?
Extensibility Can the output be upgraded in the future? Does it require specialized skillsets? Is it generalized enough to be used for other purposes?
Scalability Would this approach work if the volume of data doubled? Tripled?
Stability Has it gone through thorough QA? Has it been tested at the boundary conditions? What happens if data are missing? What happens if it encounters unexpected inputs? How does it handle exceptions?
Interpretability Are the results clearly understandable? Does the process need to be transparent?
Ethics Is it legal? Does it have inherent bias?
Compliance Does it contain personally identifiable information? Does it comply with the laws of the countries in which it will be used? Does it use trusted cloud providers?
Without exception, effort is better spent in discussing and addressing the above considerations than in marginal improvements to model performance. Even a poorly designed model will work with strong phenomena, and a poorly performing model that is in use will outperform a sophisticated model that is sitting idle.
Rinse, Report, Repeat
One of the most memorable scenes from the 1990s classic Office Space involves the protagonist being chastised repeatedly by his coworkers and managers about an incorrectly prepared TPS report. The reason this was so memorable is that it reflects the experience of every office worker at some point in their careers. Bureaucracy is a self-propagating danger, and for many larger legacy organizations, entire teams are dedicated to the preparation and dissemination of unread reports. There are several compounding factors that have led to an explosion in the number of reports recently.
With legacy reporting tools, the effort required to create reports created a natural limit on the number that could be produced. This provided a control on the spread of new reports. Unfortunately, the strength and ease of use of reporting and visualization tools that are available today make creating new reports trivial.
In addition, as tenure at the management level falls over time, new managers arrive and request views of the data that are most conducive to their way of thinking. This, combined with a bias to action, can lead to a flurry of new reports, which are then promptly abandoned at the next reorganization but continue to be maintained.
Finally, a common rationalization for poor decisions by leaders is that they are due to a lack of information. In response to a personal error, and as a cover to their rascalities, it is not unusual for a new report to be requested, ostensibly to prevent the incident from recurring, but almost certainly as presentable evidence of their proactivity.
Despite all of these drivers of new reports, the organization could conceivably reach an isotonic point of stability where new reports are balanced by discontinued reports. Unfortunately, the process to sunset a report in most organizations is more onerous than the process to create a new one, leaving zombie reports that are diligently maintained by an army of analysts but unused.
For decision makers who have a more tenuous understanding of the value offering of analytics, and a desire to have better-informed decision making, the one-off choice to request that the team create a report can quickly become a pattern of distracting asks. At the expense of large transformative projects, more reports are created and maintained. Over time, the value of the analytics team is questioned.
How can this be prevented? Clearly articulating the value proposition of the team and ensuring from the start that it has executive support is essential to standing up a successful team. Respectfully pushing back against reporting and other non-value-add requests, though politically sensitive, must be done. Every member of the team must be empowered and given the autonomy to question the work that is being requested.
Too Fast, Too Slow
Understanding the first-mover advantage associated with data and analytics, many organizations of means yet with limited analytical maturity have hired dozens of practitioners to rapidly scale up and advance their data capabilities. After 18 to 24 months, these large technocratic organizations review the cost/benefit of the new $10M division and question the value of analytics to the enterprise. Moving too quickly, without the cultural or technical infrastructure to support it, can put an end to analytical ambitions before the first hire has been made.
Alternatively, many organizations have moved too cautiously and hired one or two early-career employees and placed them under a line manager with conflicting priorities and limited analytical understanding. The maturity of the data science team in this case often peaks with gentle scripting and automation, and executives are again left questioning the value of analytics.
It can seem that a Goldilocks approach of creating a mid-sized team could work, but the issue in both extremes has not been the number of people involved, it has been with a lack of a defined strategy. To prevent this situation, organizations need to have a clear objective in mind for the team, an understanding of what the field of advanced analytics and AI is, and the support that is required to make it a success. Further, organizations cannot look within to build out this strategy, as they thoroughly lack the capabilities to define this strategy. It is only by consultation with others and thorough benchmarking to industry and practice standards that this strategy can be developed.
More Data, More Problems
All organizations have some capacity for gathering data, and once the quantity of that data reaches a certain size, a data-centric team (whether business intelligence, data services, self-service insights, or general information technology) is naturally formed. The purview of these data teams in many cases has been confused with analytics and data science, to the point that the intersection of the two is very often considered the scope for both. Though there is certainly a strong mutually supportive relationship between these two functions, they exist in different spaces and with different incentives—one being a cost center and one being a profit center. Executives with medium-term ambitions to develop a next-generation analytical function will