Co-Organizing a Workshop at AAAI HCOMP on Designing AI Tools for the Future of Work

We recently had the honor of co-organizing a workshop at the AAAI Human Computation and Crowdsourcing Conference (HCOMP) in Pittsburgh. This workshop focused on designing AI tools for the future of work and brought together diverse perspectives and innovative ideas.

Keynote Speaker: Sara Kingsley

Caption: Sara Kingsley giving her keynote at our workshop where she explained about her research on designing human centered AI for the future of work. She is especially focused on using red-teaming techniques to conduct online audits around AI in the work place, identify biases around current AI tools, and then designing AI driven interventions to address the challenges.

One of the highlights of our workshop was having Sara Kingsley as our keynote speaker. Sara is a researcher at Carnegie Mellon University and has extensive experience working at Meta and within the US Federal government, specifically in the Secretary of Labor.

In her engaging talk, Sara shared her work using red teaming—a process where experts challenge and test systems to find vulnerabilities or weaknesses—to identify problematic job ads and content related to job advertising. She explained how she applies red teaming to ensure that job ads do not propagate harmful biases or misleading information. This approach allows her to design human-centered AI tools that can create better, more equitable AI-driven futures for workers.

This type of research is critical as it helps to identify and mitigate potential biases and harms in AI systems before they impact real users. We were especially proud to note that Sara recently won the best paper award at HCOMP’24 on this very topic. Congratulations to Sara on this well-deserved recognition! We are proud to have had her as a keynote speaker in our workshop.

Co-Design Activity with Community Partners

Caption: Our research collaborator Jesse Nava in his workforce development programs for former/current prisoners.

Another unique aspect of our workshop was the co-design activity we held with workshop participants and current and former prisoners from California’s Department of Corrections and Rehabilitation. Our community partner, Jesse Nava, joined us via call, making this session truly impactful.

During this co-design activity, participants proposed ideas for generative AI tools that could support the reintegration of former prisoners into the workforce. Jesse provided invaluable feedback on these proposals, sharing his perspective on potential harms, biases, and areas where these tools could be improved to better serve the formally incarcerated population.

This was a unique experience as it allowed us to receive direct feedback from real-world stakeholders who would be directly affected by these AI tools. The opportunity to co-design with such engaged partners highlighted the importance of including diverse voices and lived experiences in the development process.

Closing Thoughts

We concluded the workshop with a sense of excitement and renewed commitment to continue designing the future of generative AI tools together. This collaborative approach is key to creating technologies that are inclusive, fair, and genuinely supportive of the communities they aim to serve.

Thank you to everyone who participated and contributed to making this workshop a success. We look forward to future opportunities to innovate, collaborate, and create impactful AI solutions.

Launching a Kite into CHI 2024: Insights from the Premiere Scientific Conference on HCI

My research lab and I had the privilege of attending and presenting our research at the premiere scientific conference in human computer interaction (CHI'24), which was held in Hawaii this year.

This premier Human-Computer Interaction (HCI) conference showcased a plethora of innovative research focused on enhancing the interaction between humans and technology. In this blog post, I am thrilled to share a deeper dive into several studies that align and inspire our research on designing empowering tools for gig workers.

Our Research: Designing Worker-Centric Sousveillance Tools

Before we dive into the interesting new research we heard about at CHI’24, I would like to share a bit about what my lab was proud to present at the conference! We presented our new research on designing sousveillance tools for gig workers, which via interviews and co-design sessions identified how gig workers imagined and desired tools that would allow them to collect their own data about their workplace, as well as any concerns gig workers could have about such technology. You might be wondering, why do gig workers need such type of tools? A problem that exists is that within gig platforms there is an information asymmetry problem where workers have less access to information about their workplace than other stakeholders within the platform. For example, workers on gig platforms normally cannot see if they are earning less than others workers or if low wages is the norm on the platform. Similarly, workers usually cannot easily share information about their clients to alert each other of when a client is a fraudster. Gig platforms have been designed in a way where workers are usually in the dark about what is happening in their workplace. This was why it was important for us to think about how tools that would allow workers to have access to their own workplace data should look like and in a way that was worker centric. Note that we used the term: “sousveillance” to refer to this technology as sousveillance is about the people without power (in this case workers) being able to conduct surveillance over those who have power (e.g., their algorithmic bosses.) This term contrasts with surveillance which is about people in power monitoring those who do not have power (e.g., bosses monitoring workers). My students: undergraduate Maya De Los Santos and PhD student Kimberly Do were who presented our research. I am very proud of them and the research they conducted with my students, Dr. Michael Muller and myself.

Link to our paper

Relevant Research Highlights

CHI'24 offered a range of presentations of scientific papers that enriched our understanding of HCI's role in labor dynamics (an important aspect of our research), each bringing unique insights that intersect with our research goals. Some of these papers include:

• Self-Tracking in the Gig Economy: From The Pennsylvania State University, researchers delved into how gig workers engage in self-tracking to manage their responsibilities across different identities. This study provides a nuanced view of the self-surveillance gig workers perform to balance personal and platform demands, complementing our research on external surveillance.
  Paper link
• AI and Worker Wellbeing: A study by Northeastern University and the University of Chicago examined the resistance and acceptance of AI systems that infer workers' wellbeing from digital traces. This research is crucial as we consider ethical implications in our sousveillance tools, ensuring they support rather than undermine worker autonomy.
  Paper link
• Interaction Challenges with AI in Programming: Insights from Wellesley College and Northeastern University into how beginning programmers interact with AI in coding presented an interesting parallel to our work. Understanding these interaction barriers helps inform our design of more intuitive interfaces for gig workers interacting with AI tools.
  Paper link
• Designing with Incarcerated Workers: The University of California, Irvine shared compelling work on using participatory design with marginalized groups, like recently incarcerated youth, to create mixed reality tools. Their approach underscores the value of involving underrepresented communities in design processes, a principle central to our research ethos. We have also recently been able to start working with incarnated individuals in California, such as Jesse Nava. This research from UC Irvine helped us to start to identify how we could potentially conduct co-design sessions with prisoners. We are looking forward to continuing this research.
  Paper Link
• Temporal Flexibility and Crowd Work: Research from University College London highlighted the constraints on crowdworkers' temporal flexibility, underscoring similar challenges faced by gig workers in managing work schedules under rigid platform algorithms. This research was especially relevant for other research we are conducting on understanding how workers’ manage their time and how we can best design tools that support their different temporal preferences, as well as understand when the platform might be forcing onto workers certain time constraints that are unnecessary and that hurt workers.
  Paper Link
• Data Labeling and AI Interventions in Crowdsourcing: A study from the University of Washington introduced 'LabelAId', a tool that uses AI to improve the quality and knowledge of crowdworkers performing data labeling. This aligns with our interest in tools that enhance worker capabilities and autonomy.
  Paper link
• Cognitive Behavioral Therapy-Inspired Digital Interventions: The University of British Columbia's exploration of therapy-inspired digital tools for knowledge workers tackled the balance between productivity and well-being, a balance we aim to address in gig work environments.
  Paper link

Expanding Our Horizons

These presentations and papers not only expanded our understanding of the challenges faced by workers in the gig economy but also illustrated the breadth of opportunities for HCI research to intervene positively. Each study provided valuable insights into different aspects of how technology interfaces with labor dynamics, from enhancing worker autonomy to addressing systemic issues through design.

Looking Forward

Inspired by the innovative ideas and critical discussions at CHI'24, we are excited to continue refining our projects. The conference has invigorated our commitment to developing HCI solutions that genuinely empower workers and contribute positively to the broader discourse on labor and technology.

Stay tuned for more updates as we apply these enriched perspectives to our ongoing and future research projects, continuing to advocate for and develop technologies that uphold the dignity and rights of workers.

Recap: AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2022)

This week we attended the AAAI conference on Human Computation and Crowdsourcing (HCOMP'22). We were excited about attending for several reasons: (1) we were organizing HCOMP's CrowdCamp, excited about having the power to drive the direction of this event within the conference!, (2) it was the 10-year anniversary of the conference and we were elated to reflect collectively on where we have come as a field over the years, (3) we chaired one of the keynotes of HCOMP, in particular, our PhD hero, Dr. Seth Cooper, and (4) we had an important announcement to share with the community!
WE WILL BE GENERAL CO-CHAIRS OF HCOMP’23!

Organizing CrowdCamp.

This year, Dr. Anhong Guo from the University of Michigan and me had the honor of organizing HCOMP's CrowdCamp, a very unique part of the HCOMP conference. It is a type of mini hackathon where you get together with crowdsourcing experts and define the novel research papers and prototypes that push forward the state of the art around crowdsourcing. Previous CrowdCamps led to key papers in the field, such as the Future of Crowd Work paper and my own CHI paper on Subcontracting Micro Work.

This year, when we put out the call for CrowdCamp, we witnessed an interesting dynamic. A large number of participants were students, novices to crowdsourcing, but they had great interest in learning and then impacting the field. This dynamic reminded me of what I had encountered when I organized my first hackathon, FixIT: the participants had great visions and energy for changing the world! But they also had limited skills to execute their ideas. They lacked data to determine if their ideas were actually something worth pursuing. To address these challenges, in the past, I gave hackathon participants bootcamps to ramp up their technical skills (this allowed them to execute some of their visions). We also taught these participants about human centered design to empower them to create artifacts and solutions that match people's needs, and not a hammer in need of nails.

For CrowdCamp, we decided to do a similar thing:
We had a mini-bootcamp, organized by Toloka (a crowdsourcing platform), that explained how to design and create crowd-powered systems. The bootcamp started with a short introduction on what is crowdsourcing, common types of crowdsourcing projects (like image/text/audio/video classification) and interesting ones (like side-by-side comparison, data collection and spatial crowdsourcing). After that, the bootcamp introduced the Toloka platform and some of its unique features. Then the bootcamp briefly presented Toloka Python SDK (Toloka-Kit and Crowd-Kit) and moved to an example project on creating a crowd powered system, especially a face detection one. The code used in the Bootcamp is in the following Google Collab: https://colab.research.google.com/drive/13xef9gG8T_HXd41scOo9en0wEZ8Kp1Sz?usp=sharing.

We taught human centered design, and had a panel with real world crowdworkers who shared their experiences and needs. The participants were empowered to design better crowdworkers and create more relevant technologies for them, as well as technologies that would better coordinate crowdworkers to produce higher quality work. The crowdworkers who participated in CrowdCamp all came from Africa, and they shared how crowd work had provided them with new job opportunities that were typically not available in their country. Crowd work helped to complement their expenses (a side job). They were motivated to participate in crowd work for the additional money received, also knowing that they were contributing to something bigger than themselves (e.g., labeling images that will ultimately help to power self-driving cars.) Some of the challenges these crowdworkers experienced included unpaid training sessions. It was unclear sometimes whether the training sessions were worth it or not. They also discussed the importance of building worker communities.

CrowdCamp ended up being a success with over 70 people who had registered and then created a number of different useful tools for crowdworkers. The event was hybrid with people on the east coast joining us at Northeastern university. We had delicious pizza and given that we were in Boston, delicious Dunkin Donuts :)

Chairing Professor Seth Cooper's Keynote.

We had the honor of chairing the keynote of Professor Seth Cooper, an Associate Professor at the Khoury College of Computer Sciences. He previously worked for Pixar Animation Studios and Electronic Arts, a big game maker. Seth is also the recipient of an NSF career grant. Professor Cooper’s research has focused on using video games and crowdsourcing techniques to solve difficult scientific problems. He is the co-creator and lead designer, as well as developer of Foldit, a scientific discovery game that allows regular citizens to advance the field of biochemistry. Overall, his research combines scientific discovery games (particularly in computational structural biochemistry), serious games, and crowdsourcing games. A pioneer in the field of scientific discovery games, Dr. Cooper has shown video game players are able to outperform purely computational methods for certain types of structural biochemistry problems, effectively codifying their strategies, and integrating them in the lab to help design real synthetic molecules. He has also developed techniques to adapt the difficulty of tasks to individual game players and generate game levels.

Seth’s talk discussed how he is using crowdsourcing to improve video games, and video games to improve crowdsourcing. What does this mean? In his research, Professor Cooper integrates crowd workers to help designers improve their video games. For example, he integrates crowds to help them test just how hard or easy the game they are creating is. It enables designers to identify how easy it is for gamers to advance within the different stages of a game. The integration of crowdworkers allows gamers to easily iterate and improve their video game. Dr. Cooper is also integrating gaming to improve crowdsourcing. In particular, he has studied how he can integrate games to improve the quality of work produced by crowd workers.

During the Questions of Professor Cooper, some interesting questions emerged:
What types of biases do crowdworkers bring to the table when co-designing video games? It was unclear whether crowdworkers are actually similar to how typical gamers would play a video game. Hence, the audience wondered just how much designers actually use the results of the way crowdworkers engage with a video game. Professor Seth mentioned that in his research, he found that crowdworkers are similar to typical gamers in playing games. A difference is that typical gamers (voluntarily playing a game instead of getting paid to play) will usually focus more on the aspects of the game they like the most. Crowdworkers will focus on exploring the whole game instead of focusing on particular parts (because of the role that the payments play). Perhaps, these crowdworkers feel that by exploring the whole game, they are better showcasing to the requester (designer) that they are indeed playing the game and not slacking off. Some people have a gaming style that focuses on the "catch-them all'' approach (an exploratory mode). However, the "catch-them-all" term is used in reference to Pokemon, where people are interested in being able to explore the entire game and collect all the different elements (e.g., Pokemons).

How might we integrate game design to help crowdworkers learn? Dr. Flores-Saviaga posed an interesting question about the role games could play in facilitating the career development of these workers. Professor Seth expressed an interest in this area while also mentioning that you can imagine that workers instead of earning badges within the game could earn real certificates that translate into new job opportunities.

What gave him confidence that the gaming approach in crowdsourcing is worth pursuing? When Foldit came out, it was unclear that gaming would actually be useful for mobilizing citizen crowds to complete complex scientific tasks. The audience wanted to know what led him to explore this path. Professor Cooper explained that part of it was taking a risk down a path he was passionate about: gaming. I think for PhD students and other new researchers starting out, it can be important to trust your intuition and conduct research that personally interests you. In research, you will take risks, which makes it all the more exciting :)

Dr. Jenn Wortman’s Keynote.

We greatly enjoyed the amazing keynote given by Dr. Jenn Wortman Vaughan (@jennwvaughan) at HCOMP 2022. She presented her research in Responsible AI, specially interpretability and fairness in AI systems.

A takeaway is that there are challenges in the design of interpretability tools for data scientists such as IntrepretML or SHAP Python package, where they found that these tools lead to over-trusting and misusing how ML models work. For more info, look at her CHI 2020 paper: "Interpreting Interpretability”

Dr. Jeffrey Bigham’s Keynote.

An incredible keynote given by Dr. Jeffrey Bigham at HCOMP 2022. He presented work developed in Image Description for 17 years! He showed different connections (loops) in finding the right problem and right solution in Image Description, such as Computer Vision, Real-Time Recruitment, Gig Workers, Conversations with the Crowd, Datasets, etc.

A takeaway is that there could be different interactions or loops in the process of Applying Machine Learning and HCI as seen in the image below from the Problem selection until the Deployment of the system.

Doctoral Consortium.

The HCOMP doctoral consortium was led by Dr. Chien-Ju Ho and Dr. Alex Williams. The consortium is an opportunity for PhD students to share their research with crowdsourcing and human computation experts. Students have the opportunity to meet other PhD students, industry experts, and researchers to expand their network and receive mentoring from both industry and academia. Our lab participated in the proposal “Organizing Crowds to Detect Manipulative Content.” A lab member, Claudia Flores-Saviaga, presented the research she has done in this space for her PhD thesis.

Exciting news for the HCOMP community!

The big news I want to share is that I have the honor of being the co-organizer of next year's HCOMP! I will co-organize it with Alessandro Bozzon and Michael Bernstein. We are going to host the conference in Europe, and it will be united with the Collective Intelligence conference. Our theme is about reuniting and helping HCOMP grow in size by connecting with other fields, such as human centered design, citizen science, data visualizations, and serious games. I am excited to have the honor and opportunity to build the HCOMP conference.