Survey Data Comparison¶
In this file, the data from Abram Miller's Improving Research Software Engineering in Mathematics survey for his honors thesis is compared directly to the data from the survey conducted by Carver et al. for the paper A survey of the state of the practice for research software in the United States.
Abram Miller's survey is abbreviated as 'IRSEM'.
The survey by Carver et al. is abbreviated as 'URSSI', as it was conducted under the US Research Software Sustainability Institute.
The analysis of three questions is omitted as future work:
- RQ1: Software Development Practices Question 2
- What percent of your time did you actually spend in the following activities? (total should be 100%)
- RQ1: Software Development Practices Question 3
- What percent of time would you have liked to have spent in each activity? (total should be 100%)
- RQ4: Funding and Institutional Support Question 2
- Does your institution provide support for your research software development activities in the following ways: (total should be 100%) (question shown if Developer or Combination was selected)
To re-export this notebook as a PDF or HTML file, execute either
jupyter nbconvert --execute --to html notebooks/surveydatacomparison.ipynb
or
jupyter nbconvert --execute --to pdf notebooks/surveydatacomparison.ipynb
in the terminal. (Note: PDF conversion requires Pandoc to be installed)
Load data and functions:¶
In [1]:
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import seaborn as sns
IRSEM = pd.read_csv('../data/IRSEM_Results.csv')
URSSI = pd.read_csv('../data/URSSI_survey_anonymized20210930.csv')
# Use Seaborn colorblind palette
plt.style.use('seaborn-v0_8-colorblind')
### print functions ###
# text function: print the answer counts, percentage, and totals for a given question
def print_question(questionID="Q0.1", survey = 0, percentages = True, totals = True, multiselect=False):
if (survey == 0 or survey == 1):
question_data = IRSEM[questionID][2:]
if (multiselect):
question_data = question_data.str.split(',').explode()
if (percentages):
df = pd.concat([question_data.value_counts(),
question_data.value_counts(normalize=True).round(2)],
axis = 1, keys=["IRSEM survey: counts", "percentages"])
else:
df = pd.concat([question_data.value_counts()], axis = 1, keys=["IRSEM survey: counts"])
if (totals): df.loc['TOTAL:'] = df.sum(numeric_only=True, axis=0)
print (df)
if (survey == 0): print("\n - - - - -\n")
if (survey == 0 or survey == 2):
question_data = URSSI[questionID][2:]
if (multiselect):
question_data = question_data.str.split(',').explode()
if (percentages):
df = pd.concat([question_data.value_counts(),
question_data.value_counts(normalize=True).round(2)],
axis = 1, keys=["URSSI survey: counts", "percentages"])
else:
df = pd.concat([question_data.value_counts()], axis = 1, keys=["URSSI survey: counts"])
if (totals): df.loc['TOTAL:'] = df.sum(numeric_only=True, axis=0)
print (df)
# text function: print a set of questions (aka a Likert Matrix)
def print_question_set(question = "", statements = [], survey = 0, percentages = True, totals = True, multiselect = False):
div = "___________________"
for i, statement in enumerate(statements):
print(statement + ":\n")
print_question(question + "_" + str(i + 1), survey=survey, percentages=percentages, totals=totals, multiselect = multiselect)
if (i < len(statements) - 1):
print("\n", div*3, "\n")
### graph functions ###
# function: find counts of each answer type on each question
def get_ans_counts(questions, answerTypes):
ANSWERCOUNTS = {category: [] for category in answerTypes}
for label, data in questions:
for category in answerTypes:
count = data.value_counts().get(category, 0)
ANSWERCOUNTS[category].append(count)
#print(ANSWERCOUNTS)
return ANSWERCOUNTS
# function: get the percentages from the total counts of each category's responses
def get_percentages(questions, answerTypes):
PERCENTAGES = {category: [] for category in answerTypes}
for label, data in questions:
for category in answerTypes:
percent = data.value_counts(normalize=True).get(category, 0) * 100
PERCENTAGES[category].append(percent)
#print(totals)
return PERCENTAGES
# functions to graph a likert matrix
def get_likert_data(survey=IRSEM, questionID = "Q0.8", statements = ["1 Person", "2 - 5 people", "6 - 20 people", "20+ people"]):
DATA = []
for i, statement in enumerate(statements):
DATA.append((statement, survey[questionID + "_" + str(i + 1)][2:]))
#print(statement)
return DATA
# returns this data format:
# [('1 Person', survey["Q0.8_1"][2:]),
# ('2 - 5 people', survey["Q0.8_2"][2:]),
# ('6 - 20 people', survey["Q0.8_3"][2:]),
# ('20+ people', survey["Q0.8_4"][2:])]
def graph_likert_matrix(questionID = "Q0.8",
title = "Team Size (Question 0.8)",
IRSEMsubtitle = "IRSEM: Mathematicians",
URSSIsubtitle = "URSSI: Research Software Engineers",
statements = ["1 Person", "2 - 5 people", "6 - 20 people", "20+ people"],
answers = ['Never', 'Sometimes', 'About half the time', 'Most of the time', 'Always'],
percentage = True,
stacked = True,
labelrotation = 45):
LIKERT_DATA_IRSEM = get_likert_data(survey=IRSEM, questionID=questionID, statements=statements)
LIKERT_DATA_URSSI = get_likert_data(survey=URSSI, questionID=questionID, statements=statements)
# create dataframes
if (percentage):
df1 = pd.DataFrame(get_percentages(LIKERT_DATA_IRSEM, answers), index=[label for label in statements])
df2 = pd.DataFrame(get_percentages(LIKERT_DATA_URSSI, answers), index=[label for label in statements])
else:
df1 = pd.DataFrame(get_ans_counts(LIKERT_DATA_IRSEM, answers), index=[label for label in statements])
df2 = pd.DataFrame(get_ans_counts(LIKERT_DATA_URSSI, answers), index=[label for label in statements])
# plot
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle(title) # Title
ax1 = df1.plot.bar(rot=0, ax=ax1, stacked = stacked)
ax1.set_title(IRSEMsubtitle)
ax1.set_ylabel('Percentage %') if percentage else ax1.set_ylabel('Frequency')
ax1.tick_params('x',labelrotation=labelrotation)
if (percentage): ax1.set_yticks(np.arange(0, 100.1, 5), minor=True)
ax1.get_legend().remove()
ax2 = df2.plot.bar(rot=0, ax=ax2, stacked = stacked)
ax2.set_title(URSSIsubtitle)
ax2.tick_params('x',labelrotation=labelrotation)
if (percentage): ax2.set_yticks(np.arange(0, 100.1, 5), minor=True)
# edit parameters
plt.legend(bbox_to_anchor=(1.02, 1))
plt.gcf().set_size_inches(12, 6)
plt.show()
# functions to graph a question
# graph question in a standard format
def graph_question(questionID = "Q0.1",
title = "Respondent type (Demographics: question 1)",
IRSEMsubtitle = "IRSEM: Mathematicians",
URSSIsubtitle = "URSSI: Research Software Engineers",
statement = "",
answersIRSEM=[],
answersURSSI=[],
percentage = True,
stacked = False,
labelrotation = 0):
# get question data for each survey
QUESTION_DATA_IRSEM = [(statement, IRSEM[questionID][2:])]
QUESTION_DATA_URSSI = [(statement, URSSI[questionID][2:])]
# get lists of possible answers for each version of the question
# the pandas conversion is necessary to remove the 'nan' values in the list
if not answersIRSEM: ANSWER_DATA_IRSEM = pd.Series(IRSEM[questionID][2:].sort_values().unique()).dropna().tolist()
else: ANSWER_DATA_IRSEM = answersIRSEM
if not answersURSSI: ANSWER_DATA_URSSI = pd.Series(URSSI[questionID][2:].sort_values().unique()).dropna().tolist()
else: ANSWER_DATA_URSSI = answersURSSI
# create dataframes
if (percentage):
df1 = pd.DataFrame(get_percentages(QUESTION_DATA_IRSEM, ANSWER_DATA_IRSEM), index=[statement])
df2 = pd.DataFrame(get_percentages(QUESTION_DATA_URSSI, ANSWER_DATA_URSSI), index=[statement])
else:
df1 = pd.DataFrame(get_ans_counts(QUESTION_DATA_IRSEM, ANSWER_DATA_IRSEM), index=[statement])
df2 = pd.DataFrame(get_ans_counts(QUESTION_DATA_URSSI, ANSWER_DATA_URSSI), index=[statement])
# plot
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle(title)
ax1 = df1.plot.bar(rot=0, ax=ax1, stacked = stacked)
ax1.set_title(IRSEMsubtitle)
ax1.set_ylabel('Percentage %') if percentage else ax1.set_ylabel('Frequency')
ax1.tick_params('x',labelrotation=labelrotation)
if (percentage): ax1.set_yticks(np.arange(0, 100.1, 5), minor=True)
ax1.get_legend().remove()
ax2 = df2.plot.bar(rot=0, ax=ax2, stacked = stacked)
ax2.set_title(URSSIsubtitle)
ax2.set_ylabel('')
ax2.tick_params('x',labelrotation=labelrotation)
if (percentage): ax2.set_yticks(np.arange(0, 100.1, 5), minor=True)
# edit parameters
plt.legend(bbox_to_anchor=(1.02, 1))
plt.gcf().set_size_inches(12, 6)
plt.show()
Respondent Demographics¶
Q0.1: Which profile matches the work you perform?¶
Researchers (pure user) – I primarily use software to do my research. I have contributed to local software development, but only in a limited capacity. I have the ability to recognize and report bugs. I don’t build new software myself, but I occasionally modify existing software for my own needs.
Developers (pure software developer) – I primarily write software for others to use. I contribute to collaborative software projects. I maintain a package or distribution
Combination – I have attributes of both “Researchers” and “Developers.” I both use software for my own research and write software that other researchers use for their research
In [2]:
questionID = "Q0.1"
title = "Respondent type (Demographics: Question 1)"
# print raw data
print_question(questionID=questionID)
# graph question
graph_question(questionID=questionID, title=title,
percentage=True, stacked=False)
IRSEM survey: counts percentages
Q0.1
Combination 27.0 0.59
Researchers 15.0 0.33
Developers 4.0 0.09
TOTAL: 46.0 1.01
- - - - -
URSSI survey: counts percentages
Q0.1
Combination 544.0 0.49
Researchers 473.0 0.43
Developers 89.0 0.08
TOTAL: 1106.0 1.00
Q0.2: What type of organization do you work for?¶
Educational institution
National lab
Industry
Other
In [3]:
questionID = "Q0.2"
title = "Who respondents work for (Demographics: Question 2)"
statement = "Organization type"
# print raw data
print_question(questionID=questionID)
# graph question
graph_question(questionID=questionID, title=title, statement=statement,
percentage=True, stacked=False)
IRSEM survey: counts percentages
Q0.2
Educational institution 35.0 0.76
Other: 5.0 0.11
Industry 5.0 0.11
National lab 1.0 0.02
TOTAL: 46.0 1.00
- - - - -
URSSI survey: counts percentages
Q0.2
Educational institution 901.0 0.86
National lab 62.0 0.06
Other 49.0 0.05
Industry 39.0 0.04
TOTAL: 1051.0 1.01
Q0.3: In which country do you currently reside?¶
In [4]:
print_question("Q0.3")
IRSEM survey: counts \
Q0.3
United States of America 28.0
Germany 4.0
France 2.0
Austria 2.0
Canada 2.0
Italy 2.0
Switzerland 1.0
Colombia 1.0
South Korea 1.0
United Kingdom of Great Britain and Northern Ir... 1.0
Australia 1.0
Israel 1.0
TOTAL: 46.0
percentages
Q0.3
United States of America 0.61
Germany 0.09
France 0.04
Austria 0.04
Canada 0.04
Italy 0.04
Switzerland 0.02
Colombia 0.02
South Korea 0.02
United Kingdom of Great Britain and Northern Ir... 0.02
Australia 0.02
Israel 0.02
TOTAL: 0.98
- - - - -
URSSI survey: counts \
Q0.3
United States of America 993.0
Germany 11.0
Canada 8.0
United Kingdom of Great Britain and Northern Ir... 6.0
France 5.0
Japan 3.0
Switzerland 3.0
Netherlands 2.0
India 2.0
Iran, Islamic Republic of... 1.0
Turkey 1.0
China 1.0
Norway 1.0
Australia 1.0
Romania 1.0
Spain 1.0
South Korea 1.0
TOTAL: 1041.0
percentages
Q0.3
United States of America 0.95
Germany 0.01
Canada 0.01
United Kingdom of Great Britain and Northern Ir... 0.01
France 0.00
Japan 0.00
Switzerland 0.00
Netherlands 0.00
India 0.00
Iran, Islamic Republic of... 0.00
Turkey 0.00
China 0.00
Norway 0.00
Australia 0.00
Romania 0.00
Spain 0.00
South Korea 0.00
TOTAL: 0.98
Q0.4: What is your professional title?¶
Postdoc
Graduate Student
Research Engineer
Research Faculty
Research Software Engineer
Manager
Other
In [5]:
questionID = "Q0.4"
title = "Respondent professional title (Demographics: Question 4)"
# print raw data
print_question(questionID=questionID)
# graph question
graph_question(questionID=questionID, title=title,
percentage=False, stacked=False,
answersIRSEM=['Faculty', 'Postdoc', 'Graduate Student', 'Research Engineer', 'Research Faculty', 'Research Software Engineer', 'Manager', 'Other'],
answersURSSI=['Faculty', 'Postdoc', 'Graduate Student', 'Research Engineer', 'Research Faculty', 'Research Software Engineer', 'Manager', 'Other'])
IRSEM survey: counts percentages
Q0.4
Faculty 17.0 0.37
Postdoc 9.0 0.20
Graduate Student 8.0 0.17
Other 6.0 0.13
Research Software Engineer 3.0 0.07
Research Faculty 2.0 0.04
Research Engineer 1.0 0.02
TOTAL: 46.0 1.00
- - - - -
URSSI survey: counts percentages
Q0.4
Faculty 669.0 0.64
Other 149.0 0.14
Research Faculty 59.0 0.06
Research Software Engineer 43.0 0.04
Graduate Student 39.0 0.04
Manager 38.0 0.04
Postdoc 36.0 0.03
Research Engineer 16.0 0.02
TOTAL: 1049.0 1.01
In [6]:
questionID = "Q0.5"
title = "Respondent age ranges (Demographics: Question 5)"
# print raw data
print_question(questionID=questionID)
# graph question
graph_question(questionID=questionID, title=title,
percentage=False, stacked=False, labelrotation=30)
IRSEM survey: counts percentages
Q0.5
35 - 44 21.0 0.46
25 - 34 15.0 0.33
19 - 24 3.0 0.07
45 - 54 3.0 0.07
65 - 74 2.0 0.04
55 - 64 1.0 0.02
75 - 84 1.0 0.02
TOTAL: 46.0 1.01
- - - - -
URSSI survey: counts percentages
Q0.5
35 - 44 366.0 0.35
45 - 54 254.0 0.24
55 - 64 184.0 0.18
25 - 34 162.0 0.16
65 - 74 53.0 0.05
18 - 24 10.0 0.01
75 - 84 8.0 0.01
85 or older 1.0 0.00
TOTAL: 1038.0 1.00
Q0.6: How many years have you worked in research?¶
Less than 1 year
1 - 5 years
6 - 10 years
11 - 15 years
16 - 20 years
20 or more years
In [7]:
questionID = "Q0.6"
title = "Respondent experience (Demographics: Question 6)"
# print raw data
print_question(questionID=questionID)
# graph question
graph_question(questionID=questionID, title=title,
percentage=False, stacked=False, labelrotation=30)
IRSEM survey: counts percentages
Q0.6
11 - 15 years 11.0 0.24
1 - 5 years 11.0 0.24
6 - 10 years 9.0 0.20
20 or more years 6.0 0.13
16 - 20 years 5.0 0.11
Less than 1 year 4.0 0.09
TOTAL: 46.0 1.01
- - - - -
URSSI survey: counts percentages
Q0.6
More than 20 years 410.0 0.39
11 - 15 years 212.0 0.20
16-20 years 178.0 0.17
6 - 10 years 162.0 0.16
1 - 5 years 76.0 0.07
Less than 1 year 5.0 0.00
TOTAL: 1043.0 0.99
In [8]:
print_question("Q0.7")
IRSEM survey: counts percentages
Q0.7
Male 39.0 0.85
Female 6.0 0.13
Prefer not to say 1.0 0.02
TOTAL: 46.0 1.00
- - - - -
URSSI survey: counts percentages
Q0.7
Male 734.0 0.70
Female 268.0 0.26
Prefer not to answer 36.0 0.03
Other 4.0 0.00
TOTAL: 1042.0 0.99
Q0.8: How often do you work on projects that have the following team size?¶
| Never | Sometimes | About half the time | Most of the time | Always | |
|---|---|---|---|---|---|
| 1 person | o | o | o | o | o |
| 2 - 5 people | o | o | o | o | o |
| 6 - 20 people | o | o | o | o | o |
| 20+ people | o | o | o | o | o |
In [9]:
questionID = "Q0.8"
title = 'Team Size (Question 0.8)'
STATEMENTS = ["1 Person",
"2 - 5 people",
"6 - 20 people",
"20+ people"]
ANSWERS = ['Always',
'Most of the time',
'About half the time',
'Sometimes',
'Never']
# print raw data
print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, percentage=True)
1 Person:
IRSEM survey: counts percentages
Q0.8_1
Sometimes 22.0 0.48
Most of the time 10.0 0.22
About half the time 9.0 0.20
Never 4.0 0.09
Always 1.0 0.02
TOTAL: 46.0 1.01
- - - - -
URSSI survey: counts percentages
Q0.8_1
Sometimes 477.0 0.52
About half the time 150.0 0.16
Most of the time 120.0 0.13
Never 115.0 0.13
Always 55.0 0.06
TOTAL: 917.0 1.00
_________________________________________________________
2 - 5 people:
IRSEM survey: counts percentages
Q0.8_2
Sometimes 17.0 0.37
Most of the time 15.0 0.33
About half the time 12.0 0.26
Never 2.0 0.04
TOTAL: 46.0 1.00
- - - - -
URSSI survey: counts percentages
Q0.8_2
Most of the time 455.0 0.45
About half the time 266.0 0.27
Sometimes 208.0 0.21
Always 61.0 0.06
Never 13.0 0.01
TOTAL: 1003.0 1.00
_________________________________________________________
6 - 20 people:
IRSEM survey: counts percentages
Q0.8_3
Sometimes 21.0 0.46
Never 18.0 0.39
About half the time 4.0 0.09
Most of the time 3.0 0.07
TOTAL: 46.0 1.01
- - - - -
URSSI survey: counts percentages
Q0.8_3
Sometimes 507.0 0.55
Never 171.0 0.19
About half the time 118.0 0.13
Most of the time 89.0 0.10
Always 29.0 0.03
TOTAL: 914.0 1.00
_________________________________________________________
20+ people:
IRSEM survey: counts percentages
Q0.8_4
Never 25.0 0.54
Sometimes 18.0 0.39
About half the time 2.0 0.04
Most of the time 1.0 0.02
TOTAL: 46.0 0.99
- - - - -
URSSI survey: counts percentages
Q0.8_4
Never 478.0 0.59
Sometimes 245.0 0.30
Most of the time 35.0 0.04
About half the time 34.0 0.04
Always 13.0 0.02
TOTAL: 805.0 0.99
RQ1: Software Development Practices¶
What activities do research software developers spend their time on, and how does this impact the perceived quality and long-term accessibility of research software?
For the following questions, think about your most recently completed software project and answer based on your experiences in that project.
[Note: This section only appears for respondent types “Developer” or “Combination”]
Q1.1: To what extent was the software targeted at each of these groups of users?¶
| Not at all | Secondary target | Primary target | |
|---|---|---|---|
| Just me | o | o | o |
| Just my team | o | o | o |
| Just my discipline | o | o | o |
| More than one discipline | o | o | o |
In [10]:
questionID = "Q1.1"
title = 'Software target groups (Software Development Practices Question 1)'
STATEMENTS = ["Just me",
"Just my team",
"Just my discipline",
"More than one discipline"]
ANSWERS = ['Primary target',
'Secondary target',
'Not at all']
# print raw data
print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, percentage=True)
Just me:
IRSEM survey: counts percentages
Q1.1_1
Not at all 9.0 0.41
Primary target 9.0 0.41
Secondary target 4.0 0.18
TOTAL: 22.0 1.00
- - - - -
URSSI survey: counts percentages
Q1.1_1
Not at all 156.0 0.38
Primary target 133.0 0.32
Secondary target 122.0 0.30
TOTAL: 411.0 1.00
_________________________________________________________
Just my team:
IRSEM survey: counts percentages
Q1.1_2
Secondary target 11.0 0.50
Not at all 10.0 0.45
Primary target 1.0 0.05
TOTAL: 22.0 1.00
- - - - -
URSSI survey: counts percentages
Q1.1_2
Primary target 205.0 0.48
Secondary target 150.0 0.35
Not at all 70.0 0.16
TOTAL: 425.0 0.99
_________________________________________________________
Just my discipline:
IRSEM survey: counts percentages
Q1.1_3
Primary target 11.0 0.50
Secondary target 8.0 0.36
Not at all 3.0 0.14
TOTAL: 22.0 1.00
- - - - -
URSSI survey: counts percentages
Q1.1_3
Primary target 226.0 0.5
Secondary target 179.0 0.4
Not at all 45.0 0.1
TOTAL: 450.0 1.0
_________________________________________________________
More than one discipline:
IRSEM survey: counts percentages
Q1.1_4
Primary target 8.0 0.36
Not at all 8.0 0.36
Secondary target 6.0 0.27
TOTAL: 22.0 0.99
- - - - -
URSSI survey: counts percentages
Q1.1_4
Secondary target 181.0 0.42
Not at all 144.0 0.33
Primary target 106.0 0.25
TOTAL: 431.0 1.00
Q1.2: What percent of your time did you actually spend in the following activities? (total should be 100%)¶
Requirements gathering / documentation:
Software architecture / design:
Coding:
Testing:
Debugging:
Maintenance:
Documentation:
Meetings:
Training:
Other:
In [11]:
#TODO
Q1.3: What percent of time would you have liked to have spent in each activity? (total should be 100%)¶
Requirements gathering / documentation:
Software architecture / design:
Coding:
Testing:
Debugging:
Maintenance:
Documentation:
Meetings:
Training:
Other:
In [12]:
#TODO
Q1.4: Which of the following types of testing did you employ?¶
| Frequently | Somewhat | Rarely | Never | |
|---|---|---|---|---|
| Unit | o | o | o | o |
| Integration | o | o | o | o |
| System | o | o | o | o |
| User | o | o | o | o |
| Regression | o | o | o | o |
In [13]:
questionID = "Q1.4"
title = 'Use of testing (Software Development Practices Question 4)'
STATEMENTS = ["Unit",
"Integration",
"System",
"User",
"Regression"]
ANSWERS = ['Frequently',
'Somewhat',
'Rarely',
'Never']
# print raw data
print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, percentage=True, labelrotation=45)
Unit:
IRSEM survey: counts percentages
Q1.4_1
Frequently 9.0 0.41
Somewhat 6.0 0.27
Rarely 4.0 0.18
Never 3.0 0.14
TOTAL: 22.0 1.00
- - - - -
URSSI survey: counts percentages
Q1.4_1
Frequently 231.0 0.51
Somewhat 108.0 0.24
Rarely 62.0 0.14
Never 52.0 0.11
TOTAL: 453.0 1.00
_________________________________________________________
Integration:
IRSEM survey: counts percentages
Q1.4_2
Frequently 10.0 0.45
Somewhat 6.0 0.27
Never 4.0 0.18
Rarely 2.0 0.09
TOTAL: 22.0 0.99
- - - - -
URSSI survey: counts percentages
Q1.4_2
Somewhat 171.0 0.38
Frequently 155.0 0.34
Rarely 69.0 0.15
Never 56.0 0.12
TOTAL: 451.0 0.99
_________________________________________________________
System:
IRSEM survey: counts percentages
Q1.4_3
Frequently 8.0 0.36
Never 5.0 0.23
Somewhat 5.0 0.23
Rarely 4.0 0.18
TOTAL: 22.0 1.00
- - - - -
URSSI survey: counts percentages
Q1.4_3
Somewhat 159.0 0.36
Frequently 118.0 0.27
Rarely 84.0 0.19
Never 80.0 0.18
TOTAL: 441.0 1.00
_________________________________________________________
User:
IRSEM survey: counts percentages
Q1.4_4
Frequently 8.0 0.36
Rarely 7.0 0.32
Somewhat 5.0 0.23
Never 2.0 0.09
TOTAL: 22.0 1.00
- - - - -
URSSI survey: counts percentages
Q1.4_4
Frequently 182.0 0.40
Somewhat 126.0 0.28
Rarely 92.0 0.20
Never 51.0 0.11
TOTAL: 451.0 0.99
_________________________________________________________
Regression:
IRSEM survey: counts percentages
Q1.4_5
Never 9.0 0.41
Rarely 6.0 0.27
Frequently 5.0 0.23
Somewhat 2.0 0.09
TOTAL: 22.0 1.00
- - - - -
URSSI survey: counts percentages
Q1.4_5
Never 146.0 0.33
Rarely 108.0 0.24
Frequently 106.0 0.24
Somewhat 81.0 0.18
TOTAL: 441.0 0.99
Q1.5: Did you release the software under an open-source license?¶
Yes / No
In [14]:
print_question('Q1.5')
graph_question(questionID='Q1.5',
title='Use of open-source license (Software Development Practices Question 5)')
IRSEM survey: counts percentages
Q1.5
Yes 21.0 0.95
No 1.0 0.05
TOTAL: 22.0 1.00
- - - - -
URSSI survey: counts percentages
Q1.5
Yes 349.0 0.74
No 122.0 0.26
TOTAL: 471.0 1.00
RQ2: Development Infrastructure and Tools¶
What tools do research software developers use and what additional tools are needed to support sustainable development practices?
[Note: This section only appears for respondent types “Developer” or “Combination”]
The following questions are about software development infrastructure, tools, and practices with which you may use or are familiar.
Q2.1: In the last 5 years, how much experience do you have with each of the following types of projects:¶
| None | 1-3 projects | 4+ projects | |
|---|---|---|---|
| Single developer | o | o | o |
| Co-located team (i.e. a team, located in a single place or institution with well-coordinated funding/organization) | o | o | o |
| Distributed team (i.e. a well-defined team, located in multiple places or institutions with well-coordinated funding/organization) | o | o | o |
| Open-source community (i.e. a group of developers who are distributed and working under multiple independant funding sources) | o | o | o |
In [15]:
questionID = "Q2.1"
title = 'Project experience. (Development Infrastructure and Tools: Question 1)'
STATEMENTS = ["Single developer",
"Co-located team",
"Distributed team",
"Open-source\ncommunity"]
ANSWERS = ['1-3 projects',
'4+ projects']
# print raw data
print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, labelrotation=45, percentage=False, stacked=False)
Single developer:
IRSEM survey: counts percentages
Q2.1_1
1-3 projects 11.0 0.58
4+ projects 8.0 0.42
TOTAL: 19.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.1_1
1-3 projects 222.0 0.69
4+ projects 100.0 0.31
TOTAL: 322.0 1.00
_________________________________________________________
Co-located team:
IRSEM survey: counts percentages
Q2.1_2
1-3 projects 11.0 0.73
4+ projects 4.0 0.27
TOTAL: 15.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.1_2
1-3 projects 260.0 0.73
4+ projects 96.0 0.27
TOTAL: 356.0 1.00
_________________________________________________________
Distributed team:
IRSEM survey: counts percentages
Q2.1_3
1-3 projects 10.0 0.67
4+ projects 5.0 0.33
TOTAL: 15.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.1_3
1-3 projects 270.0 0.8
4+ projects 67.0 0.2
TOTAL: 337.0 1.0
_________________________________________________________
Open-source
community:
IRSEM survey: counts percentages
Q2.1_4
1-3 projects 12.0 0.63
4+ projects 7.0 0.37
TOTAL: 19.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.1_4
1-3 projects 188.0 0.78
4+ projects 52.0 0.22
TOTAL: 240.0 1.00
Q2.2: How well do current tools support each of the following activities?¶
| Extremely supported | Very supported | Moderately supported | Slightly supported | Not supported at all | |
|---|---|---|---|---|---|
| Requirements gathering / documentation | o | o | o | o | o |
| Software architecture / design | o | o | o | o | o |
| Coding | o | o | o | o | o |
| Testing | o | o | o | o | o |
| Debugging | o | o | o | o | o |
| Maintenance | o | o | o | o | o |
| Documentation | o | o | o | o | o |
Figure 4: Availability of tool support. (Question 2.2)
In [16]:
questionID = "Q2.2"
title = 'Availability of tool support. (Development Infrastructure and Tools: Question 2)'
STATEMENTS = ['Requirements',
'SW Arch/Design',
'Coding',
'Testing',
'Debugging',
'Maintenance',
'Documentation']
ANSWERS = ['Extremely supported',
'Very supported',
'Moderately supported',
'Slightly supported',
'Not supported at all']
# print raw data
print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, labelrotation=60, stacked=True, percentage=True)
Requirements:
IRSEM survey: counts percentages
Q2.2_1
Moderately supported 9.0 0.45
Slightly supported 6.0 0.30
Very supported 2.0 0.10
Not supported at all 2.0 0.10
Extremely supported 1.0 0.05
TOTAL: 20.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.2_1
Slightly supported 155.0 0.36
Moderately supported 127.0 0.29
Not supported at all 91.0 0.21
Very supported 51.0 0.12
Extremely supported 11.0 0.03
TOTAL: 435.0 1.01
_________________________________________________________
SW Arch/Design:
IRSEM survey: counts percentages
Q2.2_2
Moderately supported 7.0 0.35
Slightly supported 6.0 0.30
Very supported 5.0 0.25
Not supported at all 2.0 0.10
TOTAL: 20.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.2_2
Moderately supported 160.0 0.36
Slightly supported 138.0 0.31
Very supported 71.0 0.16
Not supported at all 59.0 0.13
Extremely supported 11.0 0.03
TOTAL: 439.0 0.99
_________________________________________________________
Coding:
IRSEM survey: counts percentages
Q2.2_3
Extremely supported 11.0 0.55
Very supported 6.0 0.30
Moderately supported 2.0 0.10
Slightly supported 1.0 0.05
TOTAL: 20.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.2_3
Very supported 208.0 0.47
Extremely supported 133.0 0.30
Moderately supported 84.0 0.19
Slightly supported 15.0 0.03
Not supported at all 2.0 0.00
TOTAL: 442.0 0.99
_________________________________________________________
Testing:
IRSEM survey: counts percentages
Q2.2_4
Very supported 8.0 0.40
Moderately supported 5.0 0.25
Slightly supported 4.0 0.20
Extremely supported 2.0 0.10
Not supported at all 1.0 0.05
TOTAL: 20.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.2_4
Moderately supported 167.0 0.38
Very supported 143.0 0.32
Slightly supported 66.0 0.15
Extremely supported 53.0 0.12
Not supported at all 13.0 0.03
TOTAL: 442.0 1.00
_________________________________________________________
Debugging:
IRSEM survey: counts percentages
Q2.2_5
Moderately supported 10.0 0.50
Very supported 5.0 0.25
Slightly supported 3.0 0.15
Extremely supported 2.0 0.10
TOTAL: 20.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.2_5
Moderately supported 188.0 0.43
Very supported 135.0 0.31
Slightly supported 56.0 0.13
Extremely supported 53.0 0.12
Not supported at all 10.0 0.02
TOTAL: 442.0 1.01
_________________________________________________________
Maintenance:
IRSEM survey: counts percentages
Q2.2_6
Moderately supported 8.0 0.40
Slightly supported 8.0 0.40
Very supported 3.0 0.15
Extremely supported 1.0 0.05
TOTAL: 20.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.2_6
Moderately supported 174.0 0.40
Slightly supported 128.0 0.29
Very supported 72.0 0.16
Not supported at all 44.0 0.10
Extremely supported 21.0 0.05
TOTAL: 439.0 1.00
_________________________________________________________
Documentation:
IRSEM survey: counts percentages
Q2.2_7
Moderately supported 9.0 0.45
Very supported 6.0 0.30
Extremely supported 3.0 0.15
Slightly supported 2.0 0.10
TOTAL: 20.0 1.00
- - - - -
URSSI survey: counts percentages
Q2.2_7
Moderately supported 167.0 0.38
Slightly supported 110.0 0.25
Very supported 108.0 0.24
Not supported at all 35.0 0.08
Extremely supported 22.0 0.05
TOTAL: 442.0 1.00
RQ3: Training¶
What training is available to research software developers and does this training meet their needs?
The following questions are aimed to help us understand where developers may have received formal or informal training for software development.
Q3.1: Have you received training for software development?¶
Yes / No
In [17]:
print_question("Q3.1")
IRSEM survey: counts percentages
Q3.1
No 21.0 0.58
Yes 15.0 0.42
TOTAL: 36.0 1.00
- - - - -
URSSI survey: counts percentages
Q3.1
No 567.0 0.65
Yes 302.0 0.35
TOTAL: 869.0 1.00
Q3.1.1: Where? (if answered yes to 3.1)¶
Multiple select:
Class / school
Carpentry
Online self-directed
Other
In [18]:
print_question("Q3.1.1", multiselect=True)
IRSEM survey: counts percentages
Q3.1.1
Class / school 12.0 0.55
Online self-directed 8.0 0.36
Other 2.0 0.09
TOTAL: 22.0 1.00
- - - - -
URSSI survey: counts percentages
Q3.1.1
Class / school 249.0 0.53
Online self-directed 146.0 0.31
Other 44.0 0.09
Carpentry 32.0 0.07
TOTAL: 471.0 1.00
Q3.2: Do you believe there are sufficient opportunities available for exploring or obtaining new software skills?¶
Yes / No / Don't know
In [19]:
print_question("Q3.2")
IRSEM survey: counts percentages
Q3.2
Yes 19.0 0.53
No 10.0 0.28
Don't know 7.0 0.19
TOTAL: 36.0 1.00
- - - - -
URSSI survey: counts percentages
Q3.2
Yes 451.0 0.52
No 233.0 0.27
Don't know 181.0 0.21
TOTAL: 865.0 1.00
Q3.3: Do you have sufficient time to take the types of training you need to be successful?¶
Yes / No
In [20]:
print_question("Q3.3")
IRSEM survey: counts percentages
Q3.3
Yes 18.0 0.5
No 18.0 0.5
TOTAL: 36.0 1.0
- - - - -
URSSI survey: counts percentages
Q3.3
No 642.0 0.74
Yes 222.0 0.26
TOTAL: 864.0 1.00
Q3.4: For each of the following types of training (languages, development techniques, and project management), which mode of delivery do you prefer? Please select your top two modes of delivery preferences for each of the training types.¶
| Languages | Development techniques | Project Management | |
|---|---|---|---|
| On-site custom training | ▢ | ▢ | ▢ |
| Workshops or short courses (e.g., before or after a conference) | ▢ | ▢ | ▢ |
| Carpentries / software boot camps | ▢ | ▢ | ▢ |
| Summer schools (in-person or virtual) | ▢ | ▢ | ▢ |
| MOOCs (Massive Open Online Courses), webinars, or self-directed online courses | ▢ | ▢ | ▢ |
| Other | ▢ | ▢ | ▢ |
In [21]:
questionID = "Q3.4"
title = 'Preferred training. (Training: Question 4)'
STATEMENTS = ["On-site custom training",
"Workshops/short courses",
"Carpentries/software boot camps",
"Summer schools",
"MOOCs/webinars/online courses",
"Other"]
ANSWERS = ['Languages',
'Development Techniques',
'Development techniques',
'Project Management']
# print raw data
print_question_set(question=questionID, statements=STATEMENTS, multiselect=True)
#graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, percentage=True)
On-site custom training:
IRSEM survey: counts percentages
Q3.4_1
Development techniques 18.0 0.38
Languages 15.0 0.32
Project Management 14.0 0.30
TOTAL: 47.0 1.00
- - - - -
URSSI survey: counts percentages
Q3.4_1
Project Management 307.0 0.36
Development Techniques 295.0 0.35
Languages 242.0 0.29
TOTAL: 844.0 1.00
_________________________________________________________
Workshops/short courses:
IRSEM survey: counts percentages
Q3.4_2
Development techniques 28.0 0.44
Languages 22.0 0.35
Project Management 13.0 0.21
TOTAL: 63.0 1.00
- - - - -
URSSI survey: counts percentages
Q3.4_2
Development Techniques 449.0 0.40
Project Management 346.0 0.31
Languages 331.0 0.29
TOTAL: 1126.0 1.00
_________________________________________________________
Carpentries/software boot camps:
IRSEM survey: counts percentages
Q3.4_3
Development techniques 14.0 0.52
Languages 11.0 0.41
Project Management 2.0 0.07
TOTAL: 27.0 1.00
- - - - -
URSSI survey: counts percentages
Q3.4_3
Development Techniques 297.0 0.41
Languages 286.0 0.39
Project Management 149.0 0.20
TOTAL: 732.0 1.00
_________________________________________________________
Summer schools:
IRSEM survey: counts percentages
Q3.4_4
Languages 18.0 0.45
Development techniques 18.0 0.45
Project Management 4.0 0.10
TOTAL: 40.0 1.00
- - - - -
URSSI survey: counts percentages
Q3.4_4
Development Techniques 272.0 0.39
Languages 244.0 0.35
Project Management 181.0 0.26
TOTAL: 697.0 1.00
_________________________________________________________
MOOCs/webinars/online courses:
IRSEM survey: counts percentages
Q3.4_5
Development techniques 14.0 0.40
Languages 11.0 0.31
Project Management 10.0 0.29
TOTAL: 35.0 1.00
- - - - -
URSSI survey: counts percentages
Q3.4_5
Languages 399.0 0.42
Development Techniques 307.0 0.32
Project Management 249.0 0.26
TOTAL: 955.0 1.00
_________________________________________________________
Other:
IRSEM survey: counts percentages
Q3.4_6
Development techniques 10.0 0.36
Project Management 10.0 0.36
Languages 8.0 0.29
TOTAL: 28.0 1.01
- - - - -
URSSI survey: counts percentages
Q3.4_6
Languages 64.0 0.40
Development Techniques 49.0 0.31
Project Management 47.0 0.29
TOTAL: 160.0 1.00
RQ4: Funding and Institutional Support¶
RQ4a: What is the available institutional support for research software development?
RQ4b: What sources of institutional funding are available to research software developers?
The following questions are aimed at understanding your funding and institutional support for research software activities.
Q4.1: Have you ever included costs for software development in a funding proposal?¶
Yes / No
In [22]:
print_question("Q4.1")
IRSEM survey: counts percentages
Q4.1
No 28.0 0.78
Yes 8.0 0.22
TOTAL: 36.0 1.00
- - - - -
URSSI survey: counts percentages
Q4.1
No 451.0 0.54
Yes 385.0 0.46
TOTAL: 836.0 1.00
Q4.1.1: Select all costs for software development in your funding proposals? (if answered yes to 4.1)¶
Multiple Choice:
For developing new software
For reusing existing software
For maintaining / sustaining software
In [23]:
print_question("Q4.1.1", multiselect=True)
IRSEM survey: counts percentages
Q4.1.1
For developing new software 6.0 0.6
For maintaining / sustaining software 4.0 0.4
TOTAL: 10.0 1.0
- - - - -
URSSI survey: counts percentages
Q4.1.1
For developing new software 343.0 0.48
For maintaining / sustaining software 210.0 0.29
For reusing existing software 160.0 0.22
TOTAL: 713.0 0.99
Q4.2: What percent of the funding for your software work comes from each of the following? (if Developer or Combination selected) (total should be 100%)¶
NSF:
NIH:
DoD:
NASA:
NOAA:
Your own institution (e.g. faculty salary)
Other:
In [24]:
#TODO
Q4.3: Does your institution provide support for your research software development activities in the following ways: (if Developer or Combination selected)¶
| No | Yes, but inadequate level of support | Yes, adequate level of support | |
|---|---|---|---|
| Financial | o | o | o |
| Infrastructural (repositories, computing, etc.) | o | o | o |
| Consulting / RSE support | o | o | o |
| Other | o | o | o |
In [25]:
questionID = "Q4.3"
title = 'Institutional support provided. (Funding and Institutional Support: Question 3)'
STATEMENTS = ["Financial",
"Infrastructural",
"Consulting\n/ RSE support",
"Other"]
ANSWERS = ['No',
'Yes, but inadequate level of support',
'Yes, adequate level of support']
# print raw data
print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, percentage=False, stacked=False)
Financial:
IRSEM survey: counts percentages
Q4.3_1
Yes, adequate level of support 8.0 0.42
No 7.0 0.37
Yes, but inadequate level of support 4.0 0.21
TOTAL: 19.0 1.00
- - - - -
URSSI survey: counts percentages
Q4.3_1
No 250.0 0.60
Yes, but inadequate level of support 110.0 0.26
Yes, adequate level of support 60.0 0.14
TOTAL: 420.0 1.00
_________________________________________________________
Infrastructural:
IRSEM survey: counts percentages
Q4.3_2
Yes, adequate level of support 7.0 0.37
Yes, but inadequate level of support 7.0 0.37
No 5.0 0.26
TOTAL: 19.0 1.00
- - - - -
URSSI survey: counts percentages
Q4.3_2
Yes, adequate level of support 175.0 0.41
Yes, but inadequate level of support 169.0 0.40
No 79.0 0.19
TOTAL: 423.0 1.00
_________________________________________________________
Consulting
/ RSE support:
IRSEM survey: counts percentages
Q4.3_3
No 14.0 0.74
Yes, adequate level of support 5.0 0.26
TOTAL: 19.0 1.00
- - - - -
URSSI survey: counts percentages
Q4.3_3
No 241.0 0.59
Yes, but inadequate level of support 103.0 0.25
Yes, adequate level of support 65.0 0.16
TOTAL: 409.0 1.00
_________________________________________________________
Other:
IRSEM survey: counts percentages
Q4.3_4
No 10.0 0.62
Yes, adequate level of support 5.0 0.31
Yes, but inadequate level of support 1.0 0.06
TOTAL: 16.0 0.99
- - - - -
URSSI survey: counts percentages
Q4.3_4
No 15.0 0.79
Yes, adequate level of support 3.0 0.16
Yes, but inadequate level of support 1.0 0.05
TOTAL: 19.0 1.00
Q4.4: Does your institution provide / support the research software development you need in the following ways: (if Developer or Combination selected)¶
| No | Yes, but inadequate level of support | Yes, adequate level of support | |
|---|---|---|---|
| Financial | o | o | o |
| Infrastructural (repositories, computing, etc.) | o | o | o |
| Consulting / RSE support | o | o | o |
| Other | o | o | o |
In [26]:
questionID = "Q4.4"
title = 'Institutional support needed. (Funding and Institutional Support: Question 4)'
STATEMENTS = ["Financial",
"Infrastructural",
"Consulting\n/ RSE support",
"Other"]
ANSWERS = ['No',
'Yes, but inadequate level of support',
'Yes, adequate level of support']
# print raw data
print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, percentage=False, stacked=False)
Financial:
IRSEM survey: counts percentages
Q4.4_1
No 9.0 0.47
Yes, adequate level of support 7.0 0.37
Yes, but inadequate level of support 3.0 0.16
TOTAL: 19.0 1.00
- - - - -
URSSI survey: counts percentages
Q4.4_1
No 240.0 0.64
Yes, but inadequate level of support 77.0 0.21
Yes, adequate level of support 56.0 0.15
TOTAL: 373.0 1.00
_________________________________________________________
Infrastructural:
IRSEM survey: counts percentages
Q4.4_2
Yes, adequate level of support 7.0 0.37
No 6.0 0.32
Yes, but inadequate level of support 6.0 0.32
TOTAL: 19.0 1.01
- - - - -
URSSI survey: counts percentages
Q4.4_2
Yes, but inadequate level of support 174.0 0.46
Yes, adequate level of support 127.0 0.33
No 79.0 0.21
TOTAL: 380.0 1.00
_________________________________________________________
Consulting
/ RSE support:
IRSEM survey: counts percentages
Q4.4_3
No 13.0 0.68
Yes, adequate level of support 5.0 0.26
Yes, but inadequate level of support 1.0 0.05
TOTAL: 19.0 0.99
- - - - -
URSSI survey: counts percentages
Q4.4_3
No 178.0 0.49
Yes, but inadequate level of support 125.0 0.35
Yes, adequate level of support 59.0 0.16
TOTAL: 362.0 1.00
_________________________________________________________
Other:
IRSEM survey: counts percentages
Q4.4_4
No 10.0 0.62
Yes, adequate level of support 5.0 0.31
Yes, but inadequate level of support 1.0 0.06
TOTAL: 16.0 0.99
- - - - -
URSSI survey: counts percentages
Q4.4_4
No 10.0 0.83
Yes, but inadequate level of support 2.0 0.17
TOTAL: 12.0 1.00
Q4.5: What are the most significant gaps in your current institutional / organizational support of software related research?¶
Text input
In [27]:
#print_question("Q4.5", percentages=False)
print(IRSEM["Q4.5"][2:].dropna().to_list())
["Lack of time and recognition of the work. It's better for me to keep cranking out research papers than code.", "Can't answer this well since I've just moved institutions", 'I have not yet gotten to know my institution well enough to discover who the experts are, and who is willing to support my needs in mathematical research.', 'Lack of computing power', 'No grants', 'IT does not support Linux, and a lot of mathematical research software is more easily available on Linux. My IT in particular has also failed to install version of the software that could run on Windows. :-(', 'lack of money', 'idk', 'collaboration', 'Lack of recognition as important for research, lack of funding for proprietary software', 'Work done on developing software to help with pure math research is not appreciated enough in the pure math community.', 'financial', 'It doesnt seem like people are aware of or interested in using software in their research. ', 'An arcane process for applying for reassigned time']
In [28]:
print(URSSI["Q4.5"][2:].dropna().to_list())
['Time', "Not enough support, via matching funds, subsidy, or 'baseline amount', for software support. This applies both the ongoing authorship, but also to ongoing support for the use and/or customization of existing packages. In addition to writing the software, creating examples, good documentation, and backtracking to make written software more stable and less fragile is needed -- there is a large number of 'libraries' or 'packages' written in languages like R, Python, Ruby, and the like that have large dependency lists on specific minor, and in some cases sub-minor, version of yet other libraries or packages. Creation is good, but it would be good to provide funding for documentation, more and better working examples, and for refactoring and maintenance.", 'It is a soft-money issue, finding more money to pay for the effort and convincing people the work is important.', "Recognition of how long it takes to make good software. Unrealistic delivery timelines. Insular issues, ie my main funding agency only wants to know how it will benefit THEM not science as a whole. Lack of appreciation just how important good software is. \n\nProgram managers have no concept of CI, Unit testing etc.. ie they don't know what good SE practices entail. ", 'they do not care.', 'My activities are strongly supported by my institution ', 'Lack of communication between researchers and professional software developers', 'Time and recognition of effort needed.', "It's not taken seriously. It's all about the hardware. Software engineers are treated as second class citizens.", 'The most significant gaps are lack of direction and leadership. ', 'Lack of problem definition by researchers\nLack of properly trained faculty/staff to address software development\nLack of strategic direction for software development and maintenance of resulting product.', 'Many of the researchers and groups we interact with are incredibly particular about how THEY want to do their work which generally does not align with modern good practices. ', 'Windows only platforms. No one is doing Java, no one is doing/teaching Cuda, C, etc.', 'Cultural isolation, and no RSE around.', 'We could really use an in-house set of SE standards that codes/teams need to meet depending on their intended level of maturity, and then with that a team dedicated (100% of their time) to training and consulting efforts to help projects get from point A to point B.', 'Ability and understanding of being able to release and develop open source software. The rules and approaches are not well understood or defined. \n\nAlso, the effort to write software is always underestimated and the maintenance cost is always ignored ', 'Need more flexible infrastructure for repo and cloud compute platforms.', 'we are weak in organizational planning, user interaction/support for research computing, and institutional support for research computing. \n\nWhile we have a limited amount of physical resources (network, storage, compute), we lack a coherent plan for research computing, which results poor resource decision making and a net negative user support model.', 'Distribution Support', 'The promotional track in not as well defined for faculty who primarily write software as for those who primarily write traditional publications on their research.', 'How software is maintained after initial development (who will fund etc), and the metrics upon which the researchers are judged a "success"', "We depend on a single developer. While this individual is good, it is likely we don't benefit from having more of a community of biological software developers (outside of BOSC) to be linked to, and therefore better reuse and share. The developer has never gotten to BOSC and there is not much recognition of the need to be part of the community rather than an isolated developer. ", 'Nobody values what I do.', "We don't have a repository where code can be uploaded bu the researcher, preserved, and run virtually at a future date. ", 'Often the lead scientist in not aware of the software issues/programming aspects. The focus is always on the scientific obstacles and not on the programming obstacles that need to be overcome.', "No financial support for resources (equipment, computing time, etc.) outside of the grant that pays my salary. If it's in the grant's budget, fine. if not, best of luck to you.", 'All of the above (see prior question).', 'retaining senior/experienced talent and availability of senior talent. ', 'The most significant gap to find people who are interested or willing to work in different software. People generally prefer not to involve.', 'Recognizing that software development is an academic, intellectual, creative pursuit on par with basic research. ', 'Uncertainty in funding support from agencies, lack of understanding from institutional administrators regarding software development/maintenance', 'There is no support to make our open source software available when grant funding is over (sustainability is lacking).', 'My institution is a national lab operated for DoE. The perception appears to be that "someone" in DOE funds software development so there is little to now support from basic science or applied sciences for funding software. The assumption seems to be that it just happens by magic.', 'lack on self guided training in new languages', 'support', 'Excellent question, but not sure how to answer it.\n\nI need someone at my institution like Martin Kandes -- user support Comet computer in the XSEDE system.', 'There is almost no support for software development. It is accepted in the community that development is part of the job (as a Ph.D. or Postdoc or etc) and that is the major reason that even though many good works have been done but there is not good documentation, no maintenance, and no sharing. Especially in the engineering department open source almost does not have any meaning.', 'Lack of sustaining support for "cyberinfrastructural software maintenance and enhancement". Always have to "innovate" on the next project, rather than enhance/tune/debug/support existing frameworks', 'No recognition of time/effort/resources spent on software related research.', 'HIRE PERSONNEL\nDEFINE CAREER PATHS FOR SOFTWARE EXPERTS\nRECEIVING CREDITS FOR SOFTWARE DEVELOPMENT (Difficult to publish method development, on applications studies only the first author -- code user -- typically gets the credit)\n', 'workshops dealing with advanced scientific programming', "Funding is not there, and there isn't much reward for software development...", 'Recognition of software products in tenure/promotion.', 'Modern practices like code review, training in language idioms and style.', 'My own time is so limited. There is not even enough time to leverage collaborators. When I can get their interest, they quickly lose interest because I am too busy to stay engaged and maintain their interest. Collaborators often see engagement as a losing proposition, because they could opt to work for people who are more available to work on the projects with them.', 'Funding', 'We do not have much of a university-level appreciation of software/computing-related research (e.g., no university-level research computing organization).', 'Documentation of best practices for portability and user guides.', 'cross intergation', 'There is very little formal credit given for software-related research projects', 'Little overall education on software sustainability', 'Planning and limiting the scope of work and obligations to support it over time\nInsufficient documentation nor planning for transition of maintenance responsibilities\nCommitment to adequately support entire software development lifecycle from inception through retirement', 'It would be nice to have a space to do my research .... and move my computers out of my kitchen.', 'Training for students.', 'Ability to hire full-time research software engineers.', 'Support for training.', 'Lack of personnel to fill out teams.', "Columbia U has no real support for any of our software development efforts. I honestly don't know how they could help. They don't even have a good sysadmin to help out. ", 'lack of Linux system level support for our servers and workstations, commercial rates for equipment hosting, lack of high speed internet connection', "Most students only know a language. They aren't aware of larger development concepts including even basic version control, unit testing, etc. I encourage students to learn these things and point them to online resources, but many students never become competent in this space so we get lots of bad software. I certainly share part of the blame for this, but when they come work with me they're not coming with a strong interest in learning software development. They don't consider that research and by the time they understand the importance they're finished.", 'Not enough research support and local training opportunites', 'Sustained institutional support for technicians to help with high-performance computing and development.', 'access to large data sets within the institution. \nIssues with cloud computing owing to a lack of licenses, and slow computing services through the institute cloud server', 'Experts in computer systems who maintain computers.\n\n\nToo many administrative complications that do not allow to manage time well.', 'Choice of area usually decided by user volume which means that are important needs which are not supported because of a perceived lack of critical mass. An example would be software related to optimal use of available high-performance computing resources, both CPU ad GPU based.', "It's more about the culture", 'lack of: 1. professional tech support for appropriate computing infrastructure. 2. internal support for necessary computing hardware.', 'No thought about this.', 'Our primary project, a national/international digital repository can fund most but not all operating costs through external sources, however supporting the organizational infrastructure (e.g. marketing use of the software and services, institutional commitments of the management team, and board activities) has proved extremely difficult and the University has not provided support. ', 'Lack of workflow expertise', 'The purchase of software, aside from very standard simple programs, is not covered by the institution. Purchase, and particularly updates after purchase, are not provided.', 'No recognition of software development / maintenance during the tenure and promotion process. If it does not have dollars associated, or the impact cannot be tallied, it is deemed less important.', "I don't expect my institution to directly fund much research software development. However, I do expect it to support long-term data storage and archiving, but, right now, that support is not sufficiently well developed to be easy to use.", "I don't know enough about software development to answer these questions", 'We have no software developers and rely on poorly trained students.', 'too few people to support the diverse research computing needs in the university\n(only 3 people for research in C&IT for a R1 university with 35,000 students)', "As a faculty, it's near impossible to keep up with developments. I would need something to help me stay on top of coding, while fitting in my schedule.", 'Software related research is done in a separate department than the department doing the research.', 'HPC. Access to clusters of increasing sizes for development, scaling, and production. Right now, I have access through NSF-sponsored resources to big machines but I am lacking access to smaller machines that I can use to ensure I do not waste cycles on the larger machines.', 'Red tape for computer purchases, insufficient dedicated effort to assist faculty with software and computational needs.', '1. Inadequate financial planning/budgeting\n', 'I have to do all of the development, and sometimes we have software resources I can utilize.', 'Probably I will repeat this. \nI do not have formal software training. It is a skill learned out of necessity. For example our current project is a large project using Matlab. It started out as a simple project, but grew as we understood what we could do. It would have been difficult to map out the project from the start. The goals are sometimes difficult to articulate and technical, specific to the science we do. Communicating those to our software support would require more time that they can possibly allot to my research group.', 'Providing more training and boot camps', 'No institutional funding of salaries for research software engineers.', 'repositories, developing/coding resources, free or reasonably cheap ', 'ITS is clueless.', 'Training and hardware for coprocessor-, GPU-, and FPGA-based AI applications.', 'Lack of recognition that software development/maintenance is a valuable and valued research activity (when it comes to promotion, awards, etc).', 'Well trained professional to aid with software releases and organization', 'Training on cutting edge techniques', 'Instruction\nCapacity\nExpertise', 'I noted "No" above because there is no financial support specifically for software development. There are some funds that can be used for these activities, but they are, for example faculty startup funds. ', 'Not sure this answers the question, but I think we need to offer coursework in robotics software development, rather than expecting everyone to learn all needed skills "on the job" in research labs.', 'Lack f interest and commitment from administration. Dean of engineering who does not respect software development. ', 'Funding for software development', "I'm an (unpaid) adjunct faculty member at a mid-western academic medical center. The current generation of research administrators at these types of institutions are woefully ignorant of the time and cost of developing software for reproducible research. Frankly, they don't really care about research reproducibility at all, and they don't have any experience with software development; they just care about how a faculty member generates F&A (indirect) costs from her/his grant. All they care about is money, not ideas or advancing science or medical research.", 'Young scientists (graduates) have not practical training in CSE', 'There is no support.', 'Maintenance', "Problems with licensing software tools (university has very restrictive views of EULAs, often denies us permission to purchase licenses if the vendor won't make minor changes in the EULA).\n\nLack of support for an open-source (GitHub-like) repository, technology commercialization office is intrusive, does not understand most research software is not commercializable and of interest only to other researchers.", 'no idea', 'Would love some in house consultants or resources to help frame and scope projects. Big need for long term infrastructure for management and maintenance of software and data.', 'Time allotted to students and resource provided to students to learn software development. ', 'Lack of experienced software developers who have worked in multiple large open source teams to train/consult with research students who are writing, testing, debugging, maintaining the code.', 'Not many researchers in my department do computational work', 'Training in software development and project management.', 'peer influence and support team e.g. working side by side with fellow peers not necessarily on the same code can help motivate and inch along a novice.', 'Targeting individuals in the "long-tail of science" that are not programmers or developers by training, but rather domain scientists that have picked it up by necessity. There is a preponderance of software carpentry literate scientists that are deemed incompetent by software developers. Finding resources to train scientists in these advanced user, but not quite software developer, levels is exceedingly challenging.', "It is virtually impossible to get software developments effort funded at all. I've tried many times and it was always turned down, e.g., because universities were supposed to provide funding for software developers themselves, which they don't. It's mostly spare time activities no matter how central the software is to achieving our mission and, especially, disseminating it to others.", 'more software packages should be available for us such as Origin or LAbview', 'Insufficient financial support for ongoing maintenance of projects after initial release, and I am not aware of any software consulting services offered for any stage of the work', 'Skepticism at the administrative level that software-related research and teaching is worth performing at a liberal arts college.', "I don't know.", 'DNA sequence management and cloning; biological graphing software', 'no support', 'The Minnesota supercomputing institute has reasonable hardware, but a number of issues pertaining to some software and queueing that pose significant challenges. Other university clusters seem to perform much better.', 'No academic credit as a professor. So essentially, pretty soon I have to stop developing software if I want to be evaluated highly by my peers and dean/chair in a reputable place since none of my development is regarded as academic output.', 'Need more people paid for who have the time and job to do it.', 'Funding, storage', 'We have no budget management software', 'Needs for software to perform analysis of next generation sequencing data', 'Subcritical mass of developers', 'Staff with specific knowledge of my research area.', "Our university does not provide any funds for software and you can't purchase it off a grant unless it is specifically written into the grant, but if you write it into the grant it will be stricken (because they know the software won't just be used for that project). So, no way to really obtain software unless you are lucky enough to have soft money hanging around or pay out of pocket.", 'Development efforts are not considered as critical; finding developers to support senior researchers is difficult; ', 'no infrastructure support and no time release to learn new skills in software related research', 'No office at. the university to assist ', 'No institutional infrastructure for software engineering expertise', 'Hard to fund maintenance of software systems', 'For a research software system, it is nearly impossible to fund a high-end software developer who will help to maintain and manage a large-scale (massively parallel, leadership supercomputer class) software project within a university. We need the NSF to encourge Google-level salaries of these positions so we can keep this expertise in the software research domain at universities (and even national labs). ', 'Need larger storage servers and faster processing computers, but we are outsourcing these things to a larger university with the capacity. Also we need funds for any more technical software.', 'Advice on who to contract with, who is the best fit for the job', 'School computers are managed remotely, and users not allowed to install anything without case-by-case applications for permission. Support for computing is for staff hired specifically to manage computers or create clinical databases, not for scientists or other users.', 'Technical and specific knowledge and training', 'Complete lack of communication about what is actually available', 'The transition from script to application. ', 'Lack of funds', 'My institution has no systematic way to educate new researchers (graduate students, undergraduate students) about software development.\n\nThere is not a member of our HPC staff with subject-matter expertise in my research area, so frequently they cannot provide an appropriate level of support.', 'Lack of people who understand my work to bounce ideas off of', 'I am not aware of any support related to my college unit; perhaps this type of work is supported in other colleges at my university.', "Each researcher here (small liberal arts college) is pretty much left to their own devices. There's not a lot of institutional support of any type other than what people can cobble together on their own.", 'Developer support, backend support, development environments', 'Financial', 'Platforms like Mathematica are not supported for graduate research, only for undergraduate education.', 'In small business it is challenging to develop new programs and adequately test them and maintain a line of stable re-usable tools (that pay the bills).', 'There is no institutional support for our research, including software-related issues.', 'We rely on government grants (NIH) for software development.', 'supercomputing support', 'Trial by fire with little background training.', 'Lack of understanding', 'No long-term support or funding for software development, only publications are recognized as a valuable contribution.', 'The institutional high-performance cluster (HPC) does not provide enough software for research, especially the GUI based software (e.g. R-studio).', "We have no institutional code repository, and our existing computing resources are too closely tied to specific funded projects- as opposed to being general infrastructure for people to use for exploratory work. Our research software engineering group is extremely overburdened, and though they do the best they can there are gaps in their capabilities. Also, our University's policies towards intellectual property are retrograde at best, and do not support research coding at all.", 'Lack of experienced programmers', 'Lack of virtual machine / cloud support on campus cluster', 'technical help to implement concepts for which we do not have the training (internal in our group) to develop ourselves.', 'training and funding ', "up to date hardware. I have to get external support for new hardware. Only little internal consultants available at the level needed, and they are faculty members from other departments. Budget cuts removed my technology support person from the institution four years ago. Replacements don't know enough to really support the innovations. I've partnered with industry to get the support I need. ", 'Personnel ', 'Our salaries are lower in a grant-funded institution than through the commercial sector so we may get new software developers/coders. When they have any experience, they then leave to make much more money in the private/start-up sector.', 'Not sure', 'We should have a software core facility to help with code development and maintenance. ', 'The disparity/uncertainty/tension that exists between colleges/departments of Electrical Engineering, Computer Science, Information Technology and Business Administration regarding "who should be teaching/researching what," in this vast realm of "software."', 'number of people who can teach, trouble-shoot, etc. Staff is over worked and often under trained themselves. ', 'There is almost none other than hardware and network infrastructure. The gaps are everywhere. ', 'They are over-protective and make it very difficult by insisting on full compliance, at all times, with all applicable laws including export controls and software licencing policies.', 'The primary gap is in physical distance and communication between the faculty/dept/students who have software development skills and the faculty/departments who use (or would use) such technology for applied purposes (app interventions, etc). ', 'There is no institution-wide structure whose focus is this.', 'Once a project is completed, there\'s no support for maintenance and it\'s no longer "research" -- basically, no interest or funding for maintenance.', 'I would prefer it if there was stronger integration between the bioinformatics groups (this is my field) and the software engineering / computer science / machine learning / statistics groups at my institution. As things stand, it is very easy to self-sequester, and this leads to stagnation in methodology and praxis.', 'Significant gap wrt building new infrastructure for existing platform.', 'Lack of cloud infrastructure for deploying web-facing software; lots of air-gapped HPC', 'It is hard to have programmers on soft money. ', 'It is hard to get support for certain vital tools, like getting gdb to run on an iMac.', 'Connections between end users in departments outside engineering and developers', "Most assistance is fee-for-service, and I don't have a budget for this. ", 'Lack of time', 'Training of students in software engineering.', 'Academic institutional support is focusing almost exclusively on security, and aims to simplify workstation management, without consideration for local research specific requirements. Instead, it would be more useful to create local "sandbox" environments, where resources can be focused on local sharing, rather than restricting use. Cloud based approaches are not always useful, since data generated locally can be enormous in size.', 'Development of software for image processing', 'Trained support staff for standard issues (installation, maintenance, support for Unix-based systems). This is present, but over-burdened. Use of scientific software from open-source projects usually involves specialized knowledge for installation and maintenance, which is often beyond my abilities.', 'Release time to actually pursue opportunities. Centralized way to learn about options.', 'People with expertise in the IT support are too insulated from scientists and faculty doing research. ', "While there exists an infrastructure, there is really no training to back it up. I don't really think I'm using Git or SVN correctly, for example.", 'We have a tremendous shortage of qualified support personnel. We can get moderately competent support to keep a desktop computer operating and can get some help with basic UNIX operating systems, but have no support whatsoever for real scientific computing.', 'It is difficult to hire qualified bioinformaticians.', 'none', 'funding!', 'They do not pay for cloud computing resources', 'lack of an agenda in the mission of the school', 'High level training of tools', 'A standard for reproducibility in the have of various operating systems etc', 'non existant', 'massive gaps - our institution is just simply backwards... ', 'Missing specific software at an affordable price', 'Access to expertise ', 'It is difficult to obtain license for some software (due to vendors and costs)', 'An integrated approach to the development, construction, testing and publication cycle of software development.', 'Software development training for students', 'There is little help in hosting pagesfor use by othes.', 'Support is mostly left to the individual researcher to seek out. Resources exist, but may not be easily deployed.', 'We are a software development group and have been for over 25 years. Our primary target is support for brain imaging and brain research.', 'Software is seen as a necessity only.', 'Research Software Engineers not available.\nResearch Software Engineers doing sys admin work because of understaffing / cost cutting / management not recognizing the importance of Research Software ', 'Resources, in terms of time, personnel, and funding. ', 'We lack expertise, resources and an understanding by administrators of the tasks and the costs.', 'Insufficient support', 'Direct financial support for development to me and to developers in my lab.', 'There is just little money or credit for this from funding agencies until the software has grown to a point where the funding is less critical', 'Easily available and highly-skilled (i.e., advanced training/tutorials) resources of all types. There are lots of resources for the beginner, but not so much for the "practitioner" or "expert-but-not-guru" level. Especially in productivity tools such as git, CI, build systems (cmake, etc), binary packaging/distribution, testing frameworks, etc.', 'Computing resources (access to compute cycles and software development tool). Also, university faculty generally not rewarded career-wise to develop software.', 'Consulting service for usage of software. ', 'Recongnition of scholarly contribution, providing funds for open access publications.', 'Financial gaps', 'software consultants- short classes', "I can't tell since my research areas are not on this topic so it's hard for me to give a good comment. ", 'Customer more focussed on maintenance than research to improve the SW', 'Share software developers, training for software engineering practices for engineers (as opposed to CS majors) ', 'Poor management of high-performance computational center; lack of understanding of advanced software development needs', 'Time constraints', 'The problems I have are: (a) There is a lack of adequate tools such as version control and hosting to handles projects with code + large datasets that need to stay together. Most tools are code oriented. (b) Maintenance of software is a 24x7 job that is very difficult to keep up with while doing the rest of my job. Successful projects are in some ways a curse. (c) Securing long-term funding of software, or developing a business model around it, is difficult—it ends up being a hobby project with significant outside-of-work time spent on it. How to monetize software, especially open-source software, so it is sustainable to develop is unclear to me.', 'training', 'Computing power and data storage capacity ', 'Stable career paths for people who primarily write research-related software.', 'Computational resources are excellent, but the development of research codes could benefit from a dedicated software development group.', "I tend to use old software skills instead of using more powerful new methods because I don't have the time to learn how to use new software.", "i don't develop software, i'm a reasonably skilled user of it", 'They want me to write software as a pleasant but uneccessary side effect of the research rather than as central to the effort. ', 'Computing clusters, refreshes of development machines for faculty, support for development costs inside research groups (*not* in research computing centers)', 'Maintenance', 'High-level leadership and grassroots-level community.', 'Lack of support for ongoing maintenance and development.', "The best software support is available outside of my institution, using publicly available free services such as github and travisci. The most significant gaps are a lack of funding for specialised cloud-based architectures, such as google's tensorflow.", 'staff and budget cuts', 'Lack of understanding of development time/costs and technical debt/sustainability.', 'Not invented here syndrome\nNo institutional support for CI/CT/DevOps\nCloud-Adverse', 'There is none', 'Understaffing', 'staff with knowledge', 'Time to make projects open source\nTime to properly document\nTime to write papers & present them\nTime to communicate about software (meetings, etc)\nTime to study new technologies/education', '$', 'Almost everything.\n\n1. Lack of personnel with computing expertise. Our sys admin handles basic tasks but I have to do a lot of the administration myself.\n\n2. Lack of infrastructure. The university will host my machines but I have to provide the hardware.', 'Time and support', 'Funding and large bureaucracy ', 'Time and money', 'Parallelizing complex codes.', 'Bureaucratic rules at the university can make things harder than necessary (e.g., adding external collaborators, slowness in checking FERPA compliance, issues with licensing, etc.)', 'Not enough experts on the programming language we have been using.', "The administrator's insufficient perception about the software needs and its importance. ", 'support/maintenance of software needs', 'Dedicated highly customizable machines or clusters', "I don't even know enough to be able to answer the question.", "No possibility to get programmers' support for scientific projects unless you employ a programmer", 'inability to recognize software for promotion purposes. Better instruction on software development for researchers.', 'hardware, personnel ', 'There is no support for software related research.', 'Training opportunities', 'Bioinformatics experts to analyze/prepare data for publications', "Infrastructure - adequate storage space and backup procedures, training - is either very intro or very advanced and they don't quite meet our needs", 'linux IT support', 'Limited institutional funding for qualitative analysis software', 'Lack of statistical expertise/ support', 'I recently found out that the group providing support for research software (like REDCap) at our institution had been reduced through attrition to 1 person.', 'Technical skill, people skills, and training.', 'repositories, data storage', 'Organized help with unit testing and documentation is lacking.', 'none', 'Bioinformatics support could be much better', "They don't want us to use outside servers but they also don't want novel software on inside servers. ", 'A convenient cluster computing facility on which I could run batch analysis jobs.', 'There is no direct reward for faculty using time to develop software. It is only indirect via successful publishing, or personal appreciation of other labs using our software.', 'the lack of research software scientists/programmer positions.\nIn the ever increasing involvement of computing for research projects I imagine a research software scientists to be working on several projects, in a shared time fashion.', 'Basic software engineering skills not introduced to new staff.', 'We need better OIT support of HPC', 'Software related research is not part of what the university expects us to do unless it leads to publications. Software development takes a long time and a team, which both are lacking in my institution.', "Biology faculty don't care about software, let alone sustainability because few of them do any software development work or have students that do that type of work.", 'Research software engineering (RSE) is in high demand across all disciplines at our large urban institution. I run an RSE lab funded 80% with external grants and 20% with internal university funds. However, this presents a challenge in terms of stability, retention, and growth to meet demand. More stable sources of support (internal or external) over longer spans of time would greatly improve the ability to retain top talent and provide maintenance for software artifacts that have already been developed.', 'The support may be there, but the university does not do a good job of making information about or access to those services easy to find. There is no financial support for research software development.', 'Minimal computing infrastructure support. I am out of pocket for some services, use free versions for others.', 'Allocating time for training (s/w training rather than the nearly useless compulsory training that is becoming prevalent).', 'people', 'Software development requires time. Organizational pressure is to divide time between too many things which makes focused time very limited. Experience with proposal review processes is particularly bad -- peer reviewers have some insight into community need but know almost nothing about what is required from software development. The software development lifecycle needs to be emphasized more, particularly as it applies to research software.\n\n', 'Nobody values what I do.', 'Maintaining software development continuity during funding or staffing gaps. ', 'Long term positions', 'Lack of software resources, e.g. staff for projects, strategic alignment of a software division with research goals. My institution is stuck in old paradigms.', 'Solid funding for permanent developer positions at the institutional or college level to assist with software development or integration.', 'Long term planning for projects that rely on cloud computing resources/HPC; funding for part time developers', 'Personnel.', 'Recognition of software. Not invented here syndrome.', "Developing software, especially sustainable, open software developed with best practices gets very little professional reward. For example, I'd always be better off (as far as my job marketability) writing crappy software I don't share or document that's just sufficient to get as many papers through a referee as possible.\n\nIn addition, funding agencies/universities care very little about supporting the on-going maintenance and support of a large, open codebase. They'll sometimes spend huge amounts to write that codebase, but then they abandon it.", "Recognition of software contributions is ambiguous/risky since it's largely subject to the whims of people on tenure committees.", 'Credit towards career advancement.', 'Isolated in my research, unable to attract regular collaborators.', 'It is hard to get a grant for writing good software. It is mostly about new features & new prototype algorithms. But without good implementations there is little benefit for the community.', 'No way to recognize/reward accomplishments', "Confusion of the rules about when things can and can't be open source based on our sources of funding.", 'The biggest problem we face is staffing. There are not enough qualified candidates in HPC and Scientific Computing/CSE and the job market is highly illiquid, specialize, and fractured. Furthermore, viable candidates are often recruited by tech companies because the compensation and benefits they offer is much higher. So, related to this is funding. More funding and allowing better compensation of scientists, engineers and software developers in a manner consistent with market rates would greatly improve the number of qualified candidates that can be attracted and hired.', "We don't have an exascale machine yet", 'Software maintenance is not a funded activity.', 'Lack of development tools, such as software licenses; lack of centralized hardware; lack of an active community. (I am working on changing that.)', 'Available hardware (servers) and personnel to maintain; personnel available to train others', 'There is little recognition of software development efforts in promotion & tenure considerations, or time release from teaching, etc. Therefore, there is very little time to work on software skills or improve the software development', '(Not applicable)\n', 'Institution does not have mechanisms to "fill gaps" between external funding. I feel this often causes us to work on the problems most likely to win proposals as opposed to those that are most pressing.\n\nIt\'s often not a realized gap - funding agencies typically are interested in the most pressing problems! - but an important mental block.', 'Recognizing that software development is a valid scientific goal in itself (though this problem is by no means unique to one institution).', 'The lack of guidelines towards publication/open access, or the value of code', 'Tenure and promotion (or lack thereof) for software activities', 'Inability to recruit staff because higher education salaries are nothing compared to industry', 'Career development opportunities. No funding for conferences or workshops.', 'software work is for the most part left to the grad students and post docs to do, and the turnover cycle creates a mess of software usage', 'Website hosting', 'less expertise', 'Lack of experts to provide appropriate training or support for software development and deployment.', 'Hiring good developers is quite difficult in an academic setting. Currently it is mostly done by students and those in transition.', 'Funding opportunities and positions for research software engineers. ', 'isolation, i.e. no critical mass of developers to engender support\nfinancial, adding $$ to a proposal for a dedicated developer is untenable, so I just do it myself\n', 'lack of expertise, lack of long-term support for software, lack of critical-mass of software developers, lack of community', 'Everything', 'communication of capabilities', "I wouldn't know how to answer this question", "Don't know", 'Not particularly valued by community or the funding agencies, in my field, to be frank.', 'its unclear how to get at any support as far as i can tell ', 'satisfactory', 'Site licenses for software packages; university-maintained repository for developed code; opportunities to learn software development tools.', 'Having a pool of capable people who could develop custom software for tasks', 'That is really hard to say -- I consider it my own job to figure out how to do my own research. However, I have a knowledge gap in how to use institutional servers for intensive processing.', "Software is mostly written by the graduate students, which isn't always in their or the project's interest. It would be better if sometimes we could hire programmers who didn't have to worry about publishing/graduating. Some faculty do, but usually only if they have a lot of grant money.", 'There is no on-site help for anything beyond basic IT.', 'I am at a small liberal arts institution where my teaching commitments do not always afford opportunities to work on research during the academic year. Most things about software I learned over 10 years ago and have not moved much beyond that except in my spare time. Others on campus may have experience on which I can draw. But finding those out even at a small place is not always easy.', 'Staff support to mitigate faculty time investment', 'Access to personnel that can help with software admin tasks (running/maintaining project websites, software releases, support for multiple operating systems, etc).', 'No place for software development in the promotion and tenure process. The main developer of NumPy was at my institution and failed to get tenure because his dept did not value that as an academic contribution.', 'Funding', "I'm a software user, not developer. Funds are not available for training.", 'too few staff to help optimizing and parallelizing / porting code', 'it takes a really long time to get institution to take on purchasing AND supporting research-related software.', 'No interest. No foresight.', 'everything has to be paid for on grants.', 'resources to hire expert support personnel when required for large groups/projects', 'Training ', 'lack of infrastructure, lack of qualified personnel to support faculty ', 'money, resources, time', 'Professional development and faculty learning communities lack support. Financial support for students (especially interdisciplinary training).', 'Which programs have licenses from the University - lack of more options appropriate for the hard sciences', "Varied over the years; we used to have good institutional support, but with declining budgets recently, much less of that. Now it's pretty much up to me to find help on the web.", 'Salary for technicians and additional IT staff is not available. ', 'support for software development; servers', 'Software licenses, especially university site licenses can be very useful but are expensive for single PIs to maintain. Institutions should facilitate purchase and maintenance where certain mechanism can be created to partially fund these efforts from PIs either from indirect costs or user based fee. ', 'Lack of appreciation of the need for software resources at the mid/upper levels of management. Often, a Director who is a scientist thinks that they know about software issues because they developed code at some level in college. They have limited knowledge of state-of-the-art development cycles and the challenges of developing and supporting widely used software.', 'Lack of collaboration with other universities on big clusters for computing.', 'Our software program is grant funded, and we have no institutional support for long-term sustainability, which makes most aspects of building and maintaining research software infrastructure much more difficult than it needs to be.', 'GAPS ARE ADDRESSED BY SUBCONTRACTING- NO FUNDS TO HIRE SOFTWARE RELATED SUPPORT', 'The IT department.', 'They don’t help much. ', 'Since my software development is part of my biological research project at a medical school I cannot expect much infrastructural support. System management of research computing systems within laboratories would have been nice but is not provided by the institution. ', "We do not get graduate students who have any kind of software skills or training. There are institutional obstacles to having graduate students take courses that might help. I'm in a chemistry department, and undergrad-level computer science (programming) courses are essentially off-limits to our grad students due to wait lists, and there is pressure from the department to have my students taking more chemistry courses that are not useful to the actual research that my group does.", 'We have to manage most of the software and hardware tools we use for software development. Institution provides basic IT support only.', 'The institutions in the US do not provide any support, whether for research or software. They only collect overheads and pester scientists with irrelevant online courses (ethics, grant administration, managing COI, sexual harrassement, etc). All what we get is faculty salary and space. \n\nThe funding agencies do not provide sufficient support for software. Most grants fund scientific research and only a small part of this support actually contributes to code writing by students and postdocs. THERE IS NO SUPPORT for code maintenance, testing, documentation, etc. The only way to support these parts of software development cycle is via license income, the strategy that many teams follow -- see J. Phys. Chem. Lett. 6, 2751 (2015). However, this survival strategy is now threatened by the mindless pushing of open-source requirements by the funding agencies. \n\nWhen funding agencies do give money for software development, they choose to do so in the least effective for the community way -- funding giant consortia that come with a huge bureaucratic burden and are not at all effective in creative robust and effective software infrastructure. These funding models do not promote the best practices of funding that should be grounded in intellectual merit, individual track record, and documented impact on science. They do not take into account track record of the people. As I observed many times, it is not the best teams with solid track record that get software grants, it is those that are most effective in making grand unsupported claims. There are a number of terribly inefficient projects that sucked in huge amount of money because they are "open-source". NWChem is a primary (but not the only) example of such politically motivated funding. \n\nIf we want to remain competitive in the scientific domains that heavily rely on software and computer modeling, the funding practices should be completely revised. We should go back and fund the project based on their intellectual merit and impact on science and not on whether or not they use the right amount of buzzwords. The only way to insure that decisions are sound and that the money are used effectively is to fund individuals and small teams. \n\n', 'IT generally, we have to manage our own lab computers. No backup services. Very difficult to find information about how to use the super-computer etc', "They don't understand software.", "They aren't up to date at all in the direction that science computation is going.\nThey don't have resources to train people.\nThey don't have the ability to attract and retain people with this type of experience.", "Infrastructure and consulting support seems irrelevant for development (there's github and Travis). Funding my time or other development time would be great.", 'oh god where to even begin', "'-Funds\n-Technical support", "Can't think of any", 'Between 1980 and the present NSF has seeded the development of databases in the labs of working scientists. These scientists (like me) need the tools but are not really trained to run public databases. If external support existed, like GenBank, I would get out of the software business knowing that it was being done for the community as a whole. However, there is no such central resource for Evolutionary Biology and morphology thus scientists have to step in. This is not a good model because (1) there is not long term support, (2) scientists have other obligations and often cannot maintain this and (3) seeding these on a competitive model is not a good strategy; it leads to redundancy.', 'They will not support or maintain a dynamic online database that changes over time as new data is added to it.', 'marketing polish, perhaps \n\nweb hosting (was recently truncated to strictly styled institution-wide CMS)', 'I need to raise my own funds for all hardware and software. ', 'Lack of competent manpower', 'training on development tools', 'Recognition of software development as a valuable research contribution, independent of publications in journals', 'poor administration', "Most of my issues stem from the fact that the IT support at my university doesn't really understand research or what's involved with computing for research, so I'm a bit on my own.", 'It is not clear who is the right agency to provide such support', "Lack of infrastructure report to support my and other colleagues' research, esp. with regard to emerging technologies. Specifically, we lack the ability to perform 1) lab-based pilot tests of function, usability, user-acceptance and integration/implementation factors and 2) field pilot tests for same.", 'Lack of career progression and metrics related to job performance', 'Insufficient expertise mostly devoted to general infrastructure and not available to specific projects.', 'I think the gap is in the type of training. There are very basic opportunities, and then very advanced opportunities that take more time than a research/faculty member can afford. There need to be practical opportunities for people with some background who need to do specific things.', 'Incompetence of central IT; lack of HPC resources; over-agressive security', 'My university provides consulting services and HPC computing resources on a recharge basis at several levels, as does the department. However, few funders are willing to spend resources developing software beyond the level of a research prototype, or to maintain and extend an existing software package. My own software activities are built on "broader impact" sections in proposals, on the activities of my students as they learn, and on my own time spent corralling everything into a working form.\n\nAs a university, we also lack systematic software training even for graduate students who need it. I have tried to run software carpentry-style exercises for students in the computational sciences with some success and a little seed funding from my dean, but I have had difficulty getting either the faculty or the students to commit on a regular basis. The faculty tend to agree that the students need this training, but it competes with a million other things.', " There's a lack of training and understanding of what good software development practices are and their benefits. "]
RQ5: Career Paths¶
What factors impact career advancement and hiring in research software?
The following questions are designed to provide information about career support and career pathways for research software developers.
Q5.1: In your institution, which of the following positions are available for people whose primary role is to develop research software?¶
Multiple select:
Postdoc
Research Software Engineer
Research Programmer
Software Developer
Programmer
Faculty
Research Faculty
Other
In [29]:
print_question("Q5.1", multiselect=True)
IRSEM survey: counts percentages
Q5.1
Postdoc 16.0 0.24
Research Software Engineer 14.0 0.21
Research Faculty 10.0 0.15
Software Developer 9.0 0.14
Research Programmer 6.0 0.09
Programmer 5.0 0.08
Faculty 4.0 0.06
Other 2.0 0.03
TOTAL: 66.0 1.00
- - - - -
URSSI survey: counts percentages
Q5.1
Postdoc 411.0 0.21
Software Developer 253.0 0.13
Programmer 253.0 0.13
Research Programmer 251.0 0.13
Research Faculty 242.0 0.12
Research Software Engineer 223.0 0.11
Faculty 215.0 0.11
Other 121.0 0.06
TOTAL: 1969.0 1.00
Q5.2: Is there opportunity for career advancement (e.g., tenure, more senior positions) for people whose primary role is to develop research software at your institution?¶
Yes - in my current institution
No - I would have to move to another institution\
Don’t know
In [30]:
print_question("Q5.2")
IRSEM survey: counts \
Q5.2
No - I would have to move to another institution 17.0
Yes - in my current institution 8.0
Don’t know 8.0
TOTAL: 33.0
percentages
Q5.2
No - I would have to move to another institution 0.52
Yes - in my current institution 0.24
Don’t know 0.24
TOTAL: 1.00
- - - - -
URSSI survey: counts \
Q5.2
No - I would have to move to another institution 346.0
Don’t know 260.0
Yes - in my current institution 177.0
TOTAL: 783.0
percentages
Q5.2
No - I would have to move to another institution 0.44
Don’t know 0.33
Yes - in my current institution 0.23
TOTAL: 1.00
Q5.3: With regard to hiring and maintaining a highly qualified software development staff, how important are the following concerns? (if Developer or Combination selected)¶
| Not at all important | Slightly important | Moderately important | Very important | Extremely important | |
|---|---|---|---|---|---|
| Identifying a pipeline for future staff | o | o | o | o | o |
| Attracting staff from underrepresented groups (e.g., race, gender, ethnicity, etc.) | o | o | o | o | o |
| Availability of staff who can work across disciplines | o | o | o | o | o |
| Competing with industry for top performers | o | o | o | o | o |
| Offering a viable career path | o | o | o | o | o |
| Opportunities to outsource skilled work | o | o | o | o | o |
In [31]:
print_question_set("Q5.3", ["Identifying a pipeline for future staff",
"Attracting staff from underrepresented groups (e.g., race, gender, ethnicity, etc.)",
"Availability of staff who can work across disciplines",
"Competing with industry for top performers",
"Offering a viable career path",
"Opportunities to outsource skilled work"])
questionID = "Q5.3"
title = 'Importance of hiring concerns. (Career Paths: Question 3)'
STATEMENTS = ["Identify a\nfuture staff pipeline",
"Hire from\nunderrepresented groups",
"Availability of\ncross-discipline staff",
"Industry competition\nfor top performers",
"Offering a\nviable career path",
"Opportunities to\noutsource skilled work"]
ANSWERS = ['Extremely important',
'Very important',
'Moderately important',
'Slightly important',
'Not at all important']
# print raw data
#print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, labelrotation=75)
Identifying a pipeline for future staff:
IRSEM survey: counts percentages
Q5.3_1
Very important 8.0 0.50
Moderately important 4.0 0.25
Slightly important 2.0 0.12
Extremely important 1.0 0.06
Not at all important 1.0 0.06
TOTAL: 16.0 0.99
- - - - -
URSSI survey: counts percentages
Q5.3_1
Very important 235.0 0.36
Moderately important 206.0 0.31
Extremely important 97.0 0.15
Slightly important 72.0 0.11
Not at all important 44.0 0.07
TOTAL: 654.0 1.00
_________________________________________________________
Attracting staff from underrepresented groups (e.g., race, gender, ethnicity, etc.):
IRSEM survey: counts percentages
Q5.3_2
Moderately important 6.0 0.43
Very important 3.0 0.21
Slightly important 3.0 0.21
Not at all important 2.0 0.14
TOTAL: 14.0 0.99
- - - - -
URSSI survey: counts percentages
Q5.3_2
Moderately important 206.0 0.31
Very important 185.0 0.28
Slightly important 96.0 0.15
Not at all important 89.0 0.14
Extremely important 81.0 0.12
TOTAL: 657.0 1.00
_________________________________________________________
Availability of staff who can work across disciplines:
IRSEM survey: counts percentages
Q5.3_3
Slightly important 4.0 0.27
Very important 4.0 0.27
Moderately important 3.0 0.20
Not at all important 2.0 0.13
Extremely important 2.0 0.13
TOTAL: 15.0 1.00
- - - - -
URSSI survey: counts percentages
Q5.3_3
Very important 253.0 0.38
Extremely important 191.0 0.29
Moderately important 161.0 0.24
Slightly important 38.0 0.06
Not at all important 18.0 0.03
TOTAL: 661.0 1.00
_________________________________________________________
Competing with industry for top performers:
IRSEM survey: counts percentages
Q5.3_4
Extremely important 4.0 0.27
Moderately important 4.0 0.27
Not at all important 3.0 0.20
Slightly important 3.0 0.20
Very important 1.0 0.07
TOTAL: 15.0 1.01
- - - - -
URSSI survey: counts percentages
Q5.3_4
Very important 194.0 0.30
Moderately important 181.0 0.28
Extremely important 138.0 0.21
Slightly important 80.0 0.12
Not at all important 61.0 0.09
TOTAL: 654.0 1.00
_________________________________________________________
Offering a viable career path:
IRSEM survey: counts percentages
Q5.3_5
Very important 8.0 0.50
Extremely important 4.0 0.25
Slightly important 3.0 0.19
Moderately important 1.0 0.06
TOTAL: 16.0 1.00
- - - - -
URSSI survey: counts percentages
Q5.3_5
Very important 274.0 0.42
Extremely important 211.0 0.32
Moderately important 116.0 0.18
Slightly important 27.0 0.04
Not at all important 26.0 0.04
TOTAL: 654.0 1.00
_________________________________________________________
Opportunities to outsource skilled work:
IRSEM survey: counts percentages
Q5.3_6
Slightly important 6.0 0.40
Not at all important 5.0 0.33
Very important 3.0 0.20
Moderately important 1.0 0.07
TOTAL: 15.0 1.00
- - - - -
URSSI survey: counts percentages
Q5.3_6
Moderately important 210.0 0.32
Not at all important 166.0 0.26
Slightly important 152.0 0.23
Very important 84.0 0.13
Extremely important 37.0 0.06
TOTAL: 649.0 1.00
Q5.4: In your opinion: How important were the following concerns when you were hired into your current position? (if Developer or Combination selected)¶
| Not at all important | Slightly important | Moderately important | Very important | Extremely important | |
|---|---|---|---|---|---|
| Diversity in the organization (e.g., in terms of race, gender, ethnicity) | o | o | o | o | o |
| Your experience as programmer or software engineer | o | o | o | o | o |
| Your background in science | o | o | o | o | o |
| Your knowledge of diverse programming languages | o | o | o | o | o |
| Your knowledge and capabilities across disciplines | o | o | o | o | o |
| Your potential for growth | o | o | o | o | o |
In [32]:
print_question_set("Q5.4", ["Diversity in the organization (e.g., in terms of race, gender, ethnicity)",
"Your experience as programmer or software engineer",
"Your background in science",
"Your knowledge of diverse programming languages",
"Your knowledge and capabilities across disciplines",
"Your potential for growth"])
questionID = "Q5.4"
title = 'Hiring concerns about the respondent\'s position. (Career Paths: Question 4)'
STATEMENTS = ["Diversity in\nthe organization",
"Programmer / software\nengineer experience",
"Science background",
"Diverse programming\nlanguage knowledge",
"Cross-discipline\nknowledge and capability",
"Growth potential"]
ANSWERS = ['Extremely important',
'Very important',
'Moderately important',
'Slightly important',
'Not at all important']
# print raw data
#print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, labelrotation=75)
Diversity in the organization (e.g., in terms of race, gender, ethnicity):
IRSEM survey: counts percentages
Q5.4_1
Not at all important 7.0 0.44
Slightly important 5.0 0.31
Moderately important 3.0 0.19
Very important 1.0 0.06
TOTAL: 16.0 1.00
- - - - -
URSSI survey: counts percentages
Q5.4_1
Not at all important 160.0 0.40
Moderately important 108.0 0.27
Slightly important 78.0 0.19
Very important 41.0 0.10
Extremely important 16.0 0.04
TOTAL: 403.0 1.00
_________________________________________________________
Your experience as programmer or software engineer:
IRSEM survey: counts percentages
Q5.4_2
Extremely important 4.0 0.25
Not at all important 4.0 0.25
Very important 3.0 0.19
Slightly important 3.0 0.19
Moderately important 2.0 0.12
TOTAL: 16.0 1.00
- - - - -
URSSI survey: counts percentages
Q5.4_2
Not at all important 110.0 0.27
Moderately important 82.0 0.20
Slightly important 81.0 0.20
Very important 79.0 0.19
Extremely important 54.0 0.13
TOTAL: 406.0 0.99
_________________________________________________________
Your background in science:
IRSEM survey: counts percentages
Q5.4_3
Extremely important 8.0 0.50
Very important 6.0 0.38
Slightly important 2.0 0.12
TOTAL: 16.0 1.00
- - - - -
URSSI survey: counts percentages
Q5.4_3
Extremely important 223.0 0.55
Very important 112.0 0.28
Moderately important 49.0 0.12
Slightly important 12.0 0.03
Not at all important 10.0 0.02
TOTAL: 406.0 1.00
_________________________________________________________
Your knowledge of diverse programming languages:
IRSEM survey: counts percentages
Q5.4_4
Slightly important 7.0 0.44
Not at all important 5.0 0.31
Extremely important 2.0 0.12
Very important 1.0 0.06
Moderately important 1.0 0.06
TOTAL: 16.0 0.99
- - - - -
URSSI survey: counts percentages
Q5.4_4
Not at all important 136.0 0.34
Slightly important 108.0 0.27
Moderately important 100.0 0.25
Very important 44.0 0.11
Extremely important 17.0 0.04
TOTAL: 405.0 1.01
_________________________________________________________
Your knowledge and capabilities across disciplines:
IRSEM survey: counts percentages
Q5.4_5
Very important 6.0 0.38
Moderately important 4.0 0.25
Slightly important 4.0 0.25
Extremely important 2.0 0.12
TOTAL: 16.0 1.00
- - - - -
URSSI survey: counts percentages
Q5.4_5
Very important 131.0 0.32
Moderately important 104.0 0.26
Extremely important 94.0 0.23
Slightly important 49.0 0.12
Not at all important 27.0 0.07
TOTAL: 405.0 1.00
_________________________________________________________
Your potential for growth:
IRSEM survey: counts percentages
Q5.4_6
Extremely important 7.0 0.44
Very important 5.0 0.31
Moderately important 4.0 0.25
TOTAL: 16.0 1.00
- - - - -
URSSI survey: counts percentages
Q5.4_6
Extremely important 175.0 0.43
Very important 136.0 0.34
Moderately important 62.0 0.15
Slightly important 17.0 0.04
Not at all important 15.0 0.04
TOTAL: 405.0 1.00
RQ6: Credit for Software Contributions¶
RQ6a: What do research software projects require for crediting or attributing software use?
RQ6b: How are individuals and groups given institutional credit for developing research software?
This section elicits information about citation and attribution of research software.
Q6.1: When you write a paper and the work being described uses software, how often do you use each of the following approaches to mention the software?¶
| Never | Sometimes | About half the time | Most of the time | Always | |
|---|---|---|---|---|---|
| Cite paper about the software | o | o | o | o | o |
| Cite the software user’s manual | o | o | o | o | o |
| Mention the name of the software | o | o | o | o | o |
| Mention the URL of the software | o | o | o | o | o |
| Cite the URL of the software | o | o | o | o | o |
| Cite the published / archived software itself | o | o | o | o | o |
| Other | o | o | o | o | o |
In [33]:
print_question_set("Q6.1", ["Cite paper about the software",
"Cite the software user’s manual",
"Mention the name of the software",
"Mention the URL of the software",
"Cite the URL of the software",
"Cite the published / archived software itself",
"Other"])
questionID = "Q6.1"
title = 'Software credit methods. (Credit for Software Contributions: Question 1)'
STATEMENTS = ["Cite paper about\nthe software",
"Cite the software\nuser’s manual",
"Mention the name of\nthe software",
"Mention the URL of\nthe software",
"Cite the URL of\nthe software",
"Cite the published /\narchived software itself",
"Other"]
ANSWERS = ['Always',
'Most of the time',
'About half the time',
'Sometimes',
'Never']
# print raw data
#print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, labelrotation=75)
Cite paper about the software:
IRSEM survey: counts percentages
Q6.1_1
Always 15.0 0.48
About half the time 6.0 0.19
Most of the time 6.0 0.19
Never 2.0 0.06
Sometimes 2.0 0.06
TOTAL: 31.0 0.98
- - - - -
URSSI survey: counts percentages
Q6.1_1
Always 294.0 0.39
Most of the time 198.0 0.27
Sometimes 122.0 0.16
About half the time 77.0 0.10
Never 55.0 0.07
TOTAL: 746.0 0.99
_________________________________________________________
Cite the software user’s manual:
IRSEM survey: counts percentages
Q6.1_2
Sometimes 13.0 0.45
Never 10.0 0.34
Always 3.0 0.10
About half the time 2.0 0.07
Most of the time 1.0 0.03
TOTAL: 29.0 0.99
- - - - -
URSSI survey: counts percentages
Q6.1_2
Never 293.0 0.40
Sometimes 248.0 0.34
Most of the time 70.0 0.10
About half the time 63.0 0.09
Always 51.0 0.07
TOTAL: 725.0 1.00
_________________________________________________________
Mention the name of the software:
IRSEM survey: counts percentages
Q6.1_3
Always 21.0 0.70
Most of the time 6.0 0.20
Never 1.0 0.03
Sometimes 1.0 0.03
About half the time 1.0 0.03
TOTAL: 30.0 0.99
- - - - -
URSSI survey: counts percentages
Q6.1_3
Always 476.0 0.64
Most of the time 149.0 0.20
Sometimes 57.0 0.08
About half the time 37.0 0.05
Never 24.0 0.03
TOTAL: 743.0 1.00
_________________________________________________________
Mention the URL of the software:
IRSEM survey: counts percentages
Q6.1_4
Most of the time 11.0 0.37
Always 10.0 0.33
Sometimes 5.0 0.17
Never 2.0 0.07
About half the time 2.0 0.07
TOTAL: 30.0 1.01
- - - - -
URSSI survey: counts percentages
Q6.1_4
Sometimes 217.0 0.30
Most of the time 157.0 0.22
Never 146.0 0.20
Always 140.0 0.19
About half the time 66.0 0.09
TOTAL: 726.0 1.00
_________________________________________________________
Cite the URL of the software:
IRSEM survey: counts percentages
Q6.1_5
Always 13.0 0.41
Most of the time 9.0 0.28
Sometimes 6.0 0.19
Never 2.0 0.06
About half the time 2.0 0.06
TOTAL: 32.0 1.00
- - - - -
URSSI survey: counts percentages
Q6.1_5
Sometimes 228.0 0.31
Most of the time 158.0 0.22
Always 137.0 0.19
Never 136.0 0.19
About half the time 70.0 0.10
TOTAL: 729.0 1.01
_________________________________________________________
Cite the published / archived software itself:
IRSEM survey: counts percentages
Q6.1_6
Always 10.0 0.32
Sometimes 7.0 0.23
About half the time 5.0 0.16
Most of the time 5.0 0.16
Never 4.0 0.13
TOTAL: 31.0 1.00
- - - - -
URSSI survey: counts percentages
Q6.1_6
Sometimes 240.0 0.33
Never 151.0 0.21
Most of the time 136.0 0.19
Always 136.0 0.19
About half the time 66.0 0.09
TOTAL: 729.0 1.01
_________________________________________________________
Other:
IRSEM survey: counts percentages
Q6.1_7
Never 11.0 0.65
Sometimes 3.0 0.18
About half the time 2.0 0.12
Always 1.0 0.06
TOTAL: 17.0 1.01
- - - - -
URSSI survey: counts percentages
Q6.1_7
Never 11.0 0.44
Always 8.0 0.32
Most of the time 3.0 0.12
Sometimes 2.0 0.08
About half the time 1.0 0.04
TOTAL: 25.0 1.00
Q6.2: How often do you currently receive the following types of credit for your software contributions? (if Developer or Combination selected)¶
| Never | Sometimes | About half the time | Most of the time | Always | |
|---|---|---|---|---|---|
| (Co)author on research paper | o | o | o | o | o |
| (Co)author on software paper | o | o | o | o | o |
| Acknowledgement in paper | o | o | o | o | o |
| Software cited in a paper | o | o | o | o | o |
| Funded/hired to work on the software | o | o | o | o | o |
| Other | o | o | o | o | o |
In [34]:
print_question_set("Q6.2", ["(Co)author on research paper",
"(Co)author on software paper",
"Acknowledgement in paper",
"Software cited in a paper",
"Funded/hired to work on the software",
"Other"])
questionID = "Q6.2"
title = 'Methods of receiving credit. (Credit for Software Contributions: Question 2)'
STATEMENTS = ["(Co)author\non research paper",
"(Co)author\non software paper",
"Acknowledgement\nin paper",
"Software cited\nin a paper",
"Funded/hired to work \non the software",
"Other"]
ANSWERS = ['Always',
'Most of the time',
'About half the time',
'Sometimes',
'Never']
# print raw data
#print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, labelrotation=75)
(Co)author on research paper:
IRSEM survey: counts percentages
Q6.2_1
Sometimes 8.0 0.47
Never 4.0 0.24
Most of the time 3.0 0.18
Always 2.0 0.12
TOTAL: 17.0 1.01
- - - - -
URSSI survey: counts percentages
Q6.2_1
Sometimes 133.0 0.33
Most of the time 89.0 0.22
Never 76.0 0.19
Always 57.0 0.14
About half the time 47.0 0.12
TOTAL: 402.0 1.00
_________________________________________________________
(Co)author on software paper:
IRSEM survey: counts percentages
Q6.2_2
Never 9.0 0.56
Sometimes 3.0 0.19
About half the time 2.0 0.12
Always 2.0 0.12
TOTAL: 16.0 0.99
- - - - -
URSSI survey: counts percentages
Q6.2_2
Never 134.0 0.34
Sometimes 96.0 0.24
Most of the time 72.0 0.18
Always 63.0 0.16
About half the time 31.0 0.08
TOTAL: 396.0 1.00
_________________________________________________________
Acknowledgement in paper:
IRSEM survey: counts percentages
Q6.2_3
Sometimes 10.0 0.59
Never 2.0 0.12
Most of the time 2.0 0.12
About half the time 2.0 0.12
Always 1.0 0.06
TOTAL: 17.0 1.01
- - - - -
URSSI survey: counts percentages
Q6.2_3
Sometimes 167.0 0.42
Never 119.0 0.30
About half the time 54.0 0.14
Most of the time 38.0 0.10
Always 21.0 0.05
TOTAL: 399.0 1.01
_________________________________________________________
Software cited in a paper:
IRSEM survey: counts percentages
Q6.2_4
Sometimes 6.0 0.38
About half the time 5.0 0.31
Most of the time 2.0 0.12
Never 2.0 0.12
Always 1.0 0.06
TOTAL: 16.0 0.99
- - - - -
URSSI survey: counts percentages
Q6.2_4
Sometimes 134.0 0.34
Never 89.0 0.23
Most of the time 88.0 0.22
About half the time 48.0 0.12
Always 36.0 0.09
TOTAL: 395.0 1.00
_________________________________________________________
Funded/hired to work on the software:
IRSEM survey: counts percentages
Q6.2_5
Never 9.0 0.56
Always 2.0 0.12
Sometimes 2.0 0.12
About half the time 2.0 0.12
Most of the time 1.0 0.06
TOTAL: 16.0 0.98
- - - - -
URSSI survey: counts percentages
Q6.2_5
Never 213.0 0.54
Sometimes 101.0 0.26
Most of the time 31.0 0.08
About half the time 28.0 0.07
Always 18.0 0.05
TOTAL: 391.0 1.00
_________________________________________________________
Other:
IRSEM survey: counts percentages
Q6.2_6
Never 5.0 0.71
Always 1.0 0.14
Sometimes 1.0 0.14
TOTAL: 7.0 0.99
- - - - -
URSSI survey: counts percentages
Q6.2_6
Never 16.0 0.76
About half the time 2.0 0.10
Always 2.0 0.10
Sometimes 1.0 0.05
TOTAL: 21.0 1.01
Q6.3: For your job role, does your institution allow software contributions to be considered in performance reviews or promotion cases?¶
Yes
No
Dependent on the position (programmer vs. faculty)
In [35]:
print_question("Q6.3")
IRSEM survey: counts \
Q6.3
Dependent on the position (programmer vs. faculty) 12.0
Yes 10.0
No 9.0
TOTAL: 31.0
percentages
Q6.3
Dependent on the position (programmer vs. faculty) 0.39
Yes 0.32
No 0.29
TOTAL: 1.00
- - - - -
URSSI survey: counts \
Q6.3
Yes 290.0
Dependent on the position (programmer vs. faculty) 289.0
No 165.0
TOTAL: 744.0
percentages
Q6.3
Yes 0.39
Dependent on the position (programmer vs. faculty) 0.39
No 0.22
TOTAL: 1.00
Q6.4: How important do you believe software contributions are for your own performance review or promotion case?¶
Not at all important
Slightly important
Moderately important
Very important
Extremely important
In [36]:
print_question("Q6.4")
IRSEM survey: counts percentages
Q6.4
Slightly important 12.0 0.39
Moderately important 11.0 0.35
Very important 4.0 0.13
Extremely important 2.0 0.06
Not at all important 2.0 0.06
TOTAL: 31.0 0.99
- - - - -
URSSI survey: counts percentages
Q6.4
Not at all important 265.0 0.35
Slightly important 197.0 0.26
Moderately important 167.0 0.22
Very important 80.0 0.11
Extremely important 48.0 0.06
TOTAL: 757.0 1.00
RQ7: Diversity and Inclusion¶
How do current Research Software Projects document diversity statements and what support is needed to further diversity initiatives?
Please answer the following questions based on the software project you work on most. If the questions do not apply, please select N/A.
Q7.1: How well does your project do the following:¶
| Terrible | Poor | Average | Good | Excellent | 🚫N/A | |
|---|---|---|---|---|---|---|
| Recruit participants from underrepresented groups | o | o | o | o | o | o |
| Retain participants from underrepresented groups | o | o | o | o | o | o |
| Include participants from underrepresented groups in governance and leadership positions | o | o | o | o | o | o |
| Maintain a culture of inclusion | o | o | o | o | o | o |
In [37]:
print_question_set("Q7.1", ["Recruit participants from underrepresented groups",
"Retain participants from underrepresented groups",
"Include participants from underrepresented groups in governance and leadership positions",
"Maintain a culture of inclusion"])
questionID = "Q7.1"
title = 'How projects recruit/retain/include underrepresented groups and maintain inclusion. (Diversity and Inclusion: Question 1)'
STATEMENTS = ["Recruit participants from\nunderrepresented groups",
"Retain participants from\nunderrepresented groups",
"Include participants from\nunderrepresented groups\nin governance\nand leadership positions",
"Maintain a culture \nof inclusion"]
ANSWERS = ['Excellent',
'Good',
'Average',
'Poor',
'Terrible']
# print raw data
#print_question_set(question=questionID, statements=STATEMENTS)
graph_likert_matrix(questionID=questionID, title=title, statements=STATEMENTS, answers=ANSWERS, labelrotation=75)
Recruit participants from underrepresented groups:
IRSEM survey: counts percentages
Q7.1_1
Average 11.0 0.50
Poor 9.0 0.41
Terrible 1.0 0.05
Good 1.0 0.05
TOTAL: 22.0 1.01
- - - - -
URSSI survey: counts percentages
Q7.1_1
Average 193.0 0.34
Good 152.0 0.27
Poor 103.0 0.18
Excellent 91.0 0.16
Terrible 24.0 0.04
TOTAL: 563.0 0.99
_________________________________________________________
Retain participants from underrepresented groups:
IRSEM survey: counts percentages
Q7.1_2
Average 10.0 0.48
Poor 5.0 0.24
Good 4.0 0.19
Excellent 1.0 0.05
Terrible 1.0 0.05
TOTAL: 21.0 1.01
- - - - -
URSSI survey: counts percentages
Q7.1_2
Average 194.0 0.36
Good 173.0 0.32
Excellent 88.0 0.16
Poor 66.0 0.12
Terrible 20.0 0.04
TOTAL: 541.0 1.00
_________________________________________________________
Include participants from underrepresented groups in governance and leadership positions:
IRSEM survey: counts percentages
Q7.1_3
Poor 9.0 0.43
Average 7.0 0.33
Good 3.0 0.14
Terrible 1.0 0.05
Excellent 1.0 0.05
TOTAL: 21.0 1.00
- - - - -
URSSI survey: counts percentages
Q7.1_3
Average 159.0 0.32
Good 137.0 0.28
Poor 107.0 0.22
Excellent 54.0 0.11
Terrible 37.0 0.07
TOTAL: 494.0 1.00
_________________________________________________________
Maintain a culture of inclusion:
IRSEM survey: counts percentages
Q7.1_4
Average 10.0 0.45
Good 5.0 0.23
Poor 3.0 0.14
Excellent 3.0 0.14
Terrible 1.0 0.05
TOTAL: 22.0 1.01
- - - - -
URSSI survey: counts percentages
Q7.1_4
Good 216.0 0.38
Excellent 175.0 0.31
Average 149.0 0.26
Poor 28.0 0.05
Terrible 5.0 0.01
TOTAL: 573.0 1.01
Q7.2: What are the challenges you face in creating / maintaining a culture of inclusion?¶
Text input
In [38]:
print(IRSEM["Q7.2"][2:].dropna().to_list())
['Few qualified applicants from underrepresented groups', 'This is not important if your focus is on creating good software', 'Very few mathematicians in underrepresented groups', 'N/A. I have no idea what race, age, gender, etc most of my collaborators are. Online cultures are inclusive by default.', "When bringing new people on, there is quite a bit of work needed to get people up to speed. So, one's background an expertise play a big role in the ease of onboarding.", 'Inclusion is not discussed that much in my country', 'Finding qualified applicants.', 'I think one individual can often be extremely harmful to said culture, so it is easy to have a "one bad apple" problem', 'the challenges mainly boil down to a few opinionated individuals', 'racial bias for interest in programming', 'recruitment']
In [39]:
print(URSSI["Q7.2"][2:].dropna().to_list())
['Hidden biases', 'Without more candidates and applicants, it will be difficult. I think there is a good environment when we can get applicants who are good, but the candidate pool is insufficiently diverse.', 'Well, as a community project 1/2 the time I know people only by their GutHub handle :)', 'Potentials employees self-select themselves during the application process, since the number of applicants is always skewed toward male applications. To counteract this tendency, I always select the best male and the best female applicants. ', 'Talent pool. Applicant pool is typically not very diverse. \nLikely due to salary / compensation being low because academia.', 'Lack of direction and leadership.', 'Lack of a sufficient pool of qualified individuals who display the skills needed for the task...of course this is true even outside of diversity/inclusion question.', 'Having a bus factor superior to 1', 'Typical problems where people are entrenched in leadership positions (older white males), and they just stay in those positions forever. Then they promote similar people to take over.', 'Foreign national hiring can be difficult at a NNSA lab.', 'Speed of recruitment - good diverse candidates are hired elsewhere before we can even offer an interview', 'Both Our IT and Research labs have considerable staff turnover.', 'I am not Asian, but there is discrimination against Asian programmers and older applicants. ', 'Available talent. Our general pay scale compared to cost of living. ', 'We have put a great deal of effort into making our project a welcoming community, and all of the developers have internalized this mindset. However one challenge we face regularly is that occasional users, who are established scientists in their domain disciplines do not possess the same mindset and respond to questions from new users in a technically correct, but rather condescending way. While we try to compensate for this ourselves this could make our project appear less welcoming that it actually is (as these users are only a minority of the whole community).', 'Attracting diverse contributors in the first place is very difficult.', 'Finding skilled personnel is tough. Adding the diversity aspect on top of this is challenging if non existent. ', 'keeping diverse and experienced staff alike- temptations of industry and urban life can woo developers away from our regionally isolated small city academic org. ', 'Many of the participants come from financially disadvantaged background and are difficult to keep in school. However, I have many years of experience in inclusion in computing, helping me to handle many of these challenges.', 'For retention/recruitment : Input pipeline is small. My department is actually pretty good with emphasizing the importations of inclusion/diversity. ', 'Having a diverse pool of highly qualified applicants', 'Attracting grants to create more positions.', 'My current software project is just me, so diversity and inclusion are not relevant.', 'lack of people', 'Obtaining the best performing potential faculty member, i.e. sacrificing the meritocracy for a social issue.', 'The biggest challenge is attracting a diverse set of applicants for a position.', 'Lack of applicants from under-represented groups', "None. We are inclusive. It's just not a challenge.", 'Text based communication (e.g. code review comments) lack tone of voice, so some good intentioned brief comments can seem unfriendly.', 'Incumbent leaders who promote only those who preserve the status quo.', 'No major challenges.', 'Why would I care? I only hire people based on their abilities.', 'history ....', 'We have very few contributors', 'Nothing,', 'Management hubris\nLanguage and cultural barriers', 'I am fortunate enough to be at a very diverse institution -- recruiting and retaining diverse people and including people is pretty ingrained here.', "I'm only a graduate student so I don't really have control over hiring choices or which students to accept into our program. ", 'Denial that a problem exists ', 'Finding people and convincing them to apply.', "We have women, LGBT but no minorities to speak of. No one in the pipeline to recruit, plus we don't have competitive pay with industry so can't compete. ", 'Few faculty from under-represented groups. ', 'None. Two (out of three) of my programmers are women and I have a female graduate student. Never had any problems with this', "It's easy to attract women, but it's hard to attract other under represented groups. Therefore, it's hard to create any kind of true sense of inclusion.", "It's difficult to attract a diverse workforce to our campus other than students from foreign countries. The issue is attracting a diverse workforce that are U.S. citizens due to the homogeneous population in this area.", 'University salaries are low compared to industry for anything other than administrators', 'I do not know what that his.', 'Senior leadership seems uninterested in creating consequences for not maintaining an inclusive work environment.', 'The biggest problem seems to be the "pipeline", i.e., the lack of a diverse pool of applicants in the first place.', 'My group consists of graduate students and post-docs.\nThe make-up of the group depends on who is interested in joining the group and the skills they bring to the group.\nThe focus is science. The programming is to get the science done.', 'supply of qualified personnel from underrepresented groups at the graduate student level and above - the problem starts much earlier...', 'Getting students to be excited about their involvement.', 'A lack of diversity in my institution and region.', 'Lack of available qualified candidates', 'External individuals are often not as calibrated about how to maintain cultures of inclusion, often saying inappropriate comments.', 'I will take anyone willing to work on the project. Pickings of incoming physics graduate students interested in developing software is pretty slim.', 'No challenge', 'Very few people from underrepresented groups in my home state. I have to recruit from other states.', 'Small pool from which to recruit diverse candidates.', 'Lack of awareness of issues - we have struggled to attract a more diverse team.', 'The main challenge is finding qualified people among underrepresented groups.', 'Mostly, a lack of researchers from underrepresented groups interested in these aspects.', 'I would NEVER exclude anyone who is capable!', 'There are no female applicants for experienced positions in this field, and only a handful senior women in the field as a whole.', 'Sufficient diversity of applicants, especially domestic ones.', 'Competition for diversity candidates - they are rare and everyone wants them.', 'hierarchy among PIs and academic units with different priorities', 'location', 'Attracting diverse participants with common research interests to mine', 'Prevailing external culture is not one of inclusion.\nSurvival in that culture competes with internal culture of inclusion.', "There's always one person who can be a bit intimidating.", 'Background and country of origin matters sometimes more than mere color of skin. When peers are not helpful in creating that culture of inclusion, PIs should set the tone and they often fail to. Lastly, Diversity departments sometimes focus on the wrong things like skin color rather than background and history and they miss to represent the people not falling into specific categories. ', "It is hard to get female students excited about software development in their research careers. That's fine if they truly choose so, but deprives them of the opportunity to be personally associated with the successes that the software achieves. There's a risk they are externally perceived as pure users, not developers.", 'none', "The size of the working group is small and there is turnover due to involvement of graduate students and post-docs. I don't have a say in who joins the project, so personalities/prejudices of incoming folks can be problematic", "Our university is very good at it! I don't think we have a hard time, but I'm not in software development.", 'i do not know what that is.', 'Lack of current underrepresented groups', 'We do not attract the applicants to the job postings.', 'subcritical mass of developers', 'Lack of qualified applicants', 'I actually have very little opportunity to participate in software development or resources to include diverse trainees or staff.', 'Actually hiring people is the first step toward inclusion. ', 'the biggest issue is connecting with underrepresented groups, once they are in they usually stay', 'Dedication ', 'There are very few applicants from underrepresented group for software positions. ', 'Being culturally effective requires different things for different people/groups/situations', "Attracting underrepresented groups to attend our university due to it's location (cold Northeast in a small town geography).", "Participants don't always want to divulge personal information.", 'New students put on the project are overwhelmed.', 'My institute within the university does not have a lot of racial diversity to begin with. Part of this is because it is small, and part is because we only are able to hire US citizens (limited racial diversity exists across the entire field, when considering only US citizens).', 'The biology - computer divide is challenging in that it results in the nature barrier between team members. This barrier leads to miscommunication and acrimony. ', 'New Hampshire', 'Lack of people will/able to work on the project', 'The main people I work on the software project with are undergraduate chemistry students. At my very small institution, there are an average of about 5-10 chemistry majors who graduate per year, so the situation varies wildly each year with different students. I try to make the work as inviting and welcoming as possible, but every year is different.', 'Finding the appropriate personnel', 'This issue is not at all important to our software development. I have never encountered a piece of software in which I could discern the race of its author. ', 'Not enough candidates that apply for positions available.', 'Identifying candidates in a highly competitive area (many jobs few candidates...and small business tends to pay less).', 'Nothing to do with software, but it is always a challenge to recruit underrepresented groups to biological sciences.', 'I would like to hire from under-represented groups but I do not know how to find such talent.', 'Largely single-developer, not applicable', 'communication is lacking amongst different groups, perhaps due to cultural, ethic, and other barriers both external and internal. Diversity has become more of a divisive issue than a uniting factor.', 'The same problems facing the rest of computer science- outdated and harmful patterns in our field make it less than welcoming to many people from under-indexed backgrounds. We have put a lot of effort on our team into recruiting a diverse staff and building an inclusive team, but it needs constant and intentional effort to keep it that way, as we face a bit of a headwind. \n\n', 'availability of qualified candidates, ', 'small enough organization that career progression within the organization is limited.', 'not many applicants with required knowledge , training and interest.', 'Inclusivity - making sure the underrepresented individuals feel like they are full-fledged members of the project/group/community, valued and respected. And overburdening these individuals with enriching but potentially distracting opportunities (e.g. participation in outreach activities).', 'Student collaborators offer the most opportunity, but diversity seems to narrow as they progress, so that there is less diversity in the pool of post-graduate developers.', 'Making everyone feel comfortable contributing and asking questions. Lots of users or new contributors are worried about asking “dumb” questions. I think our community is friendly but people have a default expectation that they are opening themselves up to ridicule. So we need to somehow make it more clear that that isn’t the case.', 'We include a culture of inclusion to the extent that the workforce is available in our geographic area.', 'Lack of buy in and communication', 'Identifying candidates from underrepresented groups for coding jobs. ', 'Lack of a talent pool', '1. In a majority Hispanic/minority community and institution - being able to maintain balance that includes and engages "all groups" without over-focusing on the "easily engaged" majority/minority constituency.\n\n\n2. Training, educating, and providing professional experience to the often economically disadvantaged members of many underrepresented groups, such that they are more likely to be included at higher levels of engagement.\n\n\n3. Locally retaining people from #2 above, so that we can broaden and self-sustain that culture here at home.', 'So little diversity that finding an inclusive community is difficult for folks who move in. ', 'Hard to find enough candidates', "Challenges are largely in terms of workplace expectations (e.g., timeliness) which some members don't realize is expectation. Inclusion challenges are consistently using desired pronouns. ", 'Pipeline', "I try to keep the lab as open a place as possible in every sense, and that's the established culture. Minor administrative issues in having students from different backgrounds and departments working together.", "I am a research technician, so I can only affect the lab culture so much. But here is what I do:\n\nWhen interviewing potential recruits, it is possible that unaddressed bias might lead to an ethnically or sexually uniform monoculture. Therefore, we take care to involve lab members of minority ethnicity in interviews and also insure that the male:female ratio of interview teams is balanced.\n\nAdditionally, the same interview questions are asked of all candidates (in the general sense, we of course may follow up on a given candidate's experience with specific questions) and the interview process is transparent to the lab primary investigators (and to University investigators as well, though no one has ever done an audit of our interview practices to my knowledge).\n\nPrevailing cultural trends can sometimes lead to careless behavior which causes discord along known fault lines (on the basis or race or ethnicity, on the basis of sex or gender, on the basis of creed or religion, etc). In the lab, we try to create a welcoming and professional workplace which recognizes that our field is staffed by workers from all walks of life, including many ethnic backgrounds, different sexual orientations, different ages, and so on. As such, conversation should be respectful, open-minded, and conducted with the realization that we are all trying to understand the underlying biology.", 'Finding qualified participants from underrepresented groups.', 'I write software primarily for myself', "Assistance is welcome independent of source. Sometimes helpers have resisted advice, but it's hard to assign this to culture or diversity.", "I'm a solo developer", 'There is very little diversity in the the pool of potential collaborators.', 'Academia is by nature inclusive. I have had students from all kinds of backgrounds - and this was never a question. The most important thing is that the student will need to have the proper skills, and these skills are not tied to race or ethnicity.', 'Communication, dispute resolution', "It's hard to evaluate the background of applicants from developing nations.", 'Limited pipeline of diverse participants.', 'The demography in the field is quite skewed. We have good representation of women and LGBTQ folks, OK representation of hispanic/latino and disadvantaged Asian groups, but quite poor representation of African American and Native Americans. I think we could provide a good research home for members of these less represented groups, but by the time you get to research the demographics are already skewed.', 'I do not work on software projects', 'recruitment', 'Not many at this time', 'Finding students from underrepresented grops.', 'Social constructs of other programmers.', 'Lack of pipeline of skilled staff with diverse backgrounds.', 'Finding candidates.', 'Finding candidates with strong skills and training.', ' Salaries at my college are well below average, but the cost of living at my college is well above average .', 'Relative inexperience of program participants.', 'Academia is not in good shape overall when it comes to inclusion whatsoever. We need to work a lot harder on this. ', 'available candidates', 'Very little in the region', 'Poor/absent leadership\n', 'It can be challenging to find credentialed and qualified candidates from groups who are underrepresented in CS/Engineering fields.', 'Small teams.', 'Finding members of underrepresented groups with strong quantitative skills and interest in furthering these skills ', 'Identifying qualified candidates for the team.', 'Our team is interdisciplinary, diverse and inclusive, but has not yet been successful at obtaining external funding as leaders. We have been able to obtain funding only as support for less diverse, less inclusive teams. Lack of funding for the diverse group is a huge barrier to maintaining membership and research output .', 'None, we avoid hiring bigots', 'Finding qualified individuals', 'Funding', 'Figuring out an effective way for communication between participants.', 'Adequate training for all groups', 'Stereotypes about programming.', 'Once someone is hired, things go well. The challenge is finding people to hire.', 'Shortage of qualified URM professionals', "Behavior of some senior PIs in the collaboration (it's a large, multi-institution collaboration). Some PIs in the project are very good at creating and maintaining a culture of inclusion, but a few are not and can have a large impact.", 'The smartest most capable people are often the quietest or most shy, and that makes making an inclusive group atmosphere and sharing information difficult.', "I work at a large midwest university, but it's not as diverse i wish it were. i have to recruit from a nearby city that has a bigger minority population", "I don't have many other people working with me", 'In general the biggest challenge is we have a low diversity demographic in the state traditionally, so our diversity at the faculty and staff levels are low compared to the students.', "Old white guys existing in positions of leadership, unfamiliar with navigating a system that wasn't built for them.", 'My project is too small for this to be meaningful.', 'Insufficient understanding at executive level. Excellent support and understanding from immediate management.', 'Few minority candidates. ', 'Convincing the "old boys" to share information.', 'Software work is done pro bono or as necessary for getting research finished.', 'Longtime (white, male) employees with more power than I have, kind of a boys club', "I'm at an MSI and the culture of inclusion is inherent in the institution", 'HIerarchical nature of the company; macho/male-dominated culture in software development/IT', 'Finding suitable applicants with appropriate training.', 'Finding people with both the interest and the skills required is always difficult.', "Scale - projects just aren't big enough to have large and diverse teams", 'None. ', "I can't get anyone to even work on it. Period.", 'That we cannot include everyone', ' - clashing background cultures (American vs European). I don\'t think this is a fundamental issue, but it can make it hard to talk about somewhat difficult topics only via text given the diversity of backgrounds, non-native english, etc. One example I dealt with was a German contributor who many people thought was be very hostile, talking to the contributor they thought they were being extremely polite and direct (this particular situation was improved via a private conversation). \n - balancing suggesting small changes to behavior (many things that act against inclusion are unintentional or arise due to miss-understanding of pure-text communications) without becoming "personality police". ', 'Availability of qualified people from underrepresented groups', 'Weak pipeline of diversity candidates', 'Access to underrepresented groups', 'Not enough underrepresented minorities qualified and interested in these positions.', 'availability of interested students/postdocs', 'Please note that I am referring to projects performed by teams of undergraduate research students and me. I have little control over the students who inquire about working in my lab, and I hesitate to recruit because I usually have more lab students than I can effectively mentor. ', "Older generations that don't think it's important", 'Getting good candidates.', 'General under-representation of women in software so lack of exposure to diversity; lack of visibility of women in software, particularly in leadership positions, can mean respect is harder to earn for women.', 'Limited pool of qualified individuals in my area.', 'The desire is there, very niche, can be tough', 'Have on occasion run across individuals that are skilled but socially toxic. ', 'Having leadership know know how and when to pursue inclusion.', 'The pipeline is dry in regard to people other than white or Indian-born males.', "We haven't worked on many projects that reach out to people outside of our team. Creating an inclusive team is a huge priority, but reaching out to others outside of the team isn't something we have experience with/it's not obvious how to find good ways to do this.", 'Not being recognized for the skills and background that I have.', 'People willing to do other tasks than code.', "Uh, we live in a deeply racist and sexist society, and it's particularly bad in physics. You could write a dissertation on this.\n\nI have been fortunate to work in groups that care about these things more than average. A male leader on one project I actively work on frequently brings women, including women of color, into the collaboration and mentors them. The project also has a woman in a leadership role.", 'Certain members of the project have a reputation for colorful language in _internal_ discussions (never with new or external users/developers). Even after adopting a code of conduct and working consciously to change the present culture, the reputation remains. Although one of the project co-leads is a woman, she works remotely and is more involved in management than direct software development or routine interaction with junior colleagues who might become developers. The most active developers strive to recruit diverse research groups, but the inertia will take time to overcome.', 'It’s a small community. Time is spent on other things.', "The silliness of this obsession. My institution is considered a scientific backwater in terms of mathematics, but our graduate program is nearly entirely female, and a lot of them are pretty good students, certainly better than some of my fellow students at my PhD-granting institution (which made a big brouhaha of including women). All my graduate students have been women, including my one PhD student. One of my undergraduate research students was a woman. I don't care if someone is a man or a woman or whatever; I care if s/he can do the work. I'd much rather have a hard-working woman than a lazy man, and similarly I'd rather have a hard-working man than a lazy woman. As I understand it, my students understand that and appreciate it.", 'Current trend is to move towards industry standards and norms for development practices, sometimes without considering the way software industry norms are bound up with some less than desirable traits of silicon valley tech culture.', "Such a small set of people available and willing to do the work, so we hire any and all qualified candidates that we can find. The company I work for is very small, the two founders were professors, one Indian American and one caucasian American. I work as a subcontractor for another company which is founded and run by an African American on a project let with a different African American, and a colleague who is Indian. None of these people were sought out because of their race or ethnicity, they are the ones who are the best people to work with in the field, and the ones I/we were able to connect with. I will note that the contract associated with this project was mandated from the contracting office of the corresponding government agency to go to an 8A minority owned disadvantaged business. Had they not mandated this, it likely would have gone to a different small business founded and operated by the project lead (African American) instead of the 8A company... So, for what it's worth, additional diversity was injected through the mandate of awarding to an 8A company, who then hired the diverse staff from the other company that would have likely been awarded the contract without this mandate.", 'The current project I work in is largely a remote team scattered throughout the US.', 'The universities we recruit from have even lower levels of members of underrepresented groups than we do. This makes it exceptionally difficult to recruit more members of said groups.', 'Lack of cultural and interpersonal sensitivity in some communities. Inclusion requires spending lots of time with students/postdocs to communicate, demonstrate, and advise, both one-on-one mentoring and in small groups.', 'Developer decision-making tends to be in person at biannual meetings for my main project, a collaboration between a US-based and a UK-based team. Developers with visa restrictions recently have not been able to travel and are thus left out of these meetings entirely. Moreover, these meetings involve a lot of talking over and interrupting each other, so people who are not used to being the biggest shirts and loudest voices in the room are almost never heard. ', 'Finding qualified personnel who are also diverse.', 'There are few candidates. We need tools to make the pipeline fatter.', 'N/A\n', "Having effective communication between people from diverse groups. It's very important to understand the style of communication that is acceptable with each group and communicate in a manner that is comfortable for them.", 'We tend to use the excuse of "very busy" -- regardless of team size, there is always a new algorithm to implement, a new bug to fix, a new feature to design -- to not pay attention to non-technical issues like this. There\'s no conscious objection or opposition to maintaining a culture of inclusion.\n\nTypically, when we make progress in these issues, it\'s because an individual has a passion in this area and prioritizes it in how they allocate their time and contributions.', "This question is underdefined. Who is 'underrepresented'? Non-white, non-christian, non-american born would be considered 'underrepresented' in most of the US. But hardly for scientific programming, where it is very hard to find anyone outside of those categories. ", "I'm a student, so I don't have much control over the applicant pool.", 'Too many white men at the top. When I put myself down for not being technical (and female) no-one speaks up to question it or even to offer support or training. But when the male software engineer with mental health issues says something slightly down about himself everyone rallies round.', 'Unfunded open source community projects are difficult as the main barrier to entry for skilled contributors is their time, and the main barrier for mentoring new contributors is our time.', "I'm not in a position of power to do this", 'Existing team is not very diverse', "Bosses are least bothered about it and don't like subordinate staff taking any initiative or interest without their permission or acceptance.", 'lack of interest, laziness, lack of motivation, lack of funding institutions demanding it', 'pipeline', 'Broadening participation and inclusion are priorities in my current position. The major challenge is sufficient funding to continue these efforts at the necessary level. ', 'Getting enough funding to actually hire support staff.', 'pipeline for the specific roles/skills we need is highly skewed. have struggled to figure out how to iterate on making postings more appealing to more diverse candidates', 'motivation', 'We have a hard time recruiting. Anyone who can do the job is welcome.', 'We achieve inclusion through summer internship hires. We otherwise only have 2 software developers working on our project (both part-time) and they require *very* specific skills. So we have to hire for skills/knowledge regardless of inclusion. We make a strong effort to achieve diversity/inclusion in our larger (temporary) summer hires.', 'I\'m not sure I understand the above questions. What does it mean for a "project" to maintain a culture of inclusion? Are you asking about recruiting software developers? My "software projects" are usually computer science research projects developed by grad students. I believe my lab does well at diversity and inclusion, and I haven\'t faced significant challenges to doing so, because I am selective about the students I work with.', 'Inclusion of diverse people requires diversity, of which there is extremely little to begin with.', 'My institution does not have a very diverse student body on which to draw.', 'Diversity in hiring implies diversity in backgrounds and incoming skill levels. This implies the need for a diverse set of training materials, which is necessary but hard for a small research group to create and maintain.', 'Difficulty attracting people into the field, so they can even begin to participate in our work.', 'making successful hires from under-represented groups, as competition for them is strong', 'The whole project is about this - improving access and inclusion in the classroom, partnering faculty with a student mentor to talk about inclusive practices. Challenges are that access services (interpreting, captioning - for deaf students) are under-supported by the university for grant/project activities - their priority is the classroom needs. Getting these services can be hard - not enough availability - even if we have the budget to pay.', "Trickle down of negative attitudes/behavior from some senior/authoritative individuals. It doesn't take much to significantly affect the atmosphere for the few minority individuals we do manage to attract at more junior levels.", 'Recruitment of qualified individuals and resources to hire', 'our geographic location', 'none', 'High turnover of research students due to being only offering up to MS degrees.', 'Number of available trainees / junior developers with software knowledge and career interest', 'Finding good applicants. ', 'I am a group leader so it is my priority ', 'Not a significant pool of candidates in for academic and research institution, in addition industry offers much competitive pay package than state or private academic institutions where most talent goes. ', 'Access to a diverse student body ', 'The term "inclusion" often doesn\'t include conservative, white males.', 'Fixed staff positions from collaborators make it difficult to build diverse teams.', 'There are less candidates applying for such positions.', 'Very limited number of candidates from underrepresented groups for high profile positions', 'Administration support ', 'pipeline / critical mass', 'We try to hire a diverse staff.... even when we are fortunate enough to hire underrepresented staff members, they leave quickly, often for a higher salary.', 'It is very difficult to find any applicants at all from underrepresented groups. I go after those who apply aggressively.', 'Within our software team, we have no issues with diversity. We have plenty women in leadership positions and also plenty at the level of PhD students and postdocs. Outside of the team is another matter -- the field as a whole still suffers from stereotypes and biases. One most prominent challenge is that women receive less outside recognition (awards, invited talks, reputation) for equal amount of work. I experienced it myself many times -- competing projects lead by males have garnered more visibility than mine. ', 'Actual people to include.', "The people that hit the mailing list or issue tracker are usually from some very narrow demographics already. We don't prioritize engaging with minorities, which means the representation in active participants is similarly skewed. One of the issues here is bandwidth - we don't have time to answer all support requests, and the ones that get replies are somewhat random.", 'Fewer candidates are available from underrepresented community that fit our requirements.', 'Finding qualified and interested applicants', 'Physical proximity. Many of my participants live in Africa and I communicate with them through email. Meetings in person make a difference for joint management and research. We do not have enough opportunities for such meetings, but we do work together, and sometimes manage together, through email correspondence.', 'Our projects are pretty small and most of our software development is outsourced. The software development is not an area were we spend a lot of energy focused on inclusion. ', 'Small available talent pool. Geographical location.', 'Desire to do so and work around the hype', 'I am a male working in a female-dominated industry (nursing research) so the challenge is hiring males within our organizational unit. ', 'A large enough pool of candidates to maintain a diverse group that inherently supports a culture of inclusion.', "Most of the software projects I work on involve 2-3 faculty, students, or postdocs creating and maintaining the code part time. The developers involved reflect the overall institution, which is less diverse than any of us would like (though we've been working on it and improving). The projects serve a small number of internal and external users, and typically have only limited long-term maintenance. There is not much of a user or developer community.", 'The center in which I work is highly homogeneous (white senior males) who consider they already have an inclusive organization. Any attempt to challenge such a conception is met with strong resistance. There are no active efforts to foster a real culture of inclusion. ']
Q7.3: Does your project do any measurement, tracking, or analysis of developer or user communities - with regard to diversity and/or inclusion? If so, please describe below.¶
Text input
In [40]:
print(IRSEM["Q7.3"][2:].dropna().to_list())
['no', "I'm extremely glad they don't, the notion strikes me as positively Stalinesque.", 'No', 'no']
In [41]:
print(URSSI["Q7.3"][2:].dropna().to_list())
['No', "I'm sure our HR department does. My group is small enough that we can look around.", 'No.', 'No', 'yes', 'no', 'Sent out a user survey to determine usage patterns; also a Google Group for support.', 'No.', 'Varies with project.', 'no', 'Nope.', 'Yes - we try to get numbers of how many students use our software and how many might be from underrepresented groups. ', 'no', 'No. WIthin our repositories we are agnostic - it is a user name. WIthin our training and workshops we are mindful. That is we try to get as diverse a pool as possible but it is not always possible. We need a better pipeline and ways to recruit a more diverse pool.', 'no', 'Yes! An external evaluator and constant data collection on students participating in my activities. The inclusion initiative is externally funded and therefore allows (and required) the collection of data.', 'no', 'Yes, but just informally', "No. We don't think in those categories. Diversity and inclusion are normality as they should.", 'No. Not enough resources/skills to do that. Would be valuable though.', 'No', 'yes', 'No', 'Nope', 'No', 'None that I know of', 'No', 'No', 'no, not really.', 'No.', 'No', 'Yes. Our university tracks this information at all levels. I believe that this is a high priority at my institution.', 'no', 'No.', 'no', 'When funding is available we use an outside applied anthropologist to evaluate our interactions within projects', 'No.', 'No', 'We keep track of the background of all individuals and their progress in the program, identifying when they have been able to take advantage of resources to help advance their professional development that involve specialized needs of underrepresented groups. We also conduct focus groups to identify gaps that need to be addressed.', 'So far our software is on GitHub. It is still in development. The hope is when we publish showing the application it will get community interest. But I think it will be hard for others to run without a slick interface.', 'No', 'Analysis of gender/race breakdown of participants at annual summer school.', 'Project has few external developers/users. We do keep track of diversity metrics.', 'No', 'No. We publish and communicate in the open (github) with no registration requirements and therefore no means to monitor diversity.', 'Not much', 'No', 'no', 'no', 'yes', 'No', 'No', 'no', 'no', 'no', "I don't develop software--I just use software to do research.", 'nop', "Don't know", 'Working on it.', 'N//A', 'No', 'no', 'Minor and easy measures ', 'No, not at this time.', 'Most of our students are Native American, so representation of minorities is expected. We do not generally track diversity for these projects.', 'No', 'Yes, because the grants that fund the research track this to some extent.', 'No. ', 'No', 'no', 'No, it does not.', 'No', "Don't know", 'no', 'I am not aware.', 'Yes but that is to satisfy some artificial requirement for inclusion not necessary a natural response from the community. ', "No, but that's a good idea.", 'no', 'not directly.', "Not yet. We aren't to that stage of dissemination yet. ", 'No', 'no', 'No', 'no', 'No', 'Not formally. ', 'No. ', 'No', 'Not formally, except as required in NSF reports, etc.', "I don't think so. Our projects simply aren't big enough to justify the expense at the moment. Most of the work I do involves an analyst (me) working with a single post-doctoral fellow or graduate student, so it doesn't really match the development team model.", 'No', 'No', 'no, sorry.', 'no', 'Not sure.', 'Not at the moment.', 'Yes, surveys. ', 'No', 'No', 'No. ', 'We are integrated into an environment that has NIH and NSF training grants, so we track diversity across our programs', 'No', 'The institution does', 'No', 'Not really.', 'No, but we probably should. ', 'No—most projects are done in spare time; there are no metrics of any kind.', 'Not really.', 'No', 'no', 'Not my project but at the our Office of Institutional Effectiveness does this research.', 'My project is too small for this to be meaningful.', 'No', "I don't know.", 'We are willing to take anyone as long as they have programming skills, but most biologists of any background don’t have any.', 'no', 'no', 'No.', 'No', 'I have tried this, but not enough people use it.', 'No', 'nothing systematic, personal efforts to mentor/sponsor women', 'Not really except that I record and track them myself', 'no', 'No', 'no', 'Project no (too small) but our institute executed a full internal survey on diversity and employment satisfaction. It was very telling and our leadership supported the effort first hesitantly but then fully.', 'we report on diversity and inclusion, as well as tracking our former students, as part of the broader impacts of our NSF project annual reports. ', 'no', 'No', 'No', 'No', 'When we renewed one of our federally funded projects, we did such an analysis with regard to diversity and/or inclusion.', "Not formally, that I'm aware of.", 'We track developer identification and ensure diversity among invitees and funded student participants at annual user workshops.', 'No', 'No', 'No', 'No', 'No', 'No', 'Yes. Affirmative action, active outreach to underserved communities. Tacking of career paths of former members.', 'No', 'I work on CS Education research so most projects track diversity of students.', 'no, we only track nationality', 'No.\n', 'Not that I know of.', 'No.', 'please. ', "We don't do this. I think this task is left to HR", 'Mains a list of named contributors, and in each release names the contributors and highlights the new contributors.', 'No', 'no', 'No', 'no', 'no', 'not sure', 'N/A. No in relation to software development.', 'no', 'No', 'We record demographic information when we do studies of users of our software.', 'No', 'No.', 'none of these questions are relevant for someone in a business school', 'No, unfortunately.', 'Yes.', 'no', 'no', 'Not currently. ', 'no', 'Track students for grant reporting (NSF)', 'No', 'Yes, our internships are geared toward increasing the diversity of developers and these metrics are tracked.', 'Yes, we do annual assessment surveys for science, library, and other professional communities that use our software and data infrastructure and publish the results. We also conduct pre- and post-training assessments for software and data science training programs multiple times per year.', 'no', 'Not systematically. We have basic statistics - % of female contributors in author list, for example. ', 'No.', 'No', 'no', 'Only informally noting # of participants or users from various groups in grant proposals, reports, performance reviews, etc.', 'No', 'no. ', 'no', 'No. My projects are majority women both within my College and University. My collaborators off-campus are computer science and information science researchers.', 'No.', 'No']