Run the study · Administrator
Control when participants see feedback
Combine the consensus method, saved answers, display permissions, and minimum-answer threshold.
Screenshots illustrate the controls below. Layouts and labels may vary with your survey configuration. Select an image or “View full size” to open it in a new tab.
The four checks
- Feedback dataset: Real-Time Delphi uses the current round; Multi-round Delphi uses the previous round.
- Participant answer: feedback is normally available after the person answers and saves the question.
- Minimum Responses to Display Analytics: the question must have the configured number of answers.
- Display settings: the relevant analytics tabs, statistics, comments, and optional subgroup report must be enabled.
These checks serve different purposes. Creating a round is not a substitute for setting a minimum number of panelists per question.
Configure the question or matrix column
In Question Settings, review Display Analytics Tab and Minimum Responses to Display Analytics. In Survey Settings → Statistics, check whether each measure is enabled for Admin + User. Review the question’s Statistics to Display list as well.
Survey-level options can allow analytics before an answer, filter comments by language, hide response counts, and control change alerts. If your protocol requires an independent first rating, keep feedback hidden until the rating is saved.
Explain feedback to participants
Tell participants to answer and click Save if they want to inspect feedback without moving on. They may still see no feedback if too few panelists have answered, the first Multi-round round has no earlier dataset, or their study does not display those tabs.
Consensus Change Alert can notify people of changes on other pages in Real-Time Delphi. Test it for your study length; frequent alerts can interrupt participation. A changing chart is a reason to consider the evidence, not an instruction to conform.
More screenshots and examples (2)
Detailed instructions: Consensus Change Alert
1) Emails
You can send out email alerts to your panelists manually from the main Survey List page. To send out emails, the "Consensus Change Alert" needs to be selected from the main Survey List page.
Each panelist will receive one email containing all survey questions that s/he answered and where consensus or group stability has shifted since the last time s/he answered it.
2) Popup Windows
You can also enable live consensus change alerts which are displayed to panelists in Real-time Delphi Surveys on the right side of their browser as popup windows. For it to work the Consensus Change Alert setting needs to be enabled (Survey Settings > Consensus > Consensus Change Alert). You need to specify when the change alert monitoring takes place (after a certain percentage of respondents have answered:
Consensus Alert Popup Windows being displayed to panelist:
There are 3 types of notifications"
a) Consensus Changed: when consensus has shifted for question. The background color is blue.
b) Consensus Lost: when previously reached consensus has changed to disagreement. The background color is red.
c) Consensus Reached: when consensus has been reached by panelists for question. The background color is green.
Detailed instructions: Display Settings Tab
This tab will allow you to change the display of your survey, configure buttons, their labels, banners, and so on.
Company URL
Link to your company's website. Used with banner. When panelists click on the banner image on top of survey pages, they are taken to this URL in a separate browser window. If your license allows it (requires Survey Branding services), you will see this settings enabled, otherwise it will be disabled. It requires the "http://" or "https://" prefix also.
Banner URL
Link to your banner image. A banner must in in image format such as *.png, *.jpg or *.jpeg. No PDF format is acceptable. Only a single banner can be placed on top of the pages, or in outgoing Surveylet emails. For multiple banners, merge the banners into one. If you do not have a URL for your survey banner, let us know and we will upload it for you and will send you the link. (Learn how to add your own logo.)
Banner Width
The width of your banner to be displayed on top of your survey pages and outgoing emails related to this survey. It is disabled if your license does not allow you to use your own banner (requires Survey Branding services). It accepts values in numeric format which can be affixed with a "px" or "%" sign such as 50px, or 50%. The "px" determines the banner's static size in pixels. The "%" sets the banner's dynamic size as the percentage of displayed viewing port (page). Both width and height can be set at the same time, but we recommend that you only set one of them otherwise your image may be skewed.
Banner Height
The height of your banner to be displayed on top of your survey pages and outgoing emails related to this survey. It is disabled if your license does not allow you to use your own banner (requires Survey Branding services). It accepts values in numeric format which can be affixed with a "px" or "%" sign such as 50px, or 50%. The "px" determines the banner's static size in pixels. The "%" sets the banner's dynamic size as the percentage of displayed viewing port (page). Both width and height can be set at the same time, but we recommend that you only set one of them otherwise your image may be skewed.
Show Page Selector
This setting tells Surveylet if it should display the Page Selector dropdown box on top of all survey pages which your panelists can use to navigate across your survey by directly jumping to certain pages. The Page Selector displays list of all survey pages of your survey with each page showing the title of the very first question on each page. If you want to change the page titles, you will need to adjust the first question titles on each page. The first survey page does not display the Page Selector dropdown box.
This setting tells Surveylet if it should display the Page Selector dropdown box on top of all survey pages which your panelists can use to navigate across your survey by directly jumping to certain pages. The Page Selector displays list of all survey pages of your survey with each page showing the title of the very first question on each page. If you want to change the page titles, you will need to adjust the first question titles on each page. The first survey page does not display the Page Selector dropdown box.
Show Progress Bar
Displays progress bar at the bottom of each survey page. The progress indicates the percentage of questions a panelists have so far answered from all survey questions. 100% means that panelist has answered all questions. Only questions or matrix columns whose "Include in Progress" is set to "Yes" will be included in the progress calculation. You may want to exclude optional questions, or questions on pages where a page branching may skip. Read more on "Include In Progress" question settings.
Displays progress bar at the bottom of each survey page. The progress indicates the percentage of questions a panelists have so far answered from all survey questions. 100% means that panelist has answered all questions. Only questions or matrix columns whose "Include in Progress" is set to "Yes" will be included in the progress calculation. You may want to exclude optional questions, or questions on pages where a page branching may skip. Read more on "Include In Progress" question settings.
Show Back Button
Displays the << (back) button on survey pages. On the first page, there is no << button displayed. This is a survey-level settings, so you cannot control on which pages the back button will appear or be hidden. The Back button will not save the changes made by panelist on the page. When a panelist clicks on the Back button, an informative popup window will appear informing panelist that they will need to click on the SAVE button first if they want to save any changes made on the page. This message will appear on pages that collect information from panelists. It will appear even if the panelist hasn't made any changes.
Displays the << (back) button on survey pages. On the first page, there is no << button displayed. This is a survey-level settings, so you cannot control on which pages the back button will appear or be hidden. The Back button will not save the changes made by panelist on the page. When a panelist clicks on the Back button, an informative popup window will appear informing panelist that they will need to click on the SAVE button first if they want to save any changes made on the page. This message will appear on pages that collect information from panelists. It will appear even if the panelist hasn't made any changes.
Show Save Button
Displays the SAVE link button at the bottom of all survey pages. Responses and changes to existing responses are saved for all questions on a page when user navigates away from the page using the >> (next) button or the Save linkbutton. The difference between the two is that while both buttons save the responses for all questions on the page, the >> button takes the user to the next page while the Save button keeps the user on the same page. Save button is useful in Delphi surveys as Delphi requires that a panelist provides a response to a question before s/he can see the group analytics/consensus for that given question. If panelist clicks on Save instead of Next, then s/he will be able to see analytics immediately and won't have to navigate back later.
Displays the SAVE link button at the bottom of all survey pages. Responses and changes to existing responses are saved for all questions on a page when user navigates away from the page using the >> (next) button or the Save linkbutton. The difference between the two is that while both buttons save the responses for all questions on the page, the >> button takes the user to the next page while the Save button keeps the user on the same page. Save button is useful in Delphi surveys as Delphi requires that a panelist provides a response to a question before s/he can see the group analytics/consensus for that given question. If panelist clicks on Save instead of Next, then s/he will be able to see analytics immediately and won't have to navigate back later.
Show Chat Button
Displays the Chat link button. Panelist can communicate with the survey administrators via a private chat window doing a two-way asynchronous communication. This is a one-on-one communication channel and no other panelists are involved.
Displays the Chat link button. Panelist can communicate with the survey administrators via a private chat window doing a two-way asynchronous communication. This is a one-on-one communication channel and no other panelists are involved.
When a message is sent by a panelist, an email will be sent out to all survey administrators with the panelist's message and any of them can reply to it. New unread messages will be displayed with red background on the View Responses page. Once the message has been viewed by the administrator, it will be grayed out for the administrator who has read the message (other admins will not be affected until they read it also).
Either the administrator or the panelist can initiate a private message. Messages are tied to the survey and not to individual survey round, so they message thread will be kept from survey round to survey round. Messages cannot be deleted.
Survey administrators can apply email variables to private messages, so variables such as "{~user_firstname~}" or {~user_firstname~}, etc. will work if added to the messages. These Email Variables are replaced when the message is sent. When the "{~user_password~}" is used, it will not be displayed to the survey administrator, only to panelist.
Admins may comprise a short message such as below, and copy/paste it to multiple panelists without having to change the content. The message will be customized to each recipient regardless, when the message is sent:
Hello {~user_firstname~}.
Here is your survey URL: {~user_password~} to survey {~survey_title~}.
Best regards,
Survey Admins
An private chat alert notification email will look something like this when panelist sends a private message:
Email header (sent to admin):
Email content (sent to admin):
Show ToDo Button
Displayed the To Do link button at the bottom of the page. This takes panelists to the Missed Questions page where all questions that the user has not answered yet will be displayed. For Matrix questions, Surveylet only displays the Matrix question title and not each individual cells in the matrix. It may be advisable to disable this settings for survey where skip logic is enabled as users may see questions they skipped because of their choice selection and Surveylet lists all missed questions on this page regardless the skip logic.
Displayed the To Do link button at the bottom of the page. This takes panelists to the Missed Questions page where all questions that the user has not answered yet will be displayed. For Matrix questions, Surveylet only displays the Matrix question title and not each individual cells in the matrix. It may be advisable to disable this settings for survey where skip logic is enabled as users may see questions they skipped because of their choice selection and Surveylet lists all missed questions on this page regardless the skip logic.
Show Exit Button
Displays the Exit link button at the bottom of each survey page. Exit will log panelist off the survey session.
Displays the Exit link button at the bottom of each survey page. Exit will log panelist off the survey session.
Label for Back Button
Text for the Back (<<) button displayed to panelists at the bottom of survey pages. If you want to add your own custom label in any language, then see explanation under the "Label for Save Button" section below.
Label for Save Button
Text for the Save button displayed to panelists at the bottom of survey pages. If nothing is specified, "Save" will be displayed. Surveylet automatically translates the "Back" label into other languages, however, if you want to add your own custom label in any language, then you will need to add it between HTML tags using the 2-digit international country code. In the screen shot example below, the Save Page is in English, and the Speichern is in German:
Label for Chat Button
Text for the Chat link button displayed to panelists at the bottom of survey pages. If nothing is specified, "Chat" will be displayed. If you want to add your own custom label in any language then see explanation under the "Label for Save Button" section above.
Text for the Chat link button displayed to panelists at the bottom of survey pages. If nothing is specified, "Chat" will be displayed. If you want to add your own custom label in any language then see explanation under the "Label for Save Button" section above.
Label for ToDo Button
Text for the To Do link button displayed to panelists at the bottom of survey pages. If nothing is specified, "To do" will be displayed. If you want to add your own custom label in any language then see explanation under the "Label for Save Button" section above.
Text for the To Do link button displayed to panelists at the bottom of survey pages. If nothing is specified, "To do" will be displayed. If you want to add your own custom label in any language then see explanation under the "Label for Save Button" section above.
Label for Exit Button
Text for the Exit link button displayed to panelists at the bottom of survey pages. If nothing is specified, "Exit" will be displayed. Exit will not save answers on the page, panelists must first click on Save button (or >> button) if they want to save their answers before they quit. When they quit the survey, they can still go back. It does not flag their survey progress at 100%. It simply logs them off their current survey session. If you want to add your own custom label in any language then see explanation under the "Label for Save Button" section above.
Text for the Exit link button displayed to panelists at the bottom of survey pages. If nothing is specified, "Exit" will be displayed. Exit will not save answers on the page, panelists must first click on Save button (or >> button) if they want to save their answers before they quit. When they quit the survey, they can still go back. It does not flag their survey progress at 100%. It simply logs them off their current survey session. If you want to add your own custom label in any language then see explanation under the "Label for Save Button" section above.
Show Sticky Footer
Surveylet can display the footer with the 4 link buttons (Save, Chat, To do, Exit) right under the << and >> navigation buttons, or place them on a sticky footer that will always be visible when scrolling. You have the option to move only the Save button to the sticky footer and leave all others under the navigation buttons.
Surveylet can display the footer with the 4 link buttons (Save, Chat, To do, Exit) right under the << and >> navigation buttons, or place them on a sticky footer that will always be visible when scrolling. You have the option to move only the Save button to the sticky footer and leave all others under the navigation buttons.
Display Sticky Footer: "No": the footer may not be visible when scrolling on longer pages.
Display Sticky Footer: "Yes": the footer will always be visible when scrolling.
Display Sticky Footer: "Yes - Only Save Button": the footer with the green Save button will always be visible when scrolling. The rest of the menus (Chat, To Do, Exit) will not be visible when scrolling.
Enable Sentiment Analytics
Enables sentiment analytics for open-ended questions and matrix columns. Only Essay/Textarea questions and My Comments boxes will be analyzed. Short textbox question types are not included in the analysis. This is an optional add-on service which requires Sentiment Analytics service added to your Surveylet account.
Enables sentiment analytics for open-ended questions and matrix columns. Only Essay/Textarea questions and My Comments boxes will be analyzed. Short textbox question types are not included in the analysis. This is an optional add-on service which requires Sentiment Analytics service added to your Surveylet account.
In-Survey Analytics (Admin)
Survey administrators can log on to the survey as a panelist and carry out read-only in-survey analytics. To test the survey, administrators will need to close the survey and click on Test Survey page and select a panelist to test the survey as. The panelist's answers will be read-only, and under each page the filter footer will appear that will allow administrators to filter the incoming responses. Surveylet allows up to 4 filters which are concatenated using the AND boolean operator.
Survey administrators can log on to the survey as a panelist and carry out read-only in-survey analytics. To test the survey, administrators will need to close the survey and click on Test Survey page and select a panelist to test the survey as. The panelist's answers will be read-only, and under each page the filter footer will appear that will allow administrators to filter the incoming responses. Surveylet allows up to 4 filters which are concatenated using the AND boolean operator.
The Distribution graph ("Graph") will be evaluated using the filters selected underneath the chart.
You can calculate Percentages or actual Counts using the two link buttons ("Recalculate Pct" and "Recalculate Counts") at the bottom of the page. If the panelist selected "I don't know" the Graph chart will not be shown. The Consensus or Statistics tabs will not be filtered; the filter is only applied to the Graph chart.
Streaming Analytics (Panelists)
Enables streaming analytics for panelists. If set to Yes, panelists will see question analytics refreshed as new responses are received without them having to manually refresh their browser. Only the question analytics window is refreshed, hence their survey experience is not affected. It is useful only in Real-time Delphi survey. Multi-round Delphi surveys will ignore this settings. It is mostly usable during a short team-meeting, or live conferences where instant decision-making is critical. Enabling it for a longer running survey (those that take more than just a few hours) will not provide any benefits.
Enables streaming analytics for panelists. If set to Yes, panelists will see question analytics refreshed as new responses are received without them having to manually refresh their browser. Only the question analytics window is refreshed, hence their survey experience is not affected. It is useful only in Real-time Delphi survey. Multi-round Delphi surveys will ignore this settings. It is mostly usable during a short team-meeting, or live conferences where instant decision-making is critical. Enabling it for a longer running survey (those that take more than just a few hours) will not provide any benefits.
Streaming Analytics (Admin)
Enables streaming analytics for administrators. Default is No. When set to Yes, administrators can monitor incoming responses in all charts on the Analyze Survey page in real-time without having to refresh their browsers.
Enables streaming analytics for administrators. Default is No. When set to Yes, administrators can monitor incoming responses in all charts on the Analyze Survey page in real-time without having to refresh their browsers.
Display Pre-Response Analytics
When it is enabled (set to Yes), panelist will see the consensus analytics tabs that administrator enabled under the question before s/he answers the question. If it's set to No (which is the default value), panelist must first submit a response to a question before s/he can see group analytics for that question. The Delphi methodology requires that panelist first answer the question before s/he can see group consensus and feedback analysis in order to record his/her unbiased view and monitor deviation from original position. Carefully consider this requirement for your Delphi survey before your enable this feature, as you may violate the Delphi requirements and invalidate your results in an official publication. This feature takes effect for both Standard and Matrix question types.
When it is enabled (set to Yes), panelist will see the consensus analytics tabs that administrator enabled under the question before s/he answers the question. If it's set to No (which is the default value), panelist must first submit a response to a question before s/he can see group analytics for that question. The Delphi methodology requires that panelist first answer the question before s/he can see group consensus and feedback analysis in order to record his/her unbiased view and monitor deviation from original position. Carefully consider this requirement for your Delphi survey before your enable this feature, as you may violate the Delphi requirements and invalidate your results in an official publication. This feature takes effect for both Standard and Matrix question types.
Filter Comments by Language
Only for multi-lingual surveys. When enabled, panelists will only see comments from other panelists that match their selected language. Default is disabled. For instance, if the survey design languages are English and German, and panelist's selected language is German, s/he will only see comments provided by other German-speaking panelists; all English comments will be hidden. Administrators are not affected by this settings, they will always see all comments in all languages.
Only for multi-lingual surveys. When enabled, panelists will only see comments from other panelists that match their selected language. Default is disabled. For instance, if the survey design languages are English and German, and panelist's selected language is German, s/he will only see comments provided by other German-speaking panelists; all English comments will be hidden. Administrators are not affected by this settings, they will always see all comments in all languages.
Identity Masking Question
Sets the question whose answer will be displayed in place of the Panelist's name. For instance, if you select the "What is your level of expertise?" radio button question, the answer the panelist selected for this question will be used to mark the comments made by panelists in the All Comments section.
Sets the question whose answer will be displayed in place of the Panelist's name. For instance, if you select the "What is your level of expertise?" radio button question, the answer the panelist selected for this question will be used to mark the comments made by panelists in the All Comments section.
For instance, the Identity Masking Question is set to "Indicate the level of your expertise" in Survey Settings > Display Settings:
Panelist selects an answer to this question:
Which answer will be appended to their comments (displayed in brackets with bold font):
Hide Response Counts (Panelists)
Participants will view the rating distribution as percentages but Response counts will be hidden from them from the Consensus and the Graph tabs. Administrators will still be able to see raw counts from the Analytics pages. Hiding response counts will limit dominance and bandwagon effects.
Participants will view the rating distribution as percentages but Response counts will be hidden from them from the Consensus and the Graph tabs. Administrators will still be able to see raw counts from the Analytics pages. Hiding response counts will limit dominance and bandwagon effects.
Consensus tab will no longer show People Invited, Responses Received and Revisions Made values, only Consensus Status and Group Stability values will be displayed:
. The distribution chart on the Graph tab will only show the percentages but actual response counts will be hidden:
Stakeholder Analytics embedded charts will only display percentages, not actual counts when you hover over the individual bars as shown below::
Stakeholder Analytics displayed via a popup window will also hide all response counts in both the bar charts hover help and the grid displayed below and will display only percentages:
Style Sheet
You can apply style to your question labels, descriptions, footers using Style Sheets. With style sheet, the same style will be applied to all questions and can be changed at once. This eliminates the need that you style each question separately. To use Style Sheet, basic HTML editing skills and knowledge is required. Each survey element (headers, questions, question details, choice labels, matrix column and row titles, etc.) has its corresponding style sheet entry. You may also define your own CSS class here and assign it to different questions.
You can apply style to your question labels, descriptions, footers using Style Sheets. With style sheet, the same style will be applied to all questions and can be changed at once. This eliminates the need that you style each question separately. To use Style Sheet, basic HTML editing skills and knowledge is required. Each survey element (headers, questions, question details, choice labels, matrix column and row titles, etc.) has its corresponding style sheet entry. You may also define your own CSS class here and assign it to different questions.
Available Style Sheet Elements:
.styleSurveyBody: sets the style for the entire survey page including background color.
.styleSurveyContainer: for the inner container of the survey page (set the bacground color and page viewing width here).
.styleChoiceIdontknow: for styling the 'I don't know' question choice in radio button questions.
.styleQuestion: to style question titles.
.styleQuestionDetails: to style question details section of questions.
.styleMatrixColumnTitle: to style matrix column titles.
.styleMatrixColumnChoiceTitle: to style matrix column subchoice titles.
.styleMatrixRowTitle: to style matrix row titles.
.styleMatrixCommentCount: to style comment count numbers for matrix cells.
.styleMatrixCommentCountZero : to style comment count numbers for matrix cells where comment count is 0.
.styleMatrixCommentIcon: the icon to be displayed for comment counts for matrix cells.
.styleMatrixComment: to style matrix comment text.
.styleMatrixPopupHelpIcon : to style matrix row popup help icons.
.styleMatrixPopupHelpTextArea: to style popup help text area for matrix cells.
.styleMatrixPopupHelpColumn : to style popup help columns for matrix rows.
.styleMatrixPopupHelpColumnHeader : to style the title of matrix popup help windows.
.styleMatrixStats : to style consensus statistics values in matrix cells.
.styleMatrixGraph : to style matrix distribution graphs displayed for matrix cells.
.styleMatrixStatsConsensus : to style consensus stats for matrix cells. default is green.
.styleMatrixStatsDissent : to style dissent stats for matrix cells. default is red.
.styleMatrixIdontknow : to style the "I don't know" option row for matrix questions.
.styleMatrixRowOdd: to style odd-numbered rows in matrix (will only style first column of each odd row)
.styleMatrixRowEven: to style even-numbered rows in matrix (will only style first column of each even row)
.bmi-caption: to style the Row Label matrix rows.
.styleChatButton: to style the Chat button in footer.
.styleExitButton: to style the Exit button in footer.
.styleToDoButton: to style the To Do button in footer.
.styleSaveButton: to style the Save button in footer.
.lbl: to style the question choices.
.rankOrder: to style the ranking question choices.
.numericE: to style the constant sum question choices.
.mxCatsAdapt: to style the adaptive row in matrix questions.
.styleSliderTopLabel: to style standard sliders labels.
.jqx-slider-label-top: to style numeric/constant sum slider top labels.
.stylePrefix: to style prefix text for radio buttons/sliders.
.stylePostfix: to style prefix text for radio buttons/sliders.
You can add your own custom style element and use them in your survey.
To add JavaScript to your survey, you will need to add it to the Style Sheet section by closing the Style Sheet section first ("</style>"), then inserting the actual JavaScript content between <script type="text/javascript">...</script> tags, and then reopening the Style Sheet section again (with "<style>") as shown in the example below:
Common questions
The administrator sees analytics, but panelists do not. Why?
Administrator access and participant display rules differ. Check the saved answer, feedback round, minimum-answer threshold, tabs, and Admin + User visibility.
Will feedback appear as soon as a new snapshot is created?
Only if the other display conditions are satisfied. The snapshot and minimum-answer threshold are independent settings.