What is MSF/360?
Multi-source feedback programs (MSF) or 360 are research processes that involve data collection about a professional healthcare provider's performance from multiple categories of stakeholders including peers or colleagues, co-workers and patients/clients.
Since the mid 2000's, several health regulatory authorities across Canada have adopted various types of Multi-Source Feedback programs. Many more authorities, institutions and health professionals continue to ask whether they should do the same and to inquire about the applicability of such programs in their respective profession.
The main goal of MSF is to provide participants with constructive feedback and observations upon which participants are able to devise plans to enhance their practice. While some programs may rely on MSF for performance evaluation, the vast majority of current programs are educational in nature and only seek to complement existing means in helping participants learn more about how they can deliver better quality health care to their patients and be part of a strong health care team.
MSF surveys vary depend on each health discipline. Typically, surveys will cover several areas including: medical expertise, management, communications and interaction with patients and other colleagues.
Provides direct and honest feedback from various stakeholders
Provides detailed report covering multiple practice dimensions
Compliments existing feedback instruments and tools
Why use MSF/360?
Multi-source Feedback (MSF) and Single Source Feedback (SSF) programs provide various advantages to health professionals. Whether the focus is Quality Assurance or Quality Improvement, MSF programs:
- Give the public an opportunity to provide feedback on the care they received from health professionals.
- Support health professionals in their practice as they will receive valuable feedback from the people directly impacted by their practice, including their direct staff and other health professionals with whom they interact regularly.
- Complement the various Continuing Competency Programs required by the various health regulatory authorities across the country. We know not all programs work for all health professionals.
- Provide participants with actionable feedback that can be used towards professional self-development.
- Help participants focus on elements of communication and professional behaviour.
Who uses MSF/360?
Frequently Asked Questions
How to select the most suitable feedback model?
MSF programs may look alike. But often the case, they differ substantially. Some are Quality Improvement driven while others are Quality Assurance driven. Some only serve as complementary to peer assessment programs.
How is MSF/360 used to improve service quality?
MSF programs are used in a variety of ways. Some regulatory authorities only review results that fall below set criteria; otherwise reviewees are asked to reflect on their data and make their own decisions regarding practice changes. Other authorities have developed integrated assessment models that combine MSF results with data from other assessments. Both are quality assurance initiatives.
Medical chiefs in hospital-based programs typically review results with the reviewee, provide coaching, and set goals for change as required. In this instance, MSF data has performance measurement objectives.
How many responses are required to generate meaningful and valid results?
MSF/360 programs used by medical regulatory authorities that are based on the Alberta’s Physician Achievement Review program require surveys from a minimum of 18 patients, 6 medical colleagues and 6 co-workers.
However, each MSF/360 program is unique and so are the required numbers of completed responses.
What are the different categories of reviewers that I should consider?
Reviewers can be your colleagues, medical peers, patients, parents/guardians of patients, or co-workers in other healthcare fields. Depending on the program, instructions are provided to the participant to ensure they select the most suitable individuals.
How should we select reviewers and who should they be?
Reviewers typically include colleagues, co-workers and patients/clients or parents/guardians of patients. Reviewers must be able to offer an informed opinion. In some programs, the reviewee selects the reviewers while in others the reviewers are selected by program heads or administrators. Provided there are a sufficient number of reviewers there is no evidence of selection bias in either case.
Who develops the survey instruments?
Survey instruments are developed independently or collaboratively by regulatory authorities, academic institutions and research firms like Pivotal Research. Regardless, the construction and validation of instruments must meet accepted scientific practice.
What is the value of comparing results to normative group?
Norms are useful because they provide a context for interpreting results. Suppose that a person receives an average rating of 7 out of 10. Is that rating average or above or below average? The answer is contingent on the normative distribution. If the group average is 6, a score of 7 is favorable. If the group average is 9, then a score of 7 suggests room for improvement.
How do you generate normative group data?
Norms are based on the performance of a population or a sample of reviewees that is large enough to ensure that the results are representative of the population. Group results are displayed in a distribution against which scores for a single reviewee can be compared.
How often should I conduct an MSF/360 for my practice?
Frequency varies widely depending upon the resources required to complete a multi-source feedback program. Some regulatory authorities have adopted a 5-year cycle, others a 7-year cycle. Some institutions have used shorter cycle times in order to measure the impact of a significant change in procedures, systems or training. In voluntary programs, the health professional can participate as often as they wish. Once every two years might be optimal to monitor progress while allowing sufficient time for practice changes to take effect.
How are people selected to participate in MSF/360?
Some institutions select reviewees within a department. Regulatory authorities use either random selection or non-random selection by focusing on a particular group or type of practice. In some programs a reviewee volunteers to participate.
How do you ensure the security of collected data of the program?
Data confidentiality and security are top priorities at Pivotal Research. Our applications are programmed, monitored and maintained by our resident programmers using the ASP.NET platform. We have strong security systems in place and we are vigilant about updating security patches and monitoring event logs for suspicious activity.
Pivotal Research uses SQL Server to house data and therefore we are able to use Microsoft SQL Servers securities (i.e. database permissions). Applications are run with the fewest privileges possible. A database user with limited access and privilege is allowed to perform the necessary database queries and nothing more.
Network internet security is provided by ESET, and our Internet Information Server (IIS) is configured to use process throttling to prevent malicious attackers from bringing down the application. Using a Fortigate firewall also enhances our level of security.
Our file, email and web servers are secured, and our physical firewall is locked down. Web applications are hosted on our web server, but data gathered through the portal will be stored on our SQL server. This separation provides a further layer of security for the data. Sensitive information such as usernames, passwords, or PINs are never stored in areas that may be accessible to the user's browser. All of our servers are protected by strong passwords, and complete backups are performed daily.
Pivotal Research's web server is secure and we use our SSL (Secure Socket Layer) certificate to ensure that data is encrypted.
Our Privacy and Confidentiality Protocol ensures that you can be confident in the security and privacy of all data.
Given our extensive experience hosting sensitive and confidential data for various clients, we have developed several strategies to mitigate the effects of unforeseen failures.
All of our data systems are backed up completely on a daily basis, and the backups are stored in a safe location off-site. Backup data is encrypted and can be restored from the previous five days. We maintain redundant software and hardware in case of equipment failure, and we contract same-day technical IT support to respond to software or hardware concerns.
In case of a partial or complete system failure, the first steps are to assess the impact and notify affected clients. Once the impact has been determined, systems have been restored, and data has been recovered from the backups, Pivotal Research will work closely with affected clients to develop a recovery plan.
The functions required for this project can be performed by a minimum of two Pivotal Research staff members. This redundancy ensures cover off staff resources.
What are the costs associated with participation?
Given our experience in administering MSF/360 programs spans almost 20 years, we have developed a pricing model that is quite reasonable for clients. Fees for Pivotal Research vary depending on the nature of the program, the overall number of participants, the number of stakeholders providing feedback to participants, and the data collection methodology.