Conducting a systematic literature review: reflections and resources

For the past couple of months I have been undertaking a systematic review (of sorts) of the learning outcomes (LOs) literature.  I say ‘of sorts’ because I have followed all but the final few protocols of a systematic review – I have reviewed the extent of certain features within the LOs literature but have not quality assessed studies of LOs (I am being deliberately vague here as I am currently writing up my findings for publication).  I have, therefore, termed the process I’ve been engaged in as a ‘systematically-organised review’.

Falling in love?

My aim here is not to split hairs about terminology but to reflect briefly on the nature of what I’ve been doing.  Not long after I started my review, I came across this blog post titled ‘My first systematic review: How I fell in love’.  I have to admit, at that point I was not sure whether I liked doing my review, let alone loved it.  But as time went on I got to understand more what Dr Sini Savilaakso was talking about – I had actually started to enjoy the process.  In her blog post Dr Savilaakso explains how she ‘fell in love’ with her first systematic review.  I’d like to add here a couple of points about why I’ve enjoyed the process so much too:

• To some extent there has felt less pressure within this whole process, but at the same time I have felt that I’ve been doing really important work.  Let me explain – I have felt less pressured because I am not trying to create new knowledge, arguments, theories, etc., around the issue of LOs.  I have instead been exploring what has already been ‘put out there’ to make some kind of assessment about the make-up of the LOs literature.  This is really important to do because so much that is incorporated into education does not often have its literature or research base scrutinised.  To understand if an educational approach or strategy has any kind of validity, we should be exploring (and assessing) the body of knowledge surrounding it.  This is why I feel that I have been doing important work, even though I’ve felt at the same time that it’s been a less pressurised process.

• It has helped me to feel more of an ‘expert’ in my chosen area (i.e. learning outcomes) and to talk about it with confidence due to my detailed and in-depth knowledge of the published literature.  Of course, all research should be grounded within thorough literature reviews, but undertaking a specific review in such a rigorous and robust manner (i.e. with specific and recorded search strategies, methodically searching databases etc) means there is less chance that I have gaps in my knowledge by missing important pieces of work.

Useful links

I’d like to add here some links to sites that I’ve found useful, particularly for developing my search parameters and protocols:

– Learning from research: systematic reviews for informing policy decisions: a quick guide

– Template for a Systematic Literature Review Protocol

– Sense about science: systematic reviews

– Social research update: systematic reviewing

– Systematic reviewing in the social sciences

Lessons learned

In her blog post, Dr Savilaakso also offers some useful lessons that she has learned from her experience of systematic reviewing.  I’d like to add a few of my own:

•  Dr Savilaakso says to work with team mates that you love.  I would reinforce this point, but also add the need to work with team mates who have sufficient time to give to the process too.  In my case, I was the lead researcher and conducted the whole review, but for reliability purposes I needed at certain points another particular project member to review samples of the literature retrieved so that we could compare our results.  This colleague was extremely busy with other research projects and found it hard to find the time to do these sample reviews.  Whilst for the most part it did not cause too much of a problem, I did get anxious at certain points that I could not progress with the review until we’d undertaken one of these reliability checks.  Of course, we are all very busy and to some extent this sort of situation might be unavoidable – but it is a point worth bearing in mind, particularly if you want to get your results published as soon as possible.

• Accept that at times it will feel a very tedious process.  As much as I enjoyed it overall, there were many times that all the database searching, reviewing piles of papers, etc, felt very tedious.  When you start feeling like this, my advice is to take some kind of break – go for lunch, get a cuppa, do some other work for a bit.  When I felt like this my mind would want to start rushing through everything to get it finished – but of course the danger then is that you rush by and overlook something important.  Accept that rigorous and robust results are achieved largely by work that often feels tedious and time-consuming.

• Allow time for hard to access papers to be found (either by yourself or by research/library support).  Don’t rush into writing up your results until you’re sure your literature searching is complete and you’ve retrieved all the papers you need.

As my own first review is coming to its end, I still wouldn’t say I’ve fallen in love with the process, but I have been surprised at how much I have enjoyed it – from the feelings of rigorous literature searching to the satisfaction of providing answers to my research questions.  I am currently writing up my review for publication and I hope I can share more information about it with you soon.

In the meantime, happy reviewing!