I'm working on a web application based on a LEMP stack (PHP7) and have recently started using Bitbucket Pipelines as a CI tool. It works very well to run unit tests using the following setup for bitbucket-pipelines.yml as they provide example of on their website.
However, when it comes to including acceptance testing in the CI pipeline, I have run into trouble…
Acceptance test with Codeception and MailCatcher module as part of CI
I currently run acceptance test with Codeception and the MailCatcher module from the command line in my local development environment. This works very well, even if it takes some time to run all of them.
What I would like to do is to integrate the acceptance testing in the CI pipeline so that they are executed automatically at each push to my Bitbucket repo. If any acceptance test fail I will be notified by email, which is what happens right now with the unit tests. I have looked at the alternatives below, but have unfortunately not succeeded with any.
Scenario 1: In Bitbucket Pipelines
I have tested following these two guides: https://rihards.com/2017/bitbucket-pipelines-phpunit-codeception/ and http://farza.info/?p=410 (translated into English), but have not succeeded.
Result: Failed to run Selenium Webdriver in the background. The build process paused when Selenium Webdriver has running and the script did not reach to execute the tests.
Reflection: A reflection about placing the acceptance test in Bitbucket Pipelines is that I see that the building time may be unreasonably long.
Scenario 2: Locally in Docker containers
Create a bash script that run locally in my development environment every time I push to Bitbucket. The bash script run the acceptance tests in a local Docker container (using the official Codeception Docker image). Further I run a cron job on a regular basis to check if there are any failed tests in the directory that they end up in when running Codeception. If there are any and email is sent to me.
Result: I did not manage to access Doctrine repositories which is necessary for creating mock-up objects which I use in the tests.
Scenario 3: Locally in the development environment
Create a bash script that run locally in my development environment every time I push to Bitbucket. The bash script run the acceptance tests daemonized in my development environment. The other steps are the same as in Scenario 2.
Result: I failed to run Codeception as a background process, thereby making use of script more or less meaningless.
• Does anyone have any comments on the scenarios above and suggestions setting up an effective CI workflow with both unit and browser testing?
• Has anyone used BrowerStack in Bitbucket Pipelines or otherwise in conjunction with CI for PHP applications?