starred/cypht

Fork 0

mirror of https://github.com/cypht-org/cypht.git synced 2026-04-25 13:05:53 +03:00

[GH-ISSUE #250] ✨End2End Tests #211

New issue

Closed

opened 2026-02-25 21:34:27 +03:00 by kerem · 11 comments

kerem commented

2026-02-25 21:34:27 +03:00

Owner

Originally created by @ulfgebhardt on GitHub (Jan 5, 2018).
Original GitHub issue: https://github.com/cypht-org/cypht/issues/250

Originally assigned to: @jasonmunro on GitHub.

It would be awesome to have Cypht to be End2End tested. This could reduce Bugs, espec. broken Behaviour like #225 #246 #248 #249 and would make sure developers (you ;-) can spend time on new Features rather then fixing Bugs.

Write User Stories
e.g. Setup Account
e.g. Send EMail
e.g Reply to Email
....
Test those Userstories with End2End Test
Automatically test those on Travis before merging into Master
Prevent Merges on Master if Tests detect broken Functionality
...
Profit

Grüße Ulf

Originally created by @ulfgebhardt on GitHub (Jan 5, 2018). Original GitHub issue: https://github.com/cypht-org/cypht/issues/250 Originally assigned to: @jasonmunro on GitHub. It would be awesome to have Cypht to be End2End tested. This could reduce Bugs, espec. broken Behaviour like #225 #246 #248 #249 and would make sure developers (you ;-) can spend time on new Features rather then fixing Bugs. 1. Write User Stories e.g. Setup Account e.g. Send EMail e.g Reply to Email .... 2. Test those Userstories with End2End Test 3. Automatically test those on Travis before merging into Master 4. Prevent Merges on Master if Tests detect broken Functionality 5. ... 6. Profit Grüße Ulf <3

kerem

2026-02-25 21:34:27 +03:00

closed this issue
added the
enhancement
label

kerem commented

2026-02-25 21:34:27 +03:00

Author

Owner

@jasonmunro commented on GitHub (Jan 5, 2018):

We already have a Selenium test setup, and at one time I was running the selenium tests in Travis. I disabled that for now because of intermittent failures with Browserstack, and because it slows the travis runs WAY down. Currently the selenium tests don't send/reply to an email, but it would be great to add.

@jasonmunro commented on GitHub (Jan 5, 2018): We already have a Selenium test setup, and at one time I was running the selenium tests in Travis. I disabled that for now because of intermittent failures with Browserstack, and because it slows the travis runs WAY down. Currently the selenium tests don't send/reply to an email, but it would be great to add.

kerem commented

2026-02-25 21:34:27 +03:00

Author

Owner

@ulfgebhardt commented on GitHub (Jan 5, 2018):

Regarding Performance:

Here is an example how i configured Travis to run iOS builds only on the already merged Master Branch, since it takes up to 2h to get a hold on a iOS Travis Maschine:
https://github.com/demokratie-live/democracy-client/blob/master/.travis.yml#L66

And here only for PR's and Master:
https://github.com/demokratie-live/democracy-client/blob/master/.travis.yml#L25

Grüße Ulf

@ulfgebhardt commented on GitHub (Jan 5, 2018): Regarding Performance: Here is an example how i configured Travis to run iOS builds only on the already merged Master Branch, since it takes up to 2h to get a hold on a iOS Travis Maschine: https://github.com/demokratie-live/democracy-client/blob/master/.travis.yml#L66 And here only for PR's and Master: https://github.com/demokratie-live/democracy-client/blob/master/.travis.yml#L25 Grüße Ulf <3

kerem commented

2026-02-25 21:34:27 +03:00

Author

Owner

@jasonmunro commented on GitHub (Jan 5, 2018):

The problem here is that our travis build is actually 18 different configurations, which we can run 5 at a time. Browserstack only allows me to run 2 at a time, so I don't want to fire off the selenium tests for every system configuration because I will end up getting time out errors from Browserstack. I'm not sure if I can run the selenium tests for just one build instance, but that would be good.

Another way to help with these types of issues to use unit tests. I recently built tests for the entire core module set, and started on IMAP but have more work to do (it's a very large module set). Unit test around these routines would help with regressions.

an example of our travis build:
https://travis-ci.org/jasonmunro/cypht/builds/325513985

@jasonmunro commented on GitHub (Jan 5, 2018): The problem here is that our travis build is actually 18 different configurations, which we can run 5 at a time. Browserstack only allows me to run 2 at a time, so I don't want to fire off the selenium tests for every system configuration because I will end up getting time out errors from Browserstack. I'm not sure if I can run the selenium tests for just one build instance, but that would be good. Another way to help with these types of issues to use unit tests. I recently built tests for the entire core module set, and started on IMAP but have more work to do (it's a very large module set). Unit test around these routines would help with regressions. an example of our travis build: https://travis-ci.org/jasonmunro/cypht/builds/325513985

kerem commented

2026-02-25 21:34:27 +03:00

Author

Owner

@ulfgebhardt commented on GitHub (Jan 5, 2018):

I think UnitTests have high chances to miss those kind of Problems.
Thats why i proposed End2End.

If you already have tests written i think it would be wise to use them. Its probl. somehow possible to run travis builds sequential rather then parallel.

Check this out: https://docs.travis-ci.com/user/build-stages/

Grüße Ulf

@ulfgebhardt commented on GitHub (Jan 5, 2018): I think UnitTests have high chances to miss those kind of Problems. Thats why i proposed End2End. If you already have tests written i think it would be wise to use them. Its probl. somehow possible to run travis builds sequential rather then parallel. Check this out: https://docs.travis-ci.com/user/build-stages/ Grüße Ulf <3

kerem commented

2026-02-25 21:34:27 +03:00

Author

Owner

@jasonmunro commented on GitHub (Jan 5, 2018):

I can run them sequential, but then it would take an hour to run them all instead of 15 minutes :) Adding selenium test runs to each would mean it would take a lot longer. I used to have all the unit tests and selenium tests run 2 parallel since that is my max at browserstack, and it was taking 3-4 HOURS to complete.

@jasonmunro commented on GitHub (Jan 5, 2018): I can run them sequential, but then it would take an hour to run them all instead of 15 minutes :) Adding selenium test runs to each would mean it would take a lot longer. I used to have all the unit tests and selenium tests run 2 parallel since that is my max at browserstack, and it was taking 3-4 HOURS to complete.

kerem commented

2026-02-25 21:34:28 +03:00

Author

Owner

@ulfgebhardt commented on GitHub (Jan 5, 2018):

If you do that only on master, after merge?
Whats the downside?

https://github.com/demokratie-live/democracy-client/blob/master/.travis.yml#L66

Grüße Ulf

@ulfgebhardt commented on GitHub (Jan 5, 2018): If you do that only on master, after merge? Whats the downside? > https://github.com/demokratie-live/democracy-client/blob/master/.travis.yml#L66 Grüße Ulf <3

kerem commented

2026-02-25 21:34:28 +03:00

Author

Owner

@jasonmunro commented on GitHub (Feb 7, 2018):

No real downside, it's just a pain to wait that long to find out you broke something :) I have also had some weird errors randomly from browserstack - rerun the exact same test run and it works :/ Anyway, I will see about re-enabling them somehow without slowing everything down too much.

@jasonmunro commented on GitHub (Feb 7, 2018): No real downside, it's just a pain to wait that long to find out you broke something :) I have also had some weird errors randomly from browserstack - rerun the exact same test run and it works :/ Anyway, I will see about re-enabling them somehow without slowing everything down too much.

kerem commented

2026-02-25 21:34:28 +03:00

Author

Owner

@jasonmunro commented on GitHub (Aug 23, 2018):

@ulfgebhardt Spent some time troubleshooting the Browserstack issues and I'm happy to report almost everything is fixed.

We now run the following in Travis CI when code is pushed to the master branch:

6 different PHP version (5.4 - 7.2) X 3 different DB backends (Sqlite, Mysql, Postgres) for a total of 18 jobs
One job is used to generate a coverage report for Coverage.io
4 jobs run our Selenium "end2end" tests with a browser (IE, Edge, FF, and Chrome)
total run-time is about 20 minutes, which I can live with :)

The last thing to do here is get Safari working, and add more tests!

Travis build history is located here: https://travis-ci.org/jasonmunro/cypht/builds

@jasonmunro commented on GitHub (Aug 23, 2018): @ulfgebhardt Spent some time troubleshooting the Browserstack issues and I'm happy to report almost everything is fixed. We now run the following in Travis CI when code is pushed to the master branch: - 6 different PHP version (5.4 - 7.2) X 3 different DB backends (Sqlite, Mysql, Postgres) for a total of 18 jobs - One job is used to generate a coverage report for Coverage.io - 4 jobs run our Selenium "end2end" tests with a browser (IE, Edge, FF, and Chrome) - total run-time is about 20 minutes, which I can live with :) The last thing to do here is get Safari working, and add more tests! Travis build history is located here: https://travis-ci.org/jasonmunro/cypht/builds

kerem commented

2026-02-25 21:34:28 +03:00

Author

Owner

@ulfgebhardt commented on GitHub (Aug 24, 2018):

Hey im glad to hear that - i have a test expert - maybe i get him looking into this project <3

Regarding test-times...

We use Branch based developing - so we make a branch for every feature and merge it into master if it works well. This setup might be a bit overhead for a single developer, but it has advantages regarding testing:

On push we do just basic tests like lint and "does it compile"
On PR we do advanced tests including different Platforms
After merge on Master we do extensive tests which are unpractically to run (no security here but a feedback if something is wrong)

See here and here

Grüße Ulf

@ulfgebhardt commented on GitHub (Aug 24, 2018): Hey im glad to hear that - i have a test expert - maybe i get him looking into this project <3 Regarding test-times... We use Branch based developing - so we make a branch for every feature and merge it into master if it works well. This setup might be a bit overhead for a single developer, but it has advantages regarding testing: - On push we do just basic tests like lint and "does it compile" - On PR we do advanced tests including different Platforms - After merge on Master we do extensive tests which are unpractically to run (no security here but a feedback if something is wrong) See [here](https://github.com/demokratie-live/democracy-development/blob/master/.travis.yml) and [here](https://github.com/demokratie-live/democracy-client/blob/master/.travis.yml) Grüße Ulf <3

kerem commented

2026-02-25 21:34:28 +03:00

Author

Owner

@jasonmunro commented on GitHub (Aug 28, 2018):

I'm familiar with feature branch developing, and I understand the rational. For Cypht I'm sticking with master == development branch, releases are branched from there. It's just easier since I'm the only full time contributor :). I'm actually pretty happy with how things worked out - 4 instances with selenium tests (soon to be 5 once I resolve safari issues) is plenty for now.

I'm also pleased to mention that I have a new test suite that:

navigates to the compose page
loads and submits a new message
navigates to the unread page
finds the new message in the list
opens the new message and loads the detail page

Now that selenium testing is re-enabled in Travis, I'm going to close this issue, and open some new ones for the remaining stuff (more specific issues are easier for me to get to).

@jasonmunro commented on GitHub (Aug 28, 2018): I'm familiar with feature branch developing, and I understand the rational. For Cypht I'm sticking with master == development branch, releases are branched from there. It's just easier since I'm the only full time contributor :). I'm actually pretty happy with how things worked out - 4 instances with selenium tests (soon to be 5 once I resolve safari issues) is plenty for now. I'm also pleased to mention that I have a new test suite that: - navigates to the compose page - loads and submits a new message - navigates to the unread page - finds the new message in the list - opens the new message and loads the detail page Now that selenium testing is re-enabled in Travis, I'm going to close this issue, and open some new ones for the remaining stuff (more specific issues are easier for me to get to).

kerem commented

2026-02-25 21:34:28 +03:00

Author

Owner

@ulfgebhardt commented on GitHub (Oct 8, 2018):

@ulfgebhardt commented on GitHub (Oct 8, 2018): <3