mirror of
https://github.com/retspen/webvirtcloud.git
synced 2026-04-25 23:45:56 +03:00
[GH-ISSUE #323] Ways to improve WVC #209
Labels
No labels
bug
enhancement
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/webvirtcloud#209
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Real-Gecko on GitHub (Jun 15, 2020).
Original GitHub issue: https://github.com/retspen/webvirtcloud/issues/323
Hey there! I think we need to discuss some things about development of WVC to avoid confusion with further development. Things to discuss are:
Feel free to suggest some other things to discuss in terms of development.
@Real-Gecko commented on GitHub (Jun 15, 2020):
I use VS Code for development and I use yapf with following settings:
I also use VS Code feature "Sort imports" for sorting imports of python libs. That's why you may see a lot of code shifting in my PRs without actual change in functionality. To avoid confusion I suggest we decide which formatter we use, to keep code same for all devs.
@Real-Gecko commented on GitHub (Jun 15, 2020):
Test coverage the thing I'm currently working on, slowly but surely we're getting there.
@Real-Gecko commented on GitHub (Jun 15, 2020):
I think we need to perform either migration squashing or redo migrations from scratch in the end of current dev cycle, when all changes will be settled, cause we're currently getting a lot of them.
We can also try and create a way to migrate from older version of WVC to newer, maybe with management command.
@Real-Gecko commented on GitHub (Jun 15, 2020):
We may create shell script for new installations and move all pre install tasks to there, like creation of
adminuser, maybe add ability to set some settings without actually editingsettings.pyetc.@catborise commented on GitHub (Jun 15, 2020):
hi @Real-Gecko i am a system engineer more than software developer. I never worked with a developer team, because of that i am not experienced with team working. but i prefer/tend to experience it. i can follow your guidance.
@catborise commented on GitHub (Jun 15, 2020):
migration squashing with after some period will be fine.(i did it with python2-->3 conversion.)
i moved setttings to appsettings page. for now settings.py is slimmed down.
i move them to appsettings but i think your admin app will be more suitable for that configs. it can be moved... (i am creating settings with setting.objects.using(db_alias).bulk_create migration but there should be more suitable way. i need help with that :) )
@catborise commented on GitHub (Jun 15, 2020):
after the lastest updates this approach(moving all pre install tasks) may be more suitable way.
i have never write coverage tests, i do not now how to... i should work on it.. i need time.
@Real-Gecko commented on GitHub (Jun 15, 2020):
What IDE do you use? Does it integrate with YAPF if so than everything is easy. I created
conf/requirements-dev.txt, it addsyapf,coverageanddjango-debug-toolbarto existing requirements.YAPF is used by me to auto format source files on save, really helps to keep code readable. However other devs may prefer other ways to format code, that's why I decided to discuss the issue. Settings for yapf I use ar in post 2. Try it out.
Coverage is used to run tests with coverage info, really helps to see what parts of your code are not tested. I use following command to run tests:
And this one to see the results:
You can use any browser you want instead of FF.
And finally Debug Toolbar really helps too. Installation and set up is pretty easy too.
@Real-Gecko commented on GitHub (Jun 15, 2020):
Yeah, that's cool, ability to change settings on the fly is always better than messing with settings.py.
Well that's not hard to do: one view, one model, one template. Let's do this
Populating data with migrations is acceptable, but Django devs recommend using fixtures, imo it's more flexible way, it will require some work during testing, but that's not something I would worry about.
@Real-Gecko commented on GitHub (Jun 15, 2020):
It sounds harder than it actually is. If you read tests I created you'll see they're pretty simple, but help a lot. Here's computes index page test:
Simply
GETindex page and check that response status is 200.@catborise commented on GitHub (Jun 15, 2020):
before that i was using pycharm, it is really nice ide for python(license is over and community edition lack of some capabilities).
Now i started using vscode. i can add yapf settings for sync.
@Real-Gecko commented on GitHub (Jun 16, 2020):
I see a lot of views doing more than one task which makes them hard to maintain and test. I think we need to split views functionality like so: one view - one task. For example list users is one view, create user is another, update is another etc.
@catborise commented on GitHub (Jun 16, 2020):
i do not know why retspen choose that approach but it is useful to slim down urls.
for me, it is fine to seperate small ones but instances/views.py can be painful to split :)
@Real-Gecko commented on GitHub (Jun 16, 2020):
urls.pyis just like route map user request can take to receive required answer, so it's ok for it to be big, but big views that do more than one job is bad imo.I'm currently working on computes where I've already split some functionality to smaller views. Instances are on the way.
@Real-Gecko commented on GitHub (Jun 18, 2020):
Correct me if I'm wrong: we establish new connection to libvirt host every time there's any operation on instance. Am I right?
@Real-Gecko commented on GitHub (Jun 18, 2020):
Never mind, look like I've figured it out.
@catborise commented on GitHub (Jun 18, 2020):
i think yes it starts new connection everytime but with the help of keep alive protocol, connections are cached. libvirt service on the host keep up open the connection to serve requests fastly...
@Real-Gecko commented on GitHub (Jun 18, 2020):
Yep, there's wvmConnectionManager class that pools all connections. I was just confused by wvmInstance close method.
@Real-Gecko commented on GitHub (Jun 18, 2020):
I'm currently working on instances app, some serious optimizations in progress. Please avoid any work in this app, to prevent source code clashing during PR.
@catborise commented on GitHub (Jun 18, 2020):
👍
currently i am not working on webvirtcloud master..
be careful instances app is a behemoth. :)
@Real-Gecko commented on GitHub (Jun 18, 2020):
Yep, but it's worth it, unoptimized compute instances view: more than 5 seconds to response, 41 database requests.

Optimized: 1.5 seconds, 2 database queries.

@catborise commented on GitHub (Jun 18, 2020):
wow 41 queries... it should be max five or six queries. 1. get compute 2.get instances, 3.update records, 4.check duplicates, 5.add missings, something must be out of control.
i found it: def refresh_instance_database(comp, inst_name, info) loops for all instances and updates all of them. that causes making queries relative to instance count.
i am wondering how to solve that.
@Real-Gecko commented on GitHub (Jun 18, 2020):
Currently I replaced it with this code:
But I don't like it too, cause it does not connect instances from WVM with domains on host.
If we try to connect 'em will get query hell again. So currently I think about workaround.
One solution is to get Instance objects from DB and work with it, but that way we don't get VM status on host.
@Real-Gecko commented on GitHub (Jun 18, 2020):
OK, I think I know what to do with VM status. I added
wvmInstancetoInstancemodel that we can use to get info about domain directly via libvirt:Result is looking good:
@catborise commented on GitHub (Jun 18, 2020):
i think you will seperate allinstances and host only intances list refreshing.
there is not need to check duplicate uuid and names on one host. but we should handle that among other hosts.
duplicate instance definition on other hosts could cause unwanted problems.
proxy with @cached_property is a nice approach. i like it.
@Real-Gecko commented on GitHub (Jun 18, 2020):
Actually I think no, even if there will two VMs with same UUID, but on different hosts it will not cause trouble. Cause this VMs are well on different hosts :D
But current code definitely troublesome:
As it does not take into account the fact that there could be instances with the same name on different computes.
@catborise commented on GitHub (Jun 18, 2020):
yes same name could cause confusion but doesn't same uuid cause migration prevention?
you are right unique name and uuid can be provided efficiently only within a cluster. these actions not a complete solution but only trivial checking
@Real-Gecko commented on GitHub (Jun 18, 2020):
Yep it's possible even though possibility of UUID clashing is really low, however I think in this case libvirt will throw some kind of error we will show to user, and then that's admin's task to deal with this tricky situation. In our case it's important to keep our DB in sync with hosts.
@Real-Gecko commented on GitHub (Jun 19, 2020):
I've pushed my WIP Instances branch into forked repo for testing and fun. Anyone willing to test and suggest is free to do so.
WARNING: heavy WIP, not for production use.
@Real-Gecko commented on GitHub (Jun 22, 2020):
I
base.htmlto contain this:So there will be no need to write page header in every template, things like search bar can go to
page_header_extraand all messages added bydjango.contrib.messagesframework will go below page header.@catborise commented on GitHub (Jun 22, 2020):
although effort or code length does not much change, if alignment problems does not occur, this way is more elegant than current. i like it
@Real-Gecko commented on GitHub (Jun 22, 2020):
This is just an example how things can be optimized, not a final version, anyone willing to suggest improvements is free to do so.
@catborise commented on GitHub (Jun 22, 2020):
as far as i see grouped view for index page is removed, practically.


group view:
nongrouped view:
all_host_vms = Instance.objects.all().prefetch_related('userinstance_set')does not contain host info, doesn't it?what do you think about that?
@Real-Gecko commented on GitHub (Jun 22, 2020):
I think replacing it with either accordion or tabs which will contain tables with instances. Current implementation does not work right anyway, try sorting the table by name or RAM you'll see what I mean.
@Real-Gecko commented on GitHub (Jun 22, 2020):
Accordion does not look bad IMO
@catborise commented on GitHub (Jun 22, 2020):
i added grouped view because there is not any info about host usage. especially ram usage. This info very valuable to get idea host load. other than that already nongrouped view is like that.
@Real-Gecko commented on GitHub (Jun 22, 2020):
I'll commit soon to instances branch, you'll see how it looks
@Real-Gecko commented on GitHub (Jun 22, 2020):
I've pushed changes de20f810a27ab528aff6699a33ab7fbb607be4bf, not the best looking for right now, but you get the idea, most important is the optimization of templates and DB queries.
@catborise commented on GitHub (Jun 22, 2020):
i think, database queries not criitical. query delay impact is very low compare to libvirt calls. focusing on queries would not affect very much.
@Real-Gecko commented on GitHub (Jun 22, 2020):
Actually, as far as I can tell from testing rewritten views DB queries impact is huge :)
@catborise commented on GitHub (Jun 22, 2020):
i cannot run your branch. gives error
always i get that error
@Real-Gecko commented on GitHub (Jun 22, 2020):
Is it the same error as in 2c8f092992bf657977c268e806a300cee3e30fce? If so, try commenting out
'webvirtcloud.middleware.ExceptionMiddleware',in yoursettings.pyand see where error is.@catborise commented on GitHub (Jun 22, 2020):
not reachable host caueses, i think. trying to get cpu from host info causes that.

other problem in instances.html:
{% widthratio vm.proxy.instance.info.1 1024 1 %} MiB</td>(Exception has occurred: VariableDoesNotExistFailed lookup for key [vm] in) i think this should be instance
and other in in instance_actions.html
(Exception has occurred: VariableDoesNotExist
Failed lookup for key [inst] in)
@Real-Gecko commented on GitHub (Jun 22, 2020):
Hmm, I cannot reproduce, can you remove problematic compute and retry.
@Real-Gecko commented on GitHub (Jun 22, 2020):
Fixed those.
@catborise commented on GitHub (Jun 22, 2020):
add a compute with fake name ip pass user. it accepts
@Real-Gecko commented on GitHub (Jun 22, 2020):
I tried, I do not get the error you describe, but instead I get another one :D
@catborise commented on GitHub (Jun 22, 2020):
after instance_action.html fix, click show console.: negative token generation :)

@catborise commented on GitHub (Jun 22, 2020):
forceoff power off powercycle does not work...
@catborise commented on GitHub (Jun 22, 2020):
you are a good backender :)

i am crying :)
wouldnt be nice implementation like this, if we could...

@Real-Gecko commented on GitHub (Jun 23, 2020):
Of course it's better, but how does act during sorting and filtering? Can you try and implement it?
@Real-Gecko commented on GitHub (Jun 23, 2020):
I made grouped instances index look like before 2be76c07e401d4a2875af35403ca55d550972460, but it definitely needs a rework.

@catborise commented on GitHub (Jun 24, 2020):
did you make any changes with console app?
i am going to work on some bugs and featues,
@Real-Gecko commented on GitHub (Jun 24, 2020):
Nope.
@Real-Gecko commented on GitHub (Jun 24, 2020):
Try latest commit c2e001290587344c62025e5c08689b134797935c and see if error still exists.
@catborise commented on GitHub (Jun 24, 2020):
still some errors happens not working yet because of some other errors:

@Real-Gecko commented on GitHub (Jun 24, 2020):
Ah yes,
storages/views.py, will commit that.@Real-Gecko commented on GitHub (Jun 24, 2020):
b8db11f66f1dcc997c741854cdd2b9488bac9551 grab this one.
I finally started to add instances tests and with only three of them increased overall coverage from 36 to 41 percent which is goooood :D
@catborise commented on GitHub (Jun 24, 2020):
yea it runs
but view not good. it should not show in the table
@catborise commented on GitHub (Jun 25, 2020):
we could use this datatable which has ability to group. sorting finding etc superrior than us. have you use it?
https://www.datatables.net/extensions/rowgroup/examples/initialisation/startAndEndRender.html
@Real-Gecko commented on GitHub (Jun 25, 2020):
https://github.com/wenzhixin/bootstrap-table is better IMO
There's also https://django-tables2.readthedocs.io/en/latest/ which makes templates even simpler.
@Real-Gecko commented on GitHub (Jun 25, 2020):
71d75327f21fbef126a57a378b36a5ef5584a76a
I made templates look like they did before
@catborise commented on GitHub (Jun 25, 2020):
we need "group view" or "detail view" or "tree view" like solution
https://examples.bootstrap-table.com/#extensions/treegrid.html is a suibtable one.
i do not see detail view or like that at django-tables2 .
@Real-Gecko commented on GitHub (Jun 25, 2020):
We can override built in templates https://django-tables2.readthedocs.io/en/latest/pages/custom-rendering.html#custom-template

We can even try and mix bs-table with django-tables2, first working on client side second on server side.
I think we need to think through new layout of every view, cause now it's a bit chaotic.
Currently design like this is quite popular:
Most links are on left side menu, which is collapsible, top bar is small and does not have a lot of links. I see such layout quite often in enterprise grade products.
@catborise commented on GitHub (Jun 25, 2020):
yea i like the dashboards like this. webvirtmgr has left side . webvirtcloud uses top navigation.


retspen wanted to start new project webvirtcloud v2(django rest framework + react + material ui) but for some reasons, he choose to wait... with v2 he also switched to the dasboard style view...
nearly all dashboard templates like you specified as above...
...
material-ui
i want to add a dasboard page to webvirtcloud but there was some hiccups before your reorg. may be it is more appropriate and easy adding a dashboard now before then...
@Real-Gecko commented on GitHub (Jun 25, 2020):
I'm working on instances tests, after they're done and branch is tested and merged I think we can start doing this. Instances currently has most messy layout, after splitting views I think it'll be easier to rewrite templates.
@catborise commented on GitHub (Jun 25, 2020):
stock bootstrap very limited for a dashboard style app. all templates getting help other css/js extensions.
to solve that problem patternfly extends bootstrap3 to get all necessary components.
https://www.patternfly.org/v4/documentation/core/demos/drawer
redhat uses this framework... may be with layout change it is considered to change css framework also...
@Real-Gecko commented on GitHub (Jun 25, 2020):
Was guestfsd updated for Python 3?
@catborise commented on GitHub (Jun 25, 2020):
i made some changes but i am not sure if it is working or not. i never used it.
@Real-Gecko commented on GitHub (Jun 25, 2020):
I'm currently trying it out and looks like it's a bit broken, will fix it.
@Real-Gecko commented on GitHub (Jun 25, 2020):
ffa85d01c1f66dfc0ca26573d2f190e0411c6e16 fixed gstfsd, but it requires root privileges to run, so I'm leaving change password and add public key functionality without testing.
@catborise commented on GitHub (Jun 26, 2020):
editing instance problems (retspen/master):
@Real-Gecko commented on GitHub (Jun 26, 2020):
Is the cause
must be
@Real-Gecko commented on GitHub (Jun 26, 2020):
Instances branch is almost there.
@Real-Gecko commented on GitHub (Jun 26, 2020):
There is a great app https://django-guardian.readthedocs.io/en/stable/ that allows to grant users per object permissions. I've never tried it, but it looks interesting, this will grant us more granular control over instances permissions eliminating the need for UserInstance model. But this improvement is for the future, not something we must implement right now.
@catborise commented on GitHub (Jun 26, 2020):
adding more features pushes to more granular permission. it will be nice more user roles then now. it should be considered after stabilizing current...
have you tried async django?
we need async capabilities for cloning instance/disk, migrate, backup... but i do not have an experience with it.
qemu provides async actions with "qemu_qpm" but not libvirt... may be django async does something?
@Real-Gecko commented on GitHub (Jun 26, 2020):
Async in Django is not fully there yet:
https://code.djangoproject.com/wiki/AsyncProject
https://speakerdeck.com/andrewgodwin/just-add-await-retrofitting-async-into-django?slide=42
So I think it's better to not bother at all for now, at least until Django 4.0.
In the mean time we'd better clean our source code a little and make it more consistent.
I finished views split for instances ae31225b0b531185f4cb39182c63a89c3fe7ef23 and looks like cleaned up all errors, anyone willing to test the branch is free to do so. In the mean time I'll continue writing auto tests for instances, we're already at 50% coverage. I'll create a PR to retspen/master as soon as no bugs remain. But there're still many places that can be improved in terms of code quality, especially in templates: it's a mess out there :D
@Real-Gecko commented on GitHub (Jun 26, 2020):
https://github.com/Real-Gecko/webvirtcloud/tree/instances
link for instances testing
@Real-Gecko commented on GitHub (Jun 26, 2020):
Added some more tests for instances, coverage is 55% now.
@catborise commented on GitHub (Jun 26, 2020):
i am working on reverse proxy fixing and console minor fixes. i will test it...