Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-09
16:13:18 lyarwood gmann: it's failing pretty often at the moment due to a an issue with libvirt/QEMU on bionic
16:13:56 gmann you mean on victoria gate? on master gate, it should run on Focal
16:13:56 lyarwood melwitt: https://review.opendev.org/#/q/owner:self+topic:native-zuulv3-migration+status:open - wire up a native zuulv3 multinode ceph job
16:14:13 lyarwood gmann: master gate, multinode grenade still uses bionic
16:14:23 lyarwood gmann: and that's the problem here
16:14:32 lyarwood gmann: or we can move it to NV
16:14:41 gmann lyarwood: oh we should move it to Focal
16:15:04 gmann ah legacy job
16:15:09 melwitt lyarwood: sorry, I don't understand what I'm looking at here that's related to ceph
16:15:41 gmann lyarwood: i did not move base legacy job on bionic
16:17:47 gmann lyarwood: I think we can move to zuulv3 using script for now like in PS1 - https://review.opendev.org/#/c/742056/1 and once ceph grenade base is ready then remove the use of script ?
16:18:06 lyarwood gmann: we can try
16:18:24 gmann ok let me update it.
16:22:03 lyarwood melwitt: sorry nova-grenade-multinode currently does some ceph stuff manually via the live migration hook - https://github.com/openstack/nova/blob/80b807a4c590980b9e514042778bd8c277e89e40/playbooks/legacy/nova-grenade-multinode/run.yaml#L58 & https://github.com/openstack/nova/blob/80b807a4c590980b9e514042778bd8c277e89e40/gate/live_migration/hooks/run_tests.sh#L55-L65
16:22:41 lyarwood melwitt: I wanted to break this out into a native zuulv3 job based on a multinode ceph job but that's taking a while to work out on the topic I shared above
16:23:37 melwitt lyarwood: ok, so the manual stuff could be made "native" somehow. I did not know that
16:24:25 lyarwood melwitt: yeah instead of calling specific bash functions from the plugin I just wanted to have a generic job that would deploy multinode ceph that we'd run the LM tests on in Nova
16:25:00 lyarwood melwitt: I made some progress a while ago with the key sharing etc just became stuck at the end with getting the subnode to actually connect to the ceph cluster on the main node
16:26:23 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add live migration tests for PCI, SR-IOV servers https://review.opendev.org/746950
16:26:23 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Expand SR-IOV live migration tests with NUMA https://review.opendev.org/749360
16:27:12 melwitt lyarwood: I see, thanks, that helps. I hoped to be able to help in some way since I am concerned about the coverage loss but didn't know where to look or start
16:32:17 lyarwood gmann: so are you just going to change the base job and hope that works?
16:32:46 lyarwood to tempest-multinode-full-py3
16:34:14 openstackgerrit Merged openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757
16:34:25 openstackgerrit Merged openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777
16:35:24 gmann lyarwood: not tempest but grenade-multinode and running run_tests.sh in post phase with disable to run smoke tests on new node (which run as part of grenade-multinode playbooks)
16:37:17 lyarwood gmann: right I'm asking which base job you're going to use
16:37:53 gmann lyarwood: for grenade anyways we need to use grenade-multinode
16:39:21 lyarwood gmann: right sorry, that's zuulv3 based and would just use the scripts to avoid us losing coverage
16:39:48 gmann yeah
16:40:00 lyarwood got ya
17:09:47 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Migrate nova-grenade-multinode job to zuulv3 native https://review.opendev.org/742056
17:13:12 openstackgerrit Lee Yarwood proposed openstack/nova master: Add os-volume_attachments reference docs https://review.opendev.org/760971
17:24:18 openstackgerrit Merged openstack/nova stable/victoria: [doc]: Fix glance image_metadata link https://review.opendev.org/761423
17:27:41 openstackgerrit Dat Le proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642
17:44:38 openstackgerrit sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432
17:45:08 sean-k-mooney stephenfin: lyarwood can you re +w https://review.opendev.org/751642
17:45:15 sean-k-mooney it was rebased
17:45:37 stephenfin done
17:45:42 sean-k-mooney thanks
18:07:12 openstackgerrit Ghanshyam Mann proposed openstack/nova master: DNM: Testing system scope in tempest https://review.opendev.org/740124
18:20:16 openstackgerrit Elod Illes proposed openstack/nova stable/ussuri: [doc]: Fix glance image_metadata link https://review.opendev.org/761977
18:23:06 mnaser sean-k-mooney: have you done some investigation by any chance on any potential ways of speeding up nova startup time in environments with compute nodes that have a large port count
18:23:28 mnaser in this case, it takes almost 5-6 minutes for the agent to go up in an env with ~150-160 ports
18:23:40 mnaser using osv
18:23:52 mnaser i havent tried migrating to the native ovsdb driver, maybe that might help, but a whole lot of plugging happens
18:26:42 sean-k-mooney am not really. we have seen that usign native does indeed help form your previous issues but in general we do need to ensure that all network interfacs are plugged for all vms when the compute agent starts
18:27:40 sean-k-mooney it also depend on the release i know we fixed some issue with ip adress checking on newer release
18:27:51 mnaser sean-k-mooney: wonder if it might make sense to make this happen in a threadpool or something, as it seems to be happening one-by-one
18:28:07 sean-k-mooney not really
18:28:23 mnaser because right now in some of those bigger vms, nova-compute reports down as it goes through the restart that takes ~6m
18:28:27 sean-k-mooney we could but we have to wait for them all to complete
18:29:09 sean-k-mooney so if we dispatch them to a tread pool we will get more paralium but we will the fill the pool and have to wait for it to complete
18:29:33 sean-k-mooney plugging form a nova side shoudl not take that long in general
18:29:45 sean-k-mooney with the native driver its much much faster
18:30:17 mnaser ok i guess thats' probably going to be my next step to see how that improves it
18:30:19 sean-k-mooney as its not spawnting multiple shells via privsep in that process
18:31:32 mnaser sean-k-mooney: seems valid, i'll check it out and report and let you know what sort of improvement i see :)
18:31:34 sean-k-mooney putting it in a tread pool would basically just increase the batch size at the cost of memroy during start up.
18:32:17 sean-k-mooney cool there would be a speed up form a thread pool but not as much as using the native driver
18:50:33 openstackgerrit Merged openstack/nova master: virt: Remove 'get_per_instance_usage' API https://review.opendev.org/749318
18:51:22 lyarwood \o/
18:59:50 sean-k-mooney lyarwood: was that for the removal ^
19:00:10 sean-k-mooney oh because something finally merged in the gate :P
19:00:21 sean-k-mooney both are good
19:36:21 dansmith hmm, on focal the ceph devstack plugin tells me I'm on an unsupported distro
19:36:30 dansmith lyarwood: any idea about that?
19:37:52 lyarwood dansmith: no idea, I thought it was there?
19:37:55 lyarwood # git grep f32
19:37:57 lyarwood devstack/lib/ceph: if [[ ! ${DISTRO} =~ (focal|bionic|xenial|f31|f32) ]]; then
19:38:27 dansmith hmm, I nuked opt/stack but maybe I've got residue somewhere
19:38:33 dansmith it's checking against a list that doesn't include focal
19:38:53 dansmith heh, doh
19:38:54 dansmith enable_plugin devstack-plugin-ceph /home/dan/devstack-plugin-ceph
19:59:32 sean-k-mooney oh your using your local copy
19:59:44 sean-k-mooney got an old version checked out
19:59:49 dansmith yeah, I nuked /opt/stack and then it cloned it right back there for me :P
20:00:17 sean-k-mooney yep i do that intentionally for the most part and put all my repos in /opt/repos
20:00:32 sean-k-mooney just so it does not clone over itself
20:05:21 sean-k-mooney gmann:is there a spec or something covering the use of token scopes with nova. i dont see anythin after the ussuri spec https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/policy-defaults-refresh.html
20:05:44 sean-k-mooney that does nto cover the domains
20:06:06 sean-k-mooney the 4 scopes in https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/policy-defaults-refresh.html#use-cases make sense to me
20:06:41 sean-k-mooney but i dont see a defeintion for system_member, project_admin, domain_(reader, member or admin)
20:07:36 sean-k-mooney we have some interset down stream in supporting a 3*3 matix (reader, memeber, admin) *( project, domain, system)
20:08:38 sean-k-mooney but i dont recall ever disscuing domains or proejct admin and system memebr in the context of nova upstream
20:09:52 sean-k-mooney https://review.opendev.org/#/c/698397/14/etc/rbac-persona-accounts.yaml.sample has entries for all 9 roles
20:10:31 sean-k-mooney but given we have not had any specs or cross project session on this im having a hard time relateing that to how we would support this in nova
20:10:50 sean-k-mooney gmann: i know there was a pop up team but any pointers on where this might have been discussed
20:12:55 sean-k-mooney melwitt: johnthetubaguy any input on ^
20:16:21 gmann sean-k-mooney: we have project_admin in nova for case like creating server with host specify but system_member is not there as not needed from current policy rules
20:16:41 gmann also domain scope (it's three pesona)
20:16:56 sean-k-mooney do you have docs for this
20:17:06 sean-k-mooney there were only 4 included in the spec
20:17:12 sean-k-mooney project_admin was not one of them
20:17:17 gmann yeah, I updated policy doc for that
20:18:04 sean-k-mooney ok because form a spec point of vew we only ever approved system_admin, project_memeber system_reader and project_reader
20:18:32 gmann sean-k-mooney: this does not include what all combination nova support but give an idea on new policy - https://docs.openstack.org/nova/latest/configuration/policy-concepts.html
20:18:33 sean-k-mooney so since this is an api cahnge im kind of confused why we dont have a spec for supporting other scopes and ropels in nova
20:19:23 melwitt yeah, that ^ is the doc I was searching for wrt the support in nova
20:19:24 gmann sean-k-mooney: project_admin came up during implementation only and that is for create server with specific request (like force host, zero disk flavor etc) on;y

Earlier   Later