Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-16
07:47:40 EugenMayer am I right that using --user-data has 2 flavours, using it with --config-drive true will expect to have meta-data alike content there, without using config-drive yaml based cloud-init files are expected. So the same --user-data for 2 different subsystems on init?
07:48:12 EugenMayer (reading https://docs.openstack.org/nova/queens/user/config-drive.html)
08:03:22 gibi_ o/ morning nova
08:40:04 bauzas good spec review day, Nova
09:13:10 opendevreview Merged openstack/nova-specs master: Repropose flavour and image defined ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/810867
09:33:05 jengbers We have been adding servers to server groups by changing the database and the scheduler has always handled this well.
09:33:05 jengbers Morning, I have been searching OpenStack history, but I haven't been able to find why it is not possible to add existing instances to server groups. Does anyone know if that is just never discussed or if there is a fundamental technical problem?
09:38:38 gibi jengbers: the problem is consistency. When you add a server to a group then you can create a situation when the group membership and the actaul placement of the instance contradicts
09:38:54 gibi and the question is what to do then
09:39:01 gibi a) move the instance to restore consistency
09:39:10 gibi b) reject the addition of the instance to the group
09:40:11 gibi c) allow temporary inconsitency and let the next move operation on the instance fix the group
09:40:34 gibi I think we never agreed which direction to take
09:43:05 sean-k-mooney there was a propsoal to extend it recently to allow this but the suggestion then was to have teh add do migration which i dont thinkis the right approch
09:43:37 sean-k-mooney its particaly a probelm for the affinity policy since that is more likely to break then the anti affinity policy
09:48:17 kashyap bauzas: I might not be able to make the meeting today; I see it's at 17u CET :-(
09:48:27 kashyap Morning, BTW
09:48:56 bauzas kashyap: ah ok, no worries then
09:49:25 bauzas kashyap: just add notes to your specless ask in the agenda, so we can discuss there
09:50:03 kashyap bauzas: Yep; doing that now
09:50:56 kashyap bauzas: If there are any questions, you can ask me here, I'll answer when I'm back later in the evening. I'm away from 17-18u CET
09:52:48 jengbers I guess option b) would be the least surprising.
09:59:31 gibi jengbers: with option b) the problem is that then the user first somehow move the instance to the proper place (but the user has no tool for it) then add it to the group
10:00:34 opendevreview Merged openstack/nova-specs master: Store and allow libvirt instance device buses and models to be updated https://review.opendev.org/c/openstack/nova-specs/+/810235
10:01:17 sean-k-mooney gibi: if you have the correct wiegher enabel i belvie you can ask for an isntnace to be created on the same hos tor a different host to a specific instance via a scheduler hint
10:01:44 sean-k-mooney its certenly requires a lot of knoladge of nova and the instance to do correctly
10:02:18 gibi sean-k-mooney: if you use SameHost / DifferentHost filters then you don't need server groups
10:02:21 sean-k-mooney you cant a s a normal use cold migrate to the same host or simialrly aline them all
10:02:35 sean-k-mooney gibi: ya that is also kind of true
10:02:53 gibi they implement similar logic but a very different way :)
10:03:03 sean-k-mooney yep
10:03:21 sean-k-mooney i do wish we had ways to make server groups more useful
10:03:39 sean-k-mooney but its kind fo hard to extend them for the reason above
10:03:59 gibi sean-k-mooney: for that we need to solve jengbers' problem and also extend the logic to support multiple groups per instance (or nested groups
10:04:02 gibi )
10:04:16 gibi both is painfully missing but hard to solve
10:05:23 gibi bauzas: I'm done with the spec sweep. I could not really comment on the ironic one https://review.opendev.org/c/openstack/nova-specs/+/815789 and it seems nobody commented yet
10:05:37 gibi the rest of the specs has feedback
10:05:40 sean-k-mooney ya. on second tought we should have "server aggreates" in parallel to server groups. you knwo so we can aggreate servers and give that aggreate of servers a name and even have some metadta that can be shared like this server it the primary of the aggreate and just not have it related to vm placement at all
10:05:46 bauzas gibi: I still have 3 specs to look at
10:05:55 bauzas gibi: but OK, and thanks for the fish
10:05:55 sean-k-mooney that way we can pretend server groups dont exist :)
10:06:24 gibi sean-k-mooney: :D
10:14:06 jengbers gibi, sean-k-mooney: If it was only possible for admins, that could work, because they can also migrate servers, but for users it seems quite hard.
10:15:42 gibi jengbers: yeah that could work. Feel free to propose a spec about the new API to get wider discussion around it
10:16:26 jengbers On the other hand, they can power of and start an instance. I guess that would mean it is started on a different hypervisor.
10:20:54 sean-k-mooney really there are 2 paths we could take. 1 allow normall user ask nova to cold/live migrate an instance to be consitent with a server group that it is currently not a member of and then allow them to add the server to the group after, rejecting the request if the policy is volated
10:21:33 sean-k-mooney or 2 we can have the server group add triger the migration as part fo the request
10:24:39 kashyap Is this failing for anyone else too?
10:24:41 kashyap tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_migration_with_trunk [108.238299s] ... FAILED
10:25:41 gibi kashyap: could you link the test run?
10:25:56 kashyap gibi: https://zuul.opendev.org/t/openstack/build/632f8ed30e9a4a04a32648843f227ef3
10:25:58 jkulik we've downstream extended the server-groups api to allow adding servers after the fact. we opted for not allowing to add servers if this would be against the server-group's rules
10:25:59 gibi looking
10:27:53 gibi kashyap: I think you got hit by https://bugs.launchpad.net/neutron/+bug/1940425
10:28:06 jkulik it helps customers if they already spawned an instance and forgot the server-group and now want to spawn another instance in some affinity to the existing one
10:28:08 gibi the stack trace is the same
10:28:42 gibi jkulik, jengbers: so both of you would like the same behavior, you should team up proposing this upstream :)
10:28:46 kashyap gibi: Oh, thank you
10:29:10 kashyap gibi: Now what? ... Should I do a "recheck 1940425"?
10:29:25 kashyap Or pray to the ju-ju at the bottom of the sea? Or...
10:29:26 gibi kashyap: yepp, recheck bug 1940425
10:29:46 jkulik https://github.com/sapcc/nova/commit/7220be3968ee1dd257c9add88228cc5bb9857795 is the main commit downstream for us
10:30:16 jkulik gibi: yes, we talked internally already about proposing this upstream, but small team, much work :/
10:30:18 gibi kashyap: I added your run to the bug maybe that way we can get attention to the failure as it is still happening
10:30:21 kashyap gibi: Thx for the quick spot
10:31:03 gibi jkulik: no pressure, I know that type of frustration
10:35:46 bauzas jkulik: we tried discussing this in the past upstream, but operators are very afraid by the races conditions it creates
10:36:39 bauzas jkulik: problem is, in a distributed service model like Nova, you can't get a valid answer whether you can do it, as when you validate, you don't ask the nova-compute service
10:37:48 jkulik bauzas: that reminds me ... we wanted to change the DB to disallow having a server in multiple server-groups to help with races. we haven't done that, yet. thanks :D
10:38:20 bauzas in theory we should hold new instance creations per compute once you ask for adding a new instance to the group
10:40:02 jkulik our problem is a little different, still, as we use VMware and not libvirt. thus, we have a lot of hidden hypervisors as nova only sees the cluster. therefore, hard anti-affinity doesn't really matter for us that much
10:40:48 jkulik customers want to make sure they run on different hypervisors and thus we sync the server-groups to the VMware clusters. VMware then migrates VMs around to make sure the rules apply.
10:42:46 jkulik i.e. most of our customers depend on soft-anti-affinity, which is a Weigher in nova-scheduler anyways
11:43:54 kashyap bauzas: Alright, added it to the Open Discussion here: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
11:44:48 opendevreview Rajat Dhasmana proposed openstack/nova-specs master: Add spec for volume backed server rebuild https://review.opendev.org/c/openstack/nova-specs/+/809621
12:10:11 dmitriis gibi: tyvm for the feedback
12:10:50 gibi dmitriis: you are welcome. it is a well written spec, thanks for putting in the effort
12:22:45 sean-k-mooney i dont know if i set send on my respocne to the last verions of it
12:23:02 sean-k-mooney gibi you are correct the pci passhtoug filter will be suffiect without the prefilter
12:23:29 sean-k-mooney but the prefiletr can help reduce the set if we only report the new trait on host that have off path devices
12:23:47 sean-k-mooney and the capablity to use them of course
12:24:04 sean-k-mooney ill re review that specs later today
12:26:06 gibi sean-k-mooney: yes, exactly my argument, the prefilter is not mandatory but it is good to have
13:00:21 dmitriis sean-k-mooney: ack, ty for confirming
13:02:28 sean-k-mooney i dont currently have access to hardware to test what you have done but i may have access before the end of the cycle. if i do i might reach out to you and try and test it end to end although i dont know if i will have time to do that or not
13:24:48 elodilles bauzas: i'll update now the meeting wiki #stable section if that is not interfering with you right now
13:26:55 dmitriis someone to review it sooner than later.
13:26:55 dmitriis sean-k-mooney: btw, fnordahl and I have done end-to-end testing of this in a lab. Here's a PPA https://launchpad.net/~fnordahl/+archive/ubuntu/smartnic-enablement that was used in the process (the WIP reviews are in use there). It has https://listman.redhat.com/archives/libvir-list/2021-November/msg00431.html included as well - I am trying to get
13:28:20 dmitriis it doesn't yet have the prefilter and compute capability parts that were recently added to the spec but I will work on updating the WIP review soon with that and on raising a relevant os-traits change
13:30:44 dmitriis We had a VM booted with a floating IP assigned which we then connected to via a router. The flows were offloaded into the ConnectX-6 chip present on BF2.
13:31:52 sean-k-mooney i think i have ping that patch to people dowstream already but ill let the vert team know
13:31:57 sean-k-mooney *virt
13:32:11 dmitriis ack, tyvm
13:33:28 dmitriis implementation.
13:33:28 dmitriis sean-k-mooney: besides testing overlays we also tried using VLAN provider networks. That worked as well but the only thing to note there is that collocating VMs with ports attached to overlay networks via PCI devices with the ones that are directly attached to VLAN networks is going to be problematic with the current whitelist based lookup
13:34:06 dmitriis entries in the whitelist get a physnet tag (either null for overlay networks or a physnet label)
13:34:28 sean-k-mooney correct they do
13:34:36 sean-k-mooney and technially null was never intended to be supported
13:34:36 dmitriis but there is only one vendor/device id pair

Earlier   Later