Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-09
14:42:14 mriedem if you're using versioned objects it does yes
14:42:24 efried versioned objects
14:42:34 efried you mean the RequestSpec?
14:42:39 mriedem RequestGroup
14:42:40 mriedem but yes
14:42:52 mriedem https://review.opendev.org/#/q/topic:bug/1777591+(status:open+OR+status:merged)
14:43:29 mriedem it's really 2 changes, one to backport which just disables the limit if the server is in a group
14:43:34 efried groan, okay.
14:43:43 mriedem and another to be smarter about in_tree if it's in a strict affinity policy
14:43:49 mriedem but noting that ^ is racy
14:44:34 tssurya mriedem, efired: ok so we stick to what we discussed in the ptg right ?
14:44:49 tssurya the new in_tree affinity thing stays in train then
14:45:12 tssurya mriedem: yea you might be right, I can trace it back to at least queens
14:45:16 mriedem max_placement_results was new in queens so i guess it goes to queens
14:45:33 mriedem ganso: no not hypervisor list, sec
14:45:40 efried tssurya: That's not my favorite misspelling of my handle.
14:46:02 mriedem she's channeling her inner trump
14:46:04 tssurya efried: yikes, I didn't even realised I missplled it :D
14:46:21 mriedem tssurya is a big fan, some are saying the biggest fan
14:46:27 mriedem folks
14:46:37 efried anyway, +2 on this patch, thanks for explaining
14:46:37 tssurya mriedem: :P
14:47:17 tssurya thanks efried, mriedem
14:47:20 mriedem ganso: more like this https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-hypervisor-servers
14:47:40 mriedem ganso: i'm wondering if ^ will show 4 for that node while running_vms shows 2
14:47:46 ganso mriedem: oh I see!
14:47:52 ganso mriedem: will run it and report back!
14:47:53 mriedem if ^ shows 2 servers as well, then you have some issue
14:48:07 mriedem like, the db might say 2 of those guests you think are on that node actually aren't
14:48:12 mriedem e.g. failed live migration or something
14:50:08 ganso mriedem: it does list 4 VMs! =)
14:51:00 mriedem ok so yeah the stats stuff is busted somehow, but you'd likely need to add logging statements to the code to debug it
14:51:55 ganso mriedem: ok, so... I am curious on how I could reset the value, and double check if it is being recalculated incorrectly, or if it is not being recalculated at all
14:52:42 ganso mriedem: like, if I restart the service (nova-cpu I'd assume), it would reset to 0 and recalculate? Does it help if I set the value to 0 in the database and then restart the service?
14:53:13 mriedem the periodic should set it every time
14:53:26 mriedem so you could set it to 0 in the db if you want and wait for the periodic to run, or restart the service
14:54:21 ganso mriedem: ok, will try that, thank you!
14:54:43 openstackgerrit Merged openstack/nova stable/rocky: Use migration_status during volume migrating and retyping https://review.opendev.org/657577
14:59:05 mriedem tssurya: comments in that change, but it's not just the group hint
14:59:14 ganso mriedem: sorry, one more question. I am trying to track the name of the periodic task, I haven't seen periodic_task decorators or where they are registered in resource_tracker.py. But it seems the periodic task is either "update_available_resource" or another method that invokes this one. Could you please confirm?
14:59:15 mriedem we also have same_host/different_host hints/filters
14:59:32 mriedem ganso: it's the ComputeManager.update_available_resource method
14:59:43 mriedem called on start of the compute service and in a periodic
14:59:44 ganso mriedem: oh it starts in the ComputeManager, thank you!
14:59:57 tssurya mriedem: oh hmm looking
15:01:18 mriedem https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#samehostfilter
15:01:41 mnaser so it looks like the kernel that enables nested by default (4.19) also added a gate that blocks live migrations if nested virt is enabled p
15:01:43 jangutter Did the ptg etherpad finally die?
15:02:10 mnaser ERROR nova.virt.libvirt.driver [-] [instance: 738eb188-1545-4126-b05b-54d384e55f73] Live Migration failure: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet: libvirtError: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet
15:02:29 mnaser so by default vmx is enabled, so every instance is not live migratable...
15:02:32 tssurya mriedem: I have never used that filter before, but since I am fixing it for the other two filters, I will update the patch for this one as well
15:02:54 tssurya and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#differenthostfilter like you have said
15:03:21 mriedem i really wish we could functionally test this code to recreate the bug but i don't think there is an easy way to do that
15:03:35 tssurya jangutter: I have been trying to access it too :( not sure if someone has a backup
15:04:38 tssurya mriedem: you mean the bug I am working on ?
15:11:14 mriedem tssurya: yes
15:11:36 mriedem but there isn't really a good way to do that with functional tests that don't involve stubbing placement responses, which defeats the purpose of a functional test
15:14:45 mnaser I’m trying to figure out how I can make both nested virt available as an opt in
15:15:09 mnaser But also without making complicated host aggregates and flavour explosion
15:18:29 mriedem required trait on vmx?
15:20:44 mriedem libvirt computes with vmx enabled would report the HW_CPU_X86_VMX trait right?
15:20:59 mriedem and you could have flavors and/or images which require that trait so the user can opt into nested virt that way
15:30:50 mriedem live migration is going to be hairy though right? because you could have instances land on those hosts that don't want nested virt and then you can't live migrate them?
15:31:13 mriedem not sure if "Live Migration failure: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet: libvirtError: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet" is a problem if vmx is on the source, dest or both
15:31:28 mnaser right, but then the idea is to make it opt-in and have a note that says "you won't be live migrated if you enable nested virtualize"
15:31:32 mnaser virt*
15:31:38 mriedem my point is,
15:31:45 mriedem the people that aren't opting in could still land on those hosts
15:31:52 mriedem i.e. this is the exclusion problem in the scheduler
15:31:59 mriedem for which forbidden aggregates are going to be used
15:32:16 mriedem https://specs.openstack.org/openstack/nova-specs/specs/train/approved/placement-req-filter-forbidden-aggregates.html
15:33:00 mriedem i'm not sure how you get around this without aggregates
15:34:59 mnaser hmm, so its not necessarily possible to "request" a custom cpu flag for a specific instance
15:36:55 openstackgerrit Rodrigo Barbieri proposed openstack/nova stable/queens: [DEBUG] Add functional confirm_migration_error test https://review.opendev.org/658136
15:37:12 openstackgerrit Dan Smith proposed openstack/nova master: Update the contributor doc for macos https://review.opendev.org/658137
15:37:21 dansmith trivial contributor doc update ^
15:43:58 bnemec efried: Would you be able to add the nova-specific freeze dates to https://releases.openstack.org/train/schedule.html ?
15:44:03 bnemec The keystone patch is a good example: https://review.opendev.org/#/c/653544/
15:44:22 bnemec It came up last week that people were surprised by some Nova freeze dates so it would be good to get them on the common schedule.
15:44:38 efried bnemec: can do. I think melwitt conveniently created the schedule a month or two ago in the wiki, please hold.
15:44:54 efried bnemec: https://wiki.openstack.org/wiki/Nova/Train_Release_Schedule
15:44:55 mriedem mnaser: the user would request via the flavor/image which would have the required cpu flag as a trait
15:45:31 bnemec efried: Cool, thanks.
15:46:00 bnemec I guess the main one would be spec freeze then.
15:46:42 mnaser mriedem: would it be appended to the 'existing' cpu flags or it would have to match a hypervisor that has those cpu flags (i.e I would have a bunch of nested-virt enabled computes, and those that aren't)
15:48:07 mriedem mnaser: my understanding from https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/report-cpu-features-as-traits.html is that the compute reports the cpu features as traits, and you can configure each compute with additional flags using https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu_model_extra_flags
15:48:48 mriedem i think cpu_model_extra_flags get reported as traits as well, but i would have to dig into the code to confirm
15:49:32 mnaser but if I add `cpu_model_extra_flags` that would then make every instance get nested virt
15:49:39 mnaser (on that host)
15:49:49 mriedem oh i see, you want per-instance
15:50:01 mnaser yeah, that way I don't have to maintain a whole bunch of host aggrgates
15:50:06 mriedem well, geez, that sounds like starlingx craziness bro!
15:50:28 mnaser is that something they do / looking to do ?
15:50:36 mriedem i want to say yes
15:51:19 mriedem i thought somewhat related to https://review.opendev.org/#/c/637834/ but not really what you want
15:51:29 mnaser it looks like the workflow of google is per-image based -- https://cloud.google.com/compute/docs/instances/enable-nested-virtualization-vm-instances
15:53:08 mriedem yar, i know we have things like hw_vif_model, hw_pointer_model, hw_scsi_model but no hw_cpu_model
15:54:30 mnaser yeah I guess the weird thing is `hw_cpu_model` actually ties into physical infra instead of virtual only
15:54:32 mriedem not sure where vcpu_model comes from on the instance
15:54:56 mnaser i.e. `hw_cpu_model` depends on the physical CPU having a certain capability so it might not always work

Earlier   Later