| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-06 | |||
| 10:58:01 | stephenfin | bauzas: ^ | |
| 11:01:04 | bauzas | stephenfin: +W | |
| 11:01:26 | bauzas | stephenfin: but you need to rebase the whole tree now | |
| 11:01:51 | bauzas | (sorry, if it was other thing but a commit msg, I would have proposed a FUP) | |
| 11:02:14 | stephenfin | I'm not sure if I do or if Gerrit will do it for me, but I'm reworking the doc patch so can do so shortly if needed | |
| 11:02:27 | stephenfin | I'll wait to see if there are review comments first | |
| 11:11:27 | ierdem | Hello guys, i have a question about resizing a running VM. I tried to increase vCPU counts of a running VM and after that i connected succesfully but it is too slow to run anything. When i see process list via "top" command, i saw CPU usage is approximately 1000 percent and it decrease sometimes but increase again. Any suggestions please? | |
| 11:12:30 | stephenfin | bauzas: out out? :O | |
| 11:12:34 | stephenfin | :P | |
| 11:12:46 | bauzas | out from my room :p | |
| 11:13:05 | bauzas | Who Let the Dogs out out ? | |
| 11:13:44 | bauzas | https://www.youtube.com/watch?v=Qkuu0Lwb5EM | |
| 11:15:50 | sean-k-mooney | well regardign fancy new keys. i would rather just define a custom: name spaces that should be used for user defined extra_specs and declare all other namespaced keys as owned by nova | |
| 11:16:16 | sean-k-mooney | and if a new intree filter wants to add keys it gets its own namespace | |
| 11:16:44 | sean-k-mooney | if our of tree filter need to define keys the do it via custom: | |
| 11:17:11 | sean-k-mooney | ideally with a prefix e.g. custom:myfilter_mykey | |
| 11:30:14 | sean-k-mooney | ierdem: i assume you have oversubsrtion enabled on your cloud? | |
| 11:30:25 | sean-k-mooney | ierdem: is the host over subsibed | |
| 11:31:53 | sean-k-mooney | ierdem: it should like you either have someing in the vm that is broken/maliusly consumeing cpu, the vm workload need more vcpus or the host is over subsribed | |
| 11:32:54 | sean-k-mooney | ierdem: simply seeing high cpu usage in the guest is not an indication that something is wrong form a nova point of view | |
| 11:33:36 | ierdem | hmm, how can i check if the host is oversubscribed? | |
| 11:34:32 | ierdem | by the way, before resizing vCPU count, it was working fine | |
| 11:35:00 | kplant | a resize can move the instance to another hypervisor | |
| 11:36:41 | kplant | if you just do a 'nova hypervisor-show <uuid>' | |
| 11:36:49 | kplant | you can check vcpus vs vcpus_used | |
| 11:36:58 | kplant | if vcpus_used > vcpus; you're over subscribed | |
| 11:37:38 | ierdem | ok, thanks i will try and return to you | |
| 11:38:12 | kplant | also vcpus_used == vcpus is a bad idea, unless nova knows about vcpus you're reserving | |
| 11:38:53 | sean-k-mooney | yes by default resize to same host is disabled so it normally does a move and will only stay on the same host if the weigher consider it to be the best host | |
| 11:39:13 | sean-k-mooney | if you enable same host resize in the config | |
| 11:40:28 | sean-k-mooney | kplant: if you use vcpu_pin_set or (cpu_shared_set and cpu_dedicated_set) then yes vcpu_used == vcpu shared is fine | |
| 11:40:59 | kplant | yeah, if not ideal :-) | |
| 11:41:18 | sean-k-mooney | ideally you should sue the *_sets for doing host reservation instead fo the reserved_host_cpus | |
| 11:41:25 | sean-k-mooney | *use | |
| 11:42:18 | sean-k-mooney | if you use the set you can choose which cpus to reserve and then you can use systemd to run the host process only on thos cores | |
| 11:42:34 | kplant | you can go further and use the bootloader | |
| 11:42:56 | sean-k-mooney | using isolcpus? is so you should really avoid that | |
| 11:43:07 | kplant | my only complaint with *_sets is you can't have open ended ranges | |
| 11:43:13 | kplant | like "3-" | |
| 11:43:24 | kplant | makes it easier to deal with hosts with, not vastly, different configs | |
| 11:43:40 | sean-k-mooney | you can do negations | |
| 11:43:51 | sean-k-mooney | "^0-2" | |
| 11:43:59 | kplant | ooo | |
| 11:44:10 | sean-k-mooney | i think negated ranges works | |
| 11:44:18 | sean-k-mooney | you can negate indivcual cores | |
| 11:45:01 | sean-k-mooney | we prably could add open ended ranges too you could always file a bug/blueprint | |
| 11:45:27 | kplant | i think negated ranges provide the same result | |
| 11:45:38 | kplant | nice | |
| 11:46:03 | sean-k-mooney | ill triple check the code to make sure that works | |
| 11:46:16 | sean-k-mooney | we support it for cpu_realtime_mask | |
| 11:46:27 | sean-k-mooney | but i think its supported in teh *_sets too | |
| 11:47:59 | sean-k-mooney | kplant: this is the code and it inclde the negation example for 1 core | |
| 11:48:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: template: consider openstack client besides novaclient https://review.opendev.org/717722 | |
| 11:48:46 | sean-k-mooney | kplant: yep negated ranges should work https://github.com/openstack/nova/blob/31aa4a6d7f0b8c301b093ad176ee2b44b5d3cec8/nova/virt/hardware.py#L134-L140 | |
| 11:50:05 | sean-k-mooney | sorry miss read that | |
| 11:50:53 | sean-k-mooney | the negation only works with 1 cpu | |
| 11:51:20 | kplant | looking at 113 -> 116 | |
| 11:51:26 | kplant | i think you might have been right originally | |
| 11:51:36 | kplant | it tests for ^str == '^' | |
| 11:51:39 | kplant | then proceeds as normal | |
| 11:51:47 | kplant | the next try/catch is for a range | |
| 11:52:38 | kplant | oof, that's ignoring what you highlighted though | |
| 11:53:32 | sean-k-mooney | https://github.com/openstack/nova/blob/824bc358c25564ed6603d8f94587abd8902fe5af/nova/tests/unit/virt/test_hardware.py#L132-L133 | |
| 11:53:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: doc: require openstack client change for every new API microversion https://review.opendev.org/717727 | |
| 11:53:41 | sean-k-mooney | unit tests are awsome :) | |
| 11:53:41 | kplant | that's still not a show stopper | |
| 11:53:48 | sean-k-mooney | although docs would also help | |
| 11:54:01 | kplant | as long as the _set string is _only_ negations, every other vcpu should implicitly be available for scheduling, no? | |
| 11:54:52 | sean-k-mooney | that is the behavior i would want as an enduser | |
| 11:56:26 | sean-k-mooney | but no unfortunetly that is not what the behvior is today | |
| 11:56:30 | kplant | i probably just over thought it and gave myself a reason to be lazy and use reserved_host_cpus instead of cpu_dedicated_set | |
| 11:56:41 | kplant | oh, really? | |
| 11:57:00 | openstackgerrit | Merged openstack/nova master: Add info about affinity requests to the troubleshooting doc https://review.opendev.org/715092 | |
| 11:57:01 | sean-k-mooney | it starts with an empty set cpuset_ids = set() | |
| 11:57:08 | openstackgerrit | Merged openstack/nova master: Stabilize functional tests https://review.opendev.org/717070 | |
| 11:57:09 | kplant | eek | |
| 11:57:15 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in security groups policy https://review.opendev.org/716786 | |
| 11:57:16 | sean-k-mooney | and after we loop over every thing cpuset_ids -= cpuset_reject_ids | |
| 11:57:39 | sean-k-mooney | it would not be hard to add that behavior. | |
| 11:58:02 | kplant | i would expect assuming all cpus to actually be faster code | |
| 11:58:13 | kplant | that's more likely in line with what the result will be | |
| 11:58:37 | sean-k-mooney | kplant: that code does not know how many cpus you have | |
| 11:59:44 | kplant | fair, it would come from sql | |
| 12:00:45 | sean-k-mooney | not quite where this is used is on the comptue node before we store the info in the db but anyway if you do have a propoasl for improving this feel free to write it up | |
| 12:02:09 | kplant | is this the better behavior? the current behavior forces the user to be explicit | |
| 12:02:39 | kplant | i guess i can always submit it and get input that way | |
| 12:06:40 | sean-k-mooney | kplant: if you dont set the config options all cores are assumed to be usable | |
| 12:06:59 | sean-k-mooney | kplant: so the idea was of you opt in you should say what you want | |
| 12:08:35 | kplant | i think that mindset still applies, just change the behavior from cpu_*_set to a merge behavior instead of replace | |
| 12:08:43 | kplant | i think that's reasonable | |
| 12:08:47 | sean-k-mooney | kplant: that said i have wanted to remove the reserved_host_cpus options since we first added vcpu_pin_set so makeing it nicer to use the reserved_host_cpus will help with that goal | |
| 12:08:49 | kplant | just want to make sure before i waste time with a bp | |
| 12:10:26 | kplant | waste other people's time* | |
| 12:10:33 | sean-k-mooney | kplant: well we would need to keep backwards compatbliy so we could not make it merge by defaul but we could change the behavior so that if you only specify negation then we woud assume all cpus were valid and apply the negation | |
| 12:11:23 | kplant | very fair point | |
| 12:11:37 | kplant | that would make cpu_dedicated_set = "4" == all cpus | |
| 12:12:26 | sean-k-mooney | kplant: ill file a bug | |
| 12:12:58 | kplant | appreciate that | |
| 12:16:05 | sean-k-mooney | kplant: the only issue really is that now that we have two ranges cpu_share_set and cpu_dedicated_set | |
| 12:16:18 | sean-k-mooney | it become less uesful but its still useful | |